Very slow when building from big base image

[hulucc]

Actual behavior Build takes 10 minutes over when build with a big base image doing same simple job.

Expected behavior It should be quick.

To Reproduce Steps to reproduce the behavior:

1. Just build
2. If you check the debug log, you will see it's almost recreating every file in base image

   ...
   INFO[0034] Unpacking rootfs as cmd COPY ./* ./ requires it. 
   ...
   DEBU[0100] Adding /usr/local/go/src/go/doc/testdata/e.2.golden to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Gc/Pf.pl to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/c++/8/parallel/merge.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/interface_freebsd.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/proxy_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/netinet/if_tr.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/http/httputil/reverseproxy.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/platform_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/lib2to3 to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/util-linux/README.Debian to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/info/libmount1:amd64.list to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl/5.28.1/Pod/Simple/Text.pm to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/syscall/types_netbsd.go to layer, because it was changed. 
   DEBU[0100] Adding /etc/ssl/certs/TUBITAK_Kamu_SM_SSL_Kok_Sertifikasi_-_Surum_1.pem to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/encodings/euc_jp.pyc to layer, because it was changed. 
   DEBU[0100] Adding /etc/pam.d/login to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/Asia/Ashkhabad to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/libperl5.28/changelog.Debian.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man8/ctstat.8.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/vendor/golang.org/x/sys/unix/zsyscall_darwin_arm.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/pkg/linux_amd64_race/vendor/golang.org/x/text/unicode to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/dist-packages/mercurial/thirdparty/cbor/cbor2/decoder.py to layer, because it was changed. 
   DEBU[0100] Adding /etc/rmt to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/ca-certificates/mozilla/CFCA_EV_ROOT.crt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/os/file_windows.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/git-branch.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/right/Africa/Mogadishu to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/reflect to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man8/tipc-node.8.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/python/README.Debian to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/mod/rsc.io_quote_v0.0.0-20180214005840-23179ee8a569.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/internal/cpu/cpu_no_init.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/syscall/types_linux.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/perl-base/auto/File to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/posixfile.pyc to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/gofmt/testdata/slices1.golden to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/script/mod_patterns_vendor.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/compile/internal/ssa/testdata/hist.dlv-opt.nexts to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/testdata/testprogcgo/windows to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/America/Fort_Nelson to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/America/Bahia to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/git/RelNotes/1.6.4.2.txt to layer, because it was changed. 
   DEBU[0100] Adding /etc/security/limits.d to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/src/vend/x/vendor/p to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/race/sched_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/misc/cgo/test/cgo_thread_lock.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/ssl/misc/tsget.pl to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/vi/LC_MESSAGES/gold.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/sys_openbsd_amd64.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/git-core/git-parse-remote to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/America/Rosario to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/scriptreplay.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/failssh/ssh to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue27595.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/encodings/cp1255.py to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue31777.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/src/vend/vendor/p to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/paste.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/mercurial/templates/spartan/filelogentry.tmpl to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/perl/5.28.1/Config_git.pl to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/c++/8/ext/pb_ds/detail/cc_hash_table_map_/resize_no_store_hash_fn_imps.hpp to layer, because it was changed. 
   DEBU[0100] Adding /etc/ssl/certs/1e09d511.0 to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/fr/man5/apt_preferences.5.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue12413.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/netfilter/ipset/ip_set_bitmap.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/pkg/linux_amd64_race/encoding/binary.a to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/apr-util-1/apr_dbm_db-1.so to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/Etc/GMT+4 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/time/zoneinfo_abbrs_windows.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/textproto/writer_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/memmove_s390x.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/internal/atomic/asm_ppc64x.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/gcc/x86_64-linux-gnu/8/include/pkuintrin.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/libmnl0 to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/rdma/ib_user_cm.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/openssl-list.1ssl.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/bin/x86_64-linux-gnu-ranlib to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/dist-packages/mercurial/wireprototypes.py to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/iota.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/dist-packages/mercurial/store.pyc to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/modulefinder.pyc to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/info/libstdc++6:amd64.postinst to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/git/RelNotes/1.7.8.6.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/crypto/tls/testdata/Server-SSLv3-RSA-RC4 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/os/path_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue6750.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man8/fstrim.8.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/git/RelNotes/1.8.1.5.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/math/big/arith_arm64.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/go/types/testdata/constdecl.src to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/closure.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/arcfb.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/right/Australia/Currie to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl/5.28.1/unicore/lib/Jg to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/git-core/templates/hooks to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/fi/LC_MESSAGES/ld.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/right/Europe/Monaco to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl/5.28.1/Digest.pm to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.3 to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/de/LC_MESSAGES/grep.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/script/mod_doc.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/debug/garbage.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/git-core/mergetools/kdiff3 to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/Asia/Srednekolymsk to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/netrose to layer, because it was changed. 
   DEBU[0100] Adding /lib/x86_64-linux-gnu/libcap-ng.so.0 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/shadow/root2/src/foo/foo.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/compile/internal/syntax/dumper.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/internal/poll/sendfile_windows.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/c++/8/ext/pb_ds/detail/pat_trie_/policy_access_fn_imps.hpp to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/PRC to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/zh_CN/man1/newgrp.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /etc/ssl/certs/f060240e.0 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/closedchan.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/git-core/git-merge-resolve to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man3/Dpkg::Index.3perl.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/internal/run/run.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/id/LC_MESSAGES/libidn2.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/misc/cgo/testplugin/testdata/sub/plugin1/plugin1.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/netfilter_ipv6/ip6t_NPT.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/git-gc.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/python/debian_defaults to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/c++/8/bits/allocated_ptr.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl/5.28.1/unicore/lib/GCB/PP.pl to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/vendor/golang.org/x/net/route/interface_announce.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/internal/sys/arch_mipsle.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/de/LC_MESSAGES/make.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/script/mod_upgrade_patch.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/git-core/git-remote-testsvn to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/info/libglib2.0-0:amd64.md5sums to layer, because it was changed. 
   DEBU[0100] Adding /lib/terminfo/s/screen to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man8/tc-tbf.8.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue11610.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/bench/go1/http_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/ko/LC_MESSAGES/tar.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/script/mod_case_cgo.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/Etc/GMT+10 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/image/testdata/video-005.gray.q50.progressive.jpeg to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/http/cgi to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/cgo_netbsd.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/conf_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/rdma/ib_user_ioctl_verbs.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/America/Santa_Isabel to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/doc/codewalk to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/CA.pl.1ssl.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/syscall/syscall_linux_ppc64x.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/dist-packages/mercurial/templater.py to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/info/libsystemd0:amd64.list to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/initcomma.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/bin/apt-get to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man8/sulogin.8.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/vendor/golang.org/x/arch/x86/x86asm/gnu.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/zoneinfo/posix/Pacific/Gambier to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/ca-certificates/mozilla/SZAFIR_ROOT_CA2.crt to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/sv/LC_MESSAGES/bash.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/internal/imports/testdata/android/d_linux_arm64.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/runtime/cgo/gcc_netbsd_arm.c to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/c++/8/ext/pb_ds/detail/cc_hash_table_map_/resize_store_hash_fn_imps.hpp to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl5/Debconf/Element/Teletype to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/perl5/Dpkg/Compression.pm to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/python2.7/distutils/extension.pyc to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/updates to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/id/LC_MESSAGES/Linux-PAM.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/vendor/golang.org/x/arch/arm/armasm/decode.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/pkg/linux_amd64_race/database/sql.a to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/issue4495.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/locale/C.UTF-8 to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/vendor/golang.org/x/sys/AUTHORS to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/mercurial/templates/gitweb/help.tmpl to layer, because it was changed. 
   DEBU[0100] Adding /var/lib/dpkg/info/liblz4-1:amd64.md5sums to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man5/sources.list.5.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/mercurial/NEWS.Debian.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/python/debpython/pydist.py to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/doc/ucf/changelog.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/net/internal/socktest/switch_stub.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/os/executable.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/compress/flate/testdata/huffman-text-shift.in to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/agpgart.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/man/man1/gpg-wks-server.1.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/char_lit.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/x86_64-linux-gnu/perl/debian-config-data-5.28.1/config.sh.shared.gz to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/internal/reflectlite/all_test.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/test/fixedbugs/bug468.go to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/bpfilter.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/share/locale/bg/LC_MESSAGES/grep.mo to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/go/testdata/mod/patch.example.com_direct_v1.0.0.txt to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/go/types/testdata/conversions.src to layer, because it was changed. 
   DEBU[0100] Adding /usr/include/linux/keyctl.h to layer, because it was changed. 
   DEBU[0100] Adding /usr/local/go/src/cmd/vendor/golang.org/x/sys/unix/asm_netbsd_arm64.s to layer, because it was changed. 
   DEBU[0100] Adding /usr/lib/git-core/git-show-index to layer, because it was changed. 
   ...
   INFO[0832] Taking snapshot of full filesystem...
   ...

Additional Information

• Dockerfile

  FROM golang
  WORKDIR /app
  COPY ./* ./

• Build Context Doesn't matter, it can be any file

• Kaniko Image (fully qualified with digest) latest

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [ ]
  Please check if this error is seen when you use --cache flag	- [ ]
  Please check if your dockerfile is a multistage dockerfile	- [ ]

[hulucc]

It's my problem. When the log saying Unpacking rootfs, actually it's downloading the base image. So the slowness is caused by the network. After I populate cache from mirror, everything is ok. /kaniko/warmer -f --image=<mirror>/library/golang

[LKI]

Save my day

[go-xmyang]

It's my problem. When the log saying Unpacking rootfs, actually it's downloading the base image. So the slowness is caused by the network. After I populate cache from mirror, everything is ok. /kaniko/warmer -f --image=<mirror>/library/golang

Save my day.