search

meltwater / drone-cache

A Drone plugin for caching current workspace files between builds to reduce your build times
https://underthehood.meltwater.com/blog/2019/04/10/making-drone-builds-10-times-faster/
Apache License 2.0
338 stars 81 forks source link

Tar archive does not restore file (or directory) modification time stamps #210

Closed raphendyr closed 1 year ago

raphendyr commented 2 years ago

Describe the bug

Restoring the cache doesn't restore file modification times.

This can effect application which check if a file has expired based on the time stamp. In my case, the dependency-check-maven, which updates CVE database daily and seems to check this based on the cache file timestamp.

To Reproduce

I used the following bash script to develop a fix for this, thus it works as a way to reproduce.

tag=$(git describe --dirty)
image="meltwater/drone-cache:$tag"

mkdir -p my-app/{m2,a/b/c} my-cache
touch -t 2201010101 my-app/m2/my-test-file1.txt
touch -t 2202020202 my-app/m2/my-test-file2.txt
touch -t 2212120000 my-app/a/b/c/foo.txt
touch -t 2212120000 my-app/a/b/c/bar.txt
touch -t 2212120000 my-app/a/b/c
touch -t 2212120000 my-app/a/b
touch -t 2212120000 my-app/a
ls -lR my-app
stat my-app/a

echo " ## Rebuild"
docker run --rm \
      -v "$PWD/my-app":/app \
      -v "$PWD/my-cache":/cache \
      -w /app \
      -e DRONE_REPO=octocat/hello-world \
      -e DRONE_REPO_BRANCH=master \
      -e DRONE_COMMIT_BRANCH=master \
      -e PLUGIN_BACKEND=filesystem \
      -e PLUGIN_CACHE_KEY=volume \
      -e PLUGIN_FILESYSTEM_CACHE_ROOT=/cache \
      -e PLUGIN_ARCHIVE_FORMAT=gzip \
      -e PLUGIN_MOUNT=m2,a \
      -e PLUGIN_RESTORE=false \
      -e PLUGIN_REBUILD=true \
      "$image"

rm -rf my-app/*

echo " ## Restore"
docker run --rm \
      -v "$PWD/my-app":/app \
      -v "$PWD/my-cache":/cache \
      -w /app \
      -e DRONE_REPO=octocat/hello-world \
      -e DRONE_REPO_BRANCH=master \
      -e DRONE_COMMIT_BRANCH=master \
      -e PLUGIN_BACKEND=filesystem \
      -e PLUGIN_CACHE_KEY=volume \
      -e PLUGIN_FILESYSTEM_CACHE_ROOT=/cache \
      -e PLUGIN_ARCHIVE_FORMAT=gzip \
      -e PLUGIN_MOUNT=m2,a \
      -e PLUGIN_RESTORE=true \
      -e PLUGIN_REBUILD=false \
      "$image"

ls -lR my-app
stat my-app/a

Expected behavior

After the restore at least files should have the same modification times as before rebuild. Possibly also the directories, but I'm not sure if that is mandatory or even simple.

Desktop (please complete the following information):

Additional context

I looked into a fix. It currently works for files, but not directories (duh, will be updated by the first extracted file). Also, I don't have tests yet.

hikerspath commented 2 years ago

Confirmed issue present in v1.3.0 (latest) since git describe --dirty returned v1.2.2-25-g44c16f4:

$ ./script
total 0
drwxr-xr-x  3 brianburnett  staff   96 Dec 12  2022 a
drwxr-xr-x  4 brianburnett  staff  128 Aug  9 15:46 m2

my-app/a:
total 0
drwxr-xr-x  3 brianburnett  staff  96 Dec 12  2022 b

my-app/a/b:
total 0
drwxr-xr-x  4 brianburnett  staff  128 Dec 12  2022 c

my-app/a/b/c:
total 0
-rw-r--r--  1 brianburnett  staff  0 Dec 12  2022 bar.txt
-rw-r--r--  1 brianburnett  staff  0 Dec 12  2022 foo.txt

my-app/m2:
total 0
-rw-r--r--  1 brianburnett  staff  0 Jan  1  2022 my-test-file1.txt
-rw-r--r--  1 brianburnett  staff  0 Feb  2  2022 my-test-file2.txt
16777221 95374982 drwxr-xr-x 3 brianburnett staff 0 96 "Dec 12 00:00:00 2022" "Dec 12 00:00:00 2022" "Aug  9 15:46:44 2022" "Aug  9 15:46:44 2022" 4096 0 0 my-app/a
 ## Rebuild
level=info name=drone-cache ts=2022-08-09T19:46:44.422583Z caller=main.go:492 version=dev commit=none date=unknown
level=warn name=drone-cache ts=2022-08-09T19:46:44.422731Z caller=backend.go:78 component=plugin msg="using filesystem as backend"
level=info name=drone-cache ts=2022-08-09T19:46:44.422918Z caller=rebuilder.go:40 component=plugin component=rebuilder msg="rebuilding cache"
level=info name=drone-cache ts=2022-08-09T19:46:44.422965Z caller=metadata.go:48 component=plugin msg="using provided cache key template"
level=info name=drone-cache ts=2022-08-09T19:46:44.425187Z caller=rebuilder.go:74 component=plugin component=rebuilder msg="rebuilding cache for directory" local=m2 remote=volume/m2
level=info name=drone-cache ts=2022-08-09T19:46:44.426027Z caller=rebuilder.go:125 component=plugin component=rebuilder msg="uploading archived directory" local=/app/m2 remote=volume/m2
level=info name=drone-cache ts=2022-08-09T19:46:44.428528Z caller=rebuilder.go:74 component=plugin component=rebuilder msg="rebuilding cache for directory" local=a remote=volume/a
level=info name=drone-cache ts=2022-08-09T19:46:44.428593Z caller=rebuilder.go:125 component=plugin component=rebuilder msg="uploading archived directory" local=/app/a remote=volume/a
level=info name=drone-cache ts=2022-08-09T19:46:44.428731Z caller=rebuilder.go:113 component=plugin component=rebuilder msg="archiving directory" src=/app/m2
level=info name=drone-cache ts=2022-08-09T19:46:44.430172Z caller=rebuilder.go:113 component=plugin component=rebuilder msg="archiving directory" src=/app/a
level=info name=drone-cache ts=2022-08-09T19:46:44.459193Z caller=rebuilder.go:93 component=plugin component=rebuilder msg="cache built" took=36.229ms
 ## Restore
level=info name=drone-cache ts=2022-08-09T19:46:45.192766Z caller=main.go:492 version=dev commit=none date=unknown
level=warn name=drone-cache ts=2022-08-09T19:46:45.193018Z caller=backend.go:78 component=plugin msg="using filesystem as backend"
level=info name=drone-cache ts=2022-08-09T19:46:45.193135Z caller=restorer.go:37 component=plugin component=restorer msg="restoring  cache"
level=info name=drone-cache ts=2022-08-09T19:46:45.193235Z caller=metadata.go:48 component=plugin msg="using provided cache key template"
level=info name=drone-cache ts=2022-08-09T19:46:45.193401Z caller=restorer.go:55 component=plugin component=restorer msg="restoring directory" local=m2 remote=volume/m2
level=info name=drone-cache ts=2022-08-09T19:46:45.193442Z caller=restorer.go:55 component=plugin component=restorer msg="restoring directory" local=a remote=volume/a
level=info name=drone-cache ts=2022-08-09T19:46:45.193542Z caller=restorer.go:96 component=plugin component=restorer msg="extracting archived directory" remote=volume/a local=a
level=info name=drone-cache ts=2022-08-09T19:46:45.193605Z caller=restorer.go:87 component=plugin component=restorer msg="downloading archived directory" remote=volume/a local=a
level=info name=drone-cache ts=2022-08-09T19:46:45.1938Z caller=restorer.go:96 component=plugin component=restorer msg="extracting archived directory" remote=volume/m2 local=m2
level=info name=drone-cache ts=2022-08-09T19:46:45.193885Z caller=restorer.go:87 component=plugin component=restorer msg="downloading archived directory" remote=volume/m2 local=m2
level=error name=drone-cache ts=2022-08-09T19:46:45.199941Z caller=main.go:608 err="[IMPORTANT] restore cache, restore failed, 2 errors: download from <volume/a> to <a>, extract files from downloaded archive, pipe reader failed, extract regular file, open extracted file for writing <a/b/c/bar.txt>, open a/b/c/bar.txt: no such file or directory;\ndownload from <volume/m2> to <m2>, extract files from downloaded archive, pipe reader failed, extract regular file, open extracted file for writing <m2/my-test-file1.txt>, open m2/my-test-file1.txt: no such file or directory\n"
total 0
stat: my-app/a: stat: No such file or directory