nienbo / cache-buildkite-plugin

Tarball, Rsync & S3 Cache Kit for Buildkite. Supports Linux, macOS and Windows
https://buildkite.com/plugins
MIT License
67 stars 40 forks source link

Multiple file paths for key caching #45

Open lewis785 opened 2 years ago

lewis785 commented 2 years ago

Is it possible to use multiple files for the checksum? I am trying to add caching to a repository that has multiple sub packages and would like the cache to update if any of the package-lock files were change.

Example:

node-cache: &node-cache
    id: node
    key: "v1-cache-{{ id }}-{{ runner.os }}-{{ checksum 'package-lock.json' 'src/example-1/package-lock.json' 'src/example-2/package-lock.json' }}"
    restore-keys:
        - 'v1-cache-{{ id }}-{{ runner.os }}-'
        - 'v1-cache-{{ id }}-'
    paths:
        - node_modules
        - src/**/node_modules

The cache seems to still work but says that it is unable to find the files.

Result:

find: ‘package-lock.json src/example-1/package-lock.json src/example-2/package-lock.json ’: No such file or directory
gencer commented 2 years ago

At the moment, you can either;

Multiple files like you described in your example will not work that way. But this can be implemented.

gencer commented 2 years ago

Hey @lewis785,

I'm doing my own tests on my end. What do you think about the following approaches:

Comma-separated:

node-cache: &node-cache
    id: node
    key: "v1-cache-{{ id }}-{{ runner.os }}-{{ checksum 'package-lock.json,src/example-1/package-lock.json,src/example-2/package-lock.json' }}"

Glob-pattern:

node-cache: &node-cache
    id: node
    key: "v1-cache-{{ id }}-{{ runner.os }}-{{ checksum '**/package-lock.json' }}"

P.S.: This may be miss the main /package-lock.json and only include sub-directories.

Key-based:

node-cache: &node-cache
    id: node
    key: "v1-cache-{{ id }}-{{ runner.os }}-{{ checksum }}" # if no data given, checksum key below will be used:
    checksum: # list of files to get cheksums
      - package-lock.json
      - **/package-lock.json # and glob pattern! or maybe full list of single files...

It seems Key-based is looking good and more fit to the structure.

Any ideas?

lewis785 commented 2 years ago

I think the key based approach is the best, it would provide people plenty of flexability. Additionally it keeps it consistent with how multiple paths are defined.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

gencer commented 2 years ago

Not stale.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

gencer commented 2 years ago

Not stale

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

gencer commented 2 years ago

not stale

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

gencer commented 2 years ago

Not stale