nienbo / cache-buildkite-plugin

Tarball, Rsync & S3 Cache Kit for Buildkite. Supports Linux, macOS and Windows
https://buildkite.com/plugins
MIT License
67 stars 39 forks source link

[FEATURE REQUEST] Always upload cache #46

Closed calvinbui closed 1 year ago

calvinbui commented 2 years ago

There are cache folders generated by NextJS that we can't set a key for. They use external dependencies that may have changed, but we don't have control over that. Could you add the ability to always upload the cache?

gencer commented 2 years ago

I'm a bit lost here. What do you mean by we can't set a key for and always upload the cache? Any example use case would be helpful.

calvinbui commented 2 years ago

Example: building a NextJS project creates a .next/cache folder. The folder doesn't exist beforehand so I can't determine its checksum.

This folder's contents can change between builds depending on the CMS being used (i.e. photos, text, blog posts, etc.). This means it's not realistic to do a checksum on the folder.

AFAIK this plugin does not upload the cache if it already exists or there aren't any changes to the cache key? The request I have is to have an option to always push the cache because in this scenario it will have changed.

gencer commented 2 years ago

Now I got it. I would almost suggest you to use artifacts but this cache scenario will not fit for artifacts. Yes, we can skip the key and check and add an another key something like:

steps:
  - plugins:
    - gencer/cache#v2.4.10:
        id: nextjs
        backend: s3
        key: "v1-cache-{{ id }}-{{ runner.os }} # no checksum...
        # no "restore-keys" needed as the newest one will be pulled in this scenario.
        always: true # this is your key so that upload/download will be always made no matter of what (e.g.: checksum+key not exists)
        s3:
          # ...
        paths:
          - '.next/cache'

P.S.: I was also thinking to use NextJS in next project and this will also helpful for me too. Thanks for the heads up!

calvinbui commented 2 years ago

that didn't seem to work, it's still skipping over the cache at the end

- gencer/cache#v2.4.10:
    id: nextjs
    backend: s3
    key: "ap3-website-ui-{{ id }}-{{ runner.os }}"
    always: true
    s3:
      # ...
    paths:
      - build/cache # we changed the default

image

gencer commented 2 years ago

@calvinbui I am sorry for not being clear. Please accept my apologies. I did not push the changes yet (it is on local branch) I'll push to master soon. That example is just to give you whats going to be implemented. (Something like prototype)

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

gencer commented 2 years ago

Not stale.

obendidi commented 2 years ago

+1 for this, having the same use case where I don't depend on a checksum to upload cache, and I would ideally like it to always be updated

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

calvinbui commented 2 years ago

Not stale.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

calvinbui commented 2 years ago

not stale

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

calvinbui commented 2 years ago

not stale

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

calvinbui commented 2 years ago

not stale

@gencer you mentioned you already wrote the code for this?

calvinbui commented 2 years ago

not stale