existing_remote_files to keep X previous builds

[nitsujri]

Feature Request/Discussion

Keep a number of previous build's files on hand for things like rollback and zero downtime.

Current Solution

existing_remote_files = 'delete' will immediately delete old assets after uploading new ones.

Problems of Current Solution

Our deploy process uploads the new assets but it takes about 2 minutes before the first container can rotate and 6 minutes before the last one gets changed out based on health.

This potentially leaves old containers referencing deleted assets.

Benefits of Proposed Solution

• No downtime between syncing assets and rotation of code/containers.
• Ability to rollback to previous builds/old-assets.

Thanks for this great gem!

[PikachuEXE]

How about adding a task/API to let you delete unused assets Then you can use existing_remote_files = 'keep' and delete unused assets afterward And you can delete unused assets whenever you like https://github.com/AssetSync/asset_sync/blob/cc9d971f1288c851720e9c407e5152bc29b4b214/lib/asset_sync/storage.rb#L318

[nitsujri]

How about adding a task/API to let you delete unused assets Then you can use existing_remote_files = 'keep' and delete unused assets afterward And you can delete unused assets whenever you like

Yeah a rake task works. It would need something like RUNS_TO_KEEP=3 to maintain the history for rollbacks.

History could be stored in remote_file_list_cache_file_path?

{ 'TIMESTAMP': [ files... ], 'TIMESTAMP_2': [ files... ] }

[PikachuEXE]

I mean keeping remote files with existing_remote_files = 'keep' And use new rake task / API which calls delete_extra_remote_files when you want to

[nitsujri]

As an automated solution, that will not work for our case. Removing the ability to rollback is not an option we want to lose because we won't know, automatically, when it is safe to delete the rollbacks.

It's the failsafe we never use until that one time.

Currently, we use keep and manually log into a console to run the delete function, which works well enough.

I imagined an automated solution would be ideally be similar to rake assets:clean that could be run with 100% certainty.

[PikachuEXE]

Unless sprockets provide a stable API to return files in N versions Otherwise I doubt we would be supporting that directly

But maybe a new options can be added to be used in delete_extra_remote_files https://github.com/AssetSync/asset_sync/blob/cc9d971f1288c851720e9c407e5152bc29b4b214/lib/asset_sync/storage.rb#L158-L175 Probably a proc that returns a list of file keys/paths when delete_extra_remote_files is called Those keys/paths returned will be removed from from_remote_files_to_delete and they won't be deleted Then you can use whatever Sprockets documented or not documented methods to generate that list (and you are in charged of updating the proc when API updated / broken of course) What do you think?

[nitsujri]

Definitely understood that supporting this feature directly isn't something we'd want to take on.

Looking at the Sprockets clean and the Webpacker clean, not really accessible API. Solved by: Copy/paste of code... ew?

Anyway, feel free to close this. Thanks!

[PikachuEXE]

If you can come up with a PR or there are more people needing this I can introduce it more quickly Otherwise I am busy at work usually so only have limited time to open source projects And I can't test the function myself coz I don't use this gem anymore OTL

[nitsujri]

Admittedly, we've moved off of this gem a while ago partly for this reason. A consideration was either going back to storing assets in the container itself or develop this feature.

Sadly, with it being simpler to go back to container based assets, we won't be coming up with a PR.

I did want to make the suggestion on the off chance there was interest and it could help others.

[PikachuEXE]

I did moved away from this gem to container hosted, CDN cached structure too

[PikachuEXE]

OK unless there are more people having this issue I am closing this 😃

[summera]

@nitsujri we were in a similar situation since we perform rolling deploys. We started with asset_sync and took a lot of inspiration from it and then ended up rolling our own tool. We just open sourced it - https://github.com/Loomly/s3_asset_deploy. Hopefully it's helpful!