Crates.io meeting agenda 2019-10-31 20:00 UTC (Discord, 30 min)

rust-lang / crates-io-cargo-teams

the home of the crates io team

5 stars 11 forks source link

Crates.io meeting agenda 2019-10-31 20:00 UTC (Discord, 30 min) #56

Closed pietroalbini closed 4 years ago

pietroalbini commented 4 years ago

Please edit to add agenda items, or comment if unable to edit.

Setup a S3 lifecycle policy for the database dumps. (pietro)
Use deploy keys to commit to the index instead of access tokens. (pietro)
Discuss updating Ember, and ember-data in particular (@jtgeibel)
Legal takedown requests and rust-lang/crates.io#1871 (@smarnach)
Change the meeting time not to be during EU evening? (pietro)

carols10cents commented 4 years ago

* Setup a S3 lifecycle policy for the database dumps. (pietro)

@pietroalbini I think we're overwriting the same db-dump.tar.gz file every time we take a new snapshot? Does s3 keep previous versions or something like that?

pietroalbini commented 4 years ago

@carols10cents yeah indeed, the whole crates-io bucket is configured to keep previous versions of each file indefinitely. This is something we definitely want for most of the repository (to prevent issues with accidental deletions of .crate files), but we should configure auto-deletion after a while for that file.

smarnach commented 4 years ago

There is another, orthogonal problem with the database dumps. We don't set any cache-related headers for them, so CloudFront will cache them for 24 hours by default. If the first download via a particular CloudFront edge cache happens shortly before the new dump becomes available, all requests via this edge server will see the old version for another 24 hours.

Do we need to sovle this in the uploader code (by adding an Expires header), or is it possible to somehow configure this in the bucket settings as well?

pietroalbini commented 4 years ago

Also added "Use deploy keys to commit to the index instead of access tokens".

There is another, orthogonal problem with the database dumps. We don't set any cache-related headers for them, so CloudFront will cache them for 24 hours by default. If the first download via a particular CloudFront edge cache happens shortly before the new dump becomes available, all requests via this edge server will see the old version for another 24 hours.

Do we need to sovle this in the uploader code (by adding an Expires header), or is it possible to somehow configure this in the bucket settings as well?

I'll look at what I can do on the CloudFront side.