Sort out Last-Modified and ETag issues on data.everef.net

[kennethjor]

Backblaze B2 provides the last-modified header as the current time. The etag isn't there at all. Currently, Cloudflare rewrite rules are used to provide these from various custom headers provided.

• Refactor the S3 upload to provide a standard header for both of these values. :point_left: relying on the rclone standard instead.
  • The header value should be an HTTP date for easy use in header rewrites. not possible without replacing + characters.
• Refactor data index to use these headers (#323)
• Data time sync script to modify all the present files to add these headers. should be needed
  • During the migration from S3, some original timestamp headers were added by the migration script, use these instead of the timestamp on the object.
  • Maintain S3 and B2 headers as well in order to make it compatible with rclone.
• [ ] Find out how to standardise ETag headers.

Split out:

• 322

• 323

• 324

rclone

From https://rclone.org/b2/:

The modified time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard. Other tools should be able to use this as a modified time.

From https://rclone.org/s3/:

The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.

[kennethjor]

When you set custom metadata on B2, like everef-http-last-modified: Wed 21 Oct 2015 07:28:00 GMT is comes through as x-bz-info-x-everef-http-last-modified: Wed+21+Oct+2015+07:28:00+GMT. In order to use the regex_replace function on Cloudflare to remove the plus signs, you have to be on a business plan, which is $200/month :disappointed:

[kennethjor]

Cloudflare Worker method:

• https://www.backblaze.com/docs/cloud-storage-deliver-private-backblaze-b2-content-through-cloudflare-cdn
  • https://github.com/backblaze-b2-samples/cloudflare-b2
• https://developers.cloudflare.com/workers/wrangler/

[kennethjor]

There's also Wasabi at $6.99/TB-month and Cloudflare R2 at $15/TB-month. Both offer zero-egress and S3-compatible APIs. Wasabi doesn't charge for API calls. Before writing a function for making Backblaze work properly, I should evaluate these two and see if their caching headers are useful.

[kennethjor]

Wasabi does provider proper Last-Modified and ETag headers, but like the other providers, it's not possible to force the modification timestamp. For that, we have to rely on headers. That means a Cloudflare function is required anyway.

Wasabi also seems to respond appropriately to If-None-Match and If-Modified-Since headers, which Backblaze doesn't.

Moving all the data to Wasabi, combined with a Cloudflare function to just control modification times, letting Wasabi deal with the rest, may be a good approach.

How do I use Cloudflare with Wasabi?

[kennethjor]

Made some progress with a test deployment on Wasabi. It looks like the etag header is being stripped for some reason though:

curl --head https://data2.everef.net/ccp/mer/2021/EVEOnline_MER_Aug2021.zip
HTTP/2 200 
date: Sat, 02 Mar 2024 23:14:02 GMT
content-type: application/zip
content-length: 31836430
etag: ""
last-modified: Sat, 02 Mar 2024 23:11:36 GMT

[kennethjor]

Finished the Cloudflare Workers script for accessing Wasabi. However, now that I have the script, I might as well point it at Backblaze and forego a lengthy migration.

The last-modified header is working, but there's still an issue with stripped etags:

curl --head https://data2.everef.net/ccp/mer/2021/EVEOnline_MER_Aug2021.zip
HTTP/2 200 
date: Sun, 10 Mar 2024 07:06:14 GMT
content-type: application/zip
content-length: 31836430
etag: ""
last-modified: Tue, 07 Sep 2021 16:28:00 GMT