tsibley
commented
5 months ago This is certainly odd behaviour. Off the bat, I can't say with certainty what the cause might be. It's certainly able to be understood, but it's the kind of situation where being able to reproduce and poke at the behaviour is immensely useful to gaining that understanding. It would also be useful to have seen the server's logs resulting from running your test script, as they'll show caching information.
I initially uploaded them using
aws s3 cp(so the data wasn't zipped), and then re-uploaded them usingnextstrain remote upload ...(which magically zips the data).
Between these two uploads, did you access the dataset via the nextstrain.org server?
jameshadfield
As part of writing up this issue I fixed it, but it's still really weird to me, and I'd like to understand why this happened.
Description of bug
As part of #811 I uploaded new datasets to our staging bucket, by downloading then uploading the flu/seasonal/h3n2/ha/2y JSONs and renaming them to seasonal-flu, so we could update our tests to use the latter. I initially uploaded them using
aws s3 cp(so the data wasn't zipped), and then re-uploaded them usingnextstrain remote upload ...(which magically zips the data). The strangeness is to do with the associated content-encoding metadata and/or the actual encoding of the file. The file is gzipped (as it's 1.1MB, and uncompressed it's 37MB), and the content encoding is gzip (can see this in AWS console).Using a small script to access this file (cf. the staging/zika dataset) via various APIs gives me weird results.
Test script
Run the server locally at the default port 5000 and then run the following script: ```js import fetch from 'node-fetch'; async function main() { const paths = [ "staging/seasonal-flu/h3n2/ha/2y", "staging/zika", ] /* Make a GET request to our REST API asking for JSON data */ console.log("\nREST API") for (const path of paths) { const mediaType = "application/json" const url = `http://localhost:5000/${path}` console.log("\tfetch:", url, mediaType) const req = fetch(url, accept(mediaType)); const res = await req; console.log("\t\tresponse content encoding:", res.headers.get('content-encoding')) /* print out the start of the response. Note that this is after decompression, if any, and I don't know how to truly know if decompression is happening. If content-encoding is gzip then does this mean the response has for sure been decompressed? */ const text = await res.text(); console.log("\t\tResponse:", text.slice(0,50)); } /* now do the same but hitting the Charon endpoint */ console.log("\nCharon API") for (const path of paths) { const url = `http://localhost:5000/charon/getDataset?prefix=${path}` console.log("\tfetch:", url) const req = fetch(url) const res = await req; console.log("\t\tresponse content encoding:", res.headers.get('content-encoding')) const text = await res.text(); console.log("\t\tResponse:", text.slice(0,50)); } /* for good measure hit the HTTPS S3 endpoints */ console.log("\nS3 HTTPS ") for (const path of paths) { const key = path.replace(/^staging\//, "").replace(/\//g, '_') + ".json" const url = `https://nextstrain-staging.s3.amazonaws.com/${key}`; console.log("\tfetch:", url) const req = fetch(url) const res = await req; console.log("\t\tresponse content encoding:", res.headers.get('content-encoding')) const text = await res.text(); console.log("\t\tResponse:", text.slice(0,50).replace(/\n/g, "When I run this script locally I get:
The weirdness (for me) is:
requests.test.jsand this test passes on GitHub actions but not locally. This is what led me down this rabbit hole, and hinted at a caching issue.I have a mad hunch that the server's reported
content-encoding: gzipfor the REST API response is somehow corrupted, and the script'sfetchcall is not respecting it, but I don't understand the mechanics well enough.The fix
The following patch, which creates a new (empty) cache for the server's
fetchmachinery, fixed it (response content encoding: gzip for all responses, and correct JSON contents):Note: keeping the old cache but changing to
cache: "no-store", orcache: "no-cache", orcache: "reload"did not fix the bug.Questions
Why was the cache we are using not detecting that the cached data was now invalid? We set
cache: no-cacheon the request so the response must be validated with the origin server before each reuse, but it seems we didn't pick up that the contents/metadata had changed. (I mean, is this what happened? Clearly the caching used byfetch(make-fetch-happen) is at fault here somehow.)Why was the content-encoding different on the charon (
null) vs RESTful API (gzip) response, and why did this seemingly not affect decompression (or lack thereof) of the response?P.S. Looks like MacOS will remove
/tmpfiles after 3 days, so there were a few different things that lined up in order for me to crash into this bug.