[Bug] Download rsd object is broken sometimes

[saracastel]

Background It seems that download rds is broken for some old experiments, not all.

The issue shows up when trying to download the rds object for an old experiment without going to step 7 of Data Processing first. If get embeddings is called by going to step 7, download rds works fine.

Discussion @kafkasl insights:

• A solution might be to rerun the embedding if it's not found, but the embedding work results expire, which could lead to future failures;
• the issue might be that the api is not generating the correct embedding

@cosa65 insights:

• the UI calls get embedding before downloading RDS, which means the work result should exist when the work request is made, unless somehow the result in s3 gets deleted before the cached one in the browser. However, the expiration time for the cached should be way smaller than the s3;
• the issue could be diagnosed checking the etag sent with the processed RDS against the actual etag of the generated embedding

Goals

• Investigate the issue
• Fix the bug

[alexvpickering]

please don't link to slack as it's not publicly visible -- give a summary

[saracastel]

oh sorry, didn't think about it. Updated with the summary