Use Lambda@Edge for dataset downloads

[ebezzi]

Motivation

When datasets are withdrawn by submitters, we need the ability to signal that the dataset is "gone" (HTTP 410) rather than not found (HTTP 404).

Approach

We can leverage Lambda@Edge to customize the dataset download experience that is currently being handled by Cloudfront. For example, trying to download a withdrawn datasets will now always return a 404 due to the artifacts being removed. With Lambda@Edge we could query our REST API (the datasets endpoint) and return a 410.

[metakuni]

Closing this for now.

Note, returning a 410 HTTP response code for withdrawn datasets is still desired, but we will revisit when we address this:

• There must be a policy to address withdrawn data present in CELLxGENE "derivative works"#381