Timebox DRS requests #5536

Closed achave11-ucsc closed 9 months ago

… and return an explicit 503 with a body that blames the DRS service.

[ ] Security design review completed; the Resolution of this issue does not …
- [ ] … affect authentication; for example:
- OAuth 2.0 with the application (API or Swagger UI)
- Authentication of developers with Google Cloud APIs
- Authentication of developers with AWS APIs
- Authentication with a GitLab instance in the system
- Password and 2FA authentication with GitHub
- API access token authentication with GitHub
- Authentication with
- [ ] … affect the permissions of internal users like access to
- Cloud resources on AWS and GCP
- GitLab repositories, projects and groups, administration
- an EC2 instance via SSH
- GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
- [ ] … affect the permissions of external users like access to
- TDR snapshots
- [ ] … affect permissions of service or bot accounts
- Cloud resources on AWS and GCP
- [ ] … affect audit logging in the system, like
- adding, removing or changing a log message that represents an auditable event
- changing the routing of log messages through the system
- [ ] … affect monitoring of the system
- [ ] … introduce a new software dependency like
- Python packages on PYPI
- Command-line utilities
- Docker images
- Terraform providers
- [ ] … add an interface that exposes sensitive or confidential data at the security boundary
- [ ] … affect the encryption of data at rest
- [ ] … require persistence of sensitive or confidential data that might require encryption at rest
- [ ] … require unencrypted transmission of data within the security boundary
- [ ] … affect the network security layer; for example by
- modifying, adding or removing firewall rules
- modifying, adding or removing security groups
- changing or adding a port a service, proxy or load balancer listens on
[ ] Documentation on any unchecked boxes is provided in comments below

CloudWatch Logs Insights
region: us-east-1
log- group-names: /aws/lambda/azul-service-prod
start-time: 2023-09-14T18:22:34.736Z
end-time: 2023-09-14T18:57:56.842Z
query-string:

fields @timestamp, @message, @log
| filter @message like /5613c7f8/
| sort @timestamp asc
| limit 20

@timestamp	@message	@log
2023-09-14 18:42:03.568	START RequestId: 5613c7f8-76f9-40d2-9350-4085108a020b Version: $LATEST	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:03.568	[INFO] 2023-09-14T18:42:03.568Z 5613c7f8-76f9-40d2-9350-4085108a020b azul.chalice Received GET request for '/repository/files/49c1e40c-312d-4491-8847-8805e85c1284', with {"query": {"catalog": "dcp30", "version": "2022-03-25T22:44:27.872000Z"}, "headers": {"accept": "/", "accept-encoding": "deflate, gzip", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-asn": "559", "cloudfront-viewer-country": "CH", "host": "service.azul.data.humancellatlas.org", "user-agent": "curl/7.29.0", "via": "1.1 27fe4928fb2081beeb65fdbf439bb418.cloudfront.net (CloudFront)", "x-amz-cf-id": "QlrU4KdoXnRd6uKgR7x1XEJ4QXU9glbQ4XSXTxdYIYva3LT-3RS-Fg==", "x-amzn-trace-id": "Root=1-650353fb-20a9284a7c6f9d0b3ac835db", "x-forwarded-for": "REDACTED, 64.252.134.151", "x-forwarded-port": "443", "x-forwarded-proto": "https"}}.	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:03.569	[INFO] 2023-09-14T18:42:03.568Z 5613c7f8-76f9-40d2-9350-4085108a020b azul.chalice Did not authenticate request.	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:03.594	[INFO] 2023-09-14T18:42:03.594Z 5613c7f8-76f9-40d2-9350-4085108a020b elasticsearch POST https://vpc-azul-index-prod-mjluewlvxikvhillefxotw2tbm.us-east-1.es.amazonaws.com:443/azul_v2_prod_dcp30_files_aggregate/_search [status:200 request:0.017s]	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:03.596	[INFO] 2023-09-14T18:42:03.596Z 5613c7f8-76f9-40d2-9350-4085108a020b azul.drs GET https://data.terra.bio/ga4gh/drs/v1/objects/v1_24776f45-5584-4456-87d3-71bcea97cd54_90dd4349-6bce-4f40-8ca0-8caa1784ab12 ...	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:34.606	2023-09-14T18:42:34.606Z 5613c7f8-76f9-40d2-9350-4085108a020b Task timed out after 31.04 seconds	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:34.606	END RequestId: 5613c7f8-76f9-40d2-9350-4085108a020b	542754589326:/aws/lambda/azul-service-prod
2023-09-14 18:42:34.606	REPORT RequestId: 5613c7f8-76f9-40d2-9350-4085108a020b Duration: 31037.94 ms Billed Duration: 31000 ms Memory Size: 2048 MB Max Memory Used: 218 MB	542754589326:/aws/lambda/azul-service-prod

I'll take this on, together with #5720.

For demo, find a 503 response from /repository/files and show that the cause was a request to TDR's DRS implementation timing out. Show that the response body contains the domain name of TDR.

DataBiosphere / azul

Timebox DRS requests #5536