Can't use curl to download a single manifest in one invocation

dsotirho-ucsc commented 7 months ago

Before switching to PUT, it was possible to download a manifest in a single curl invocation by simply telling it to follow redirects using -L. Now one has to specify -X 'PUT' or -T /dev/null and in either case the redirect is followed with PUT not GET which leads to a 403 {"message":"Missing Authentication Token"}

daniel@Crispin ~/temp $ curl -L -X 'PUT' -v 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C11000000414%5D%5D%7D%7D&format=curl'
*   Trying 99.84.238.165:443...
* Connected to service.daniel.dev.singlecell.gi.ucsc.edu (99.84.238.165) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=service.daniel.dev.singlecell.gi.ucsc.edu
*  start date: Jun 29 00:00:00 2023 GMT
*  expire date: Jul 28 23:59:59 2024 GMT
*  subjectAltName: host "service.daniel.dev.singlecell.gi.ucsc.edu" matched cert's "service.daniel.dev.singlecell.gi.ucsc.edu"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C11000000414%5D%5D%7D%7D&format=curl
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: service.daniel.dev.singlecell.gi.ucsc.edu]
* [HTTP/2] [1] [:path: /manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C11000000414%5D%5D%7D%7D&format=curl]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> PUT /manifest/files?catalog=dcp3&filters=%7B%22fileSize%22%3A%7B%22within%22%3A%5B%5B1%2C11000000414%5D%5D%7D%7D&format=curl HTTP/2
> Host: service.daniel.dev.singlecell.gi.ucsc.edu
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 301
< content-type: application/json
< content-length: 4
< location: https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8QgLKO1yr0HsVFEcekfZzNcif0wj_m0y_1vkKyl3WasQnoAAQ==
< date: Wed, 31 Jan 2024 20:48:16 GMT
< x-amzn-requestid: c4888d8d-7f75-4899-9b19-54186c1e0904
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubDomains
< access-control-allow-headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
< x-amz-apigw-id: SbDClHhloAMEN-w=
< retry-after: 1
< x-amzn-trace-id: Root=1-65bab210-0f5a1d9d595c98b66d9b118c;Parent=48c9169d8c34a1f4;Sampled=0;lineage=18704a40:0
< x-cache: Miss from cloudfront
< via: 1.1 af9d66efe7802df1efbc8106c86a13e6.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO5-C3
< x-amz-cf-id: QtF-u-pe448mN82HaPS40gvJaA_MRwBqUixb4K_Dq0YsLKFKzctrSg==
<
* Ignoring the response-body
* Connection #0 to host service.daniel.dev.singlecell.gi.ucsc.edu left intact
* Issue another request to this URL: 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8QgLKO1yr0HsVFEcekfZzNcif0wj_m0y_1vkKyl3WasQnoAAQ=='
* Found bundle for host: 0x600002e4c090 [can multiplex]
* Re-using existing connection with host service.daniel.dev.singlecell.gi.ucsc.edu
* [HTTP/2] [3] OPENED stream for https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8QgLKO1yr0HsVFEcekfZzNcif0wj_m0y_1vkKyl3WasQnoAAQ==
* [HTTP/2] [3] [:method: PUT]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: service.daniel.dev.singlecell.gi.ucsc.edu]
* [HTTP/2] [3] [:path: /manifest/files/k8QgLKO1yr0HsVFEcekfZzNcif0wj_m0y_1vkKyl3WasQnoAAQ==]
* [HTTP/2] [3] [user-agent: curl/8.4.0]
* [HTTP/2] [3] [accept: */*]
> PUT /manifest/files/k8QgLKO1yr0HsVFEcekfZzNcif0wj_m0y_1vkKyl3WasQnoAAQ== HTTP/2
> Host: service.daniel.dev.singlecell.gi.ucsc.edu
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 403
< content-type: application/json
< content-length: 42
< date: Wed, 31 Jan 2024 20:48:17 GMT
< x-amzn-requestid: 4b68cced-8656-448e-ad27-17fd87cd8c5f
< x-amzn-errortype: MissingAuthenticationTokenException
< x-amz-apigw-id: SbDCsE2soAMEIeA=
< x-cache: Error from cloudfront
< via: 1.1 af9d66efe7802df1efbc8106c86a13e6.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO5-C3
< x-amz-cf-id: 4oU5xe9RxzEDxOuRumfqJEQd58mQ6FyUUTCcjVPhglZDJX_1mYhU8g==
<
* Connection #0 to host service.daniel.dev.singlecell.gi.ucsc.edu left intact
{"message":"Missing Authentication Token"}

[ ] Security design review completed; the Resolution of this issue does not …
- [ ] … affect authentication; for example:
- OAuth 2.0 with the application (API or Swagger UI)
- Authentication of developers with Google Cloud APIs
- Authentication of developers with AWS APIs
- Authentication with a GitLab instance in the system
- Password and 2FA authentication with GitHub
- API access token authentication with GitHub
- Authentication with
- [ ] … affect the permissions of internal users like access to
- Cloud resources on AWS and GCP
- GitLab repositories, projects and groups, administration
- an EC2 instance via SSH
- GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
- [ ] … affect the permissions of external users like access to
- TDR snapshots
- [ ] … affect permissions of service or bot accounts
- Cloud resources on AWS and GCP
- [ ] … affect audit logging in the system, like
- adding, removing or changing a log message that represents an auditable event
- changing the routing of log messages through the system
- [ ] … affect monitoring of the system
- [ ] … introduce a new software dependency like
- Python packages on PYPI
- Command-line utilities
- Docker images
- Terraform providers
- [ ] … add an interface that exposes sensitive or confidential data at the security boundary
- [ ] … affect the encryption of data at rest
- [ ] … require persistence of sensitive or confidential data that might require encryption at rest
- [ ] … require unencrypted transmission of data within the security boundary
- [ ] … affect the network security layer; for example by
- modifying, adding or removing firewall rules
- modifying, adding or removing security groups
- changing or adding a port a service, proxy or load balancer listens on
[ ] Documentation on any unchecked boxes is provided in comments below

dsotirho-ucsc commented 7 months ago

Assignee to consider next steps.

hannes-ucsc commented 6 months ago

Spike to try -T /dev/null instead of -X PUT .

dsotirho-ucsc commented 5 months ago

Spike to try -T /dev/null instead of -X PUT .

This resulted in a PUT request with a 301 response.

(.venv) daniel@Crispin ~/repo/azul2 $ curl -v -T '/dev/null' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201000000011%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl'
*   Trying 99.84.238.113:443...
* Connected to service.daniel.dev.singlecell.gi.ucsc.edu (99.84.238.113) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=service.daniel.dev.singlecell.gi.ucsc.edu
*  start date: Jun 29 00:00:00 2023 GMT
*  expire date: Jul 28 23:59:59 2024 GMT
*  subjectAltName: host "service.daniel.dev.singlecell.gi.ucsc.edu" matched cert's "service.daniel.dev.singlecell.gi.ucsc.edu"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201000000011%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: service.daniel.dev.singlecell.gi.ucsc.edu]
* [HTTP/2] [1] [:path: /manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201000000011%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> PUT /manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201000000011%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl HTTP/2
> Host: service.daniel.dev.singlecell.gi.ucsc.edu
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 301
< content-type: application/json
< content-length: 4
< location: https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8QgHDI5fWBVRatUzWS9qf4EK5E9RgQ3UMlSe-Hks4MRBIcAAQ==
< date: Thu, 21 Mar 2024 23:42:01 GMT
< x-amzn-requestid: d38aa494-2f99-4201-9b1c-dc4d4e0a253f
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubDomains
< access-control-allow-headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
< x-frame-options: DENY
< x-amz-apigw-id: VAPXcEcLoAMEqhg=
< x-content-type-options: nosniff
< retry-after: 1
< x-amzn-trace-id: Root=1-65fcc5c8-1076bb2232a5ace66fce86f0;Parent=53e003f1a14c6843;Sampled=0;lineage=18704a40:0
< x-cache: Miss from cloudfront
< via: 1.1 45fb7111a175c6978b1378f5881fcaee.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO5-C3
< x-amz-cf-id: ikss2Eg4Bjd3K8ndgmdngUoqrDAC-Jlk3kjnBNy7itD4O4BJpS5xPA==
* HTTP error before end of send, stop sending
<
* Connection #0 to host service.daniel.dev.singlecell.gi.ucsc.edu left intact
null(.venv) daniel@Crispin ~/repo/azul2 $

dsotirho-ucsc commented 5 months ago

Using -T '/dev/null' with -L did not work. The initial request used 'PUT' as desired, however the second request also used 'PUT' resulting in a 403.

daniel@Crispin ~ $ curl -v -L -T '/dev/null' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201111110000%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl'
*   Trying 99.84.238.135:443...
* Connected to service.daniel.dev.singlecell.gi.ucsc.edu (99.84.238.135) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=service.daniel.dev.singlecell.gi.ucsc.edu
*  start date: Jun 29 00:00:00 2023 GMT
*  expire date: Jul 28 23:59:59 2024 GMT
*  subjectAltName: host "service.daniel.dev.singlecell.gi.ucsc.edu" matched cert's "service.daniel.dev.singlecell.gi.ucsc.edu"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201111110000%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: service.daniel.dev.singlecell.gi.ucsc.edu]
* [HTTP/2] [1] [:path: /manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201111110000%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
> PUT /manifest/files?catalog=dcp3&filters=%7B%0A%20%20%22cellCount%22%3A%20%7B%0A%20%20%20%20%22within%22%3A%20%5B%0A%20%20%20%20%20%20%5B%0A%20%20%20%20%20%20%20%201%2C%0A%20%20%20%20%20%20%20%201111110000%0A%20%20%20%20%20%20%5D%0A%20%20%20%20%5D%0A%20%20%7D%0A%7D&format=curl HTTP/2
> Host: service.daniel.dev.singlecell.gi.ucsc.edu
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 301
< content-type: application/json
< content-length: 4
< location: https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ==
< date: Sat, 23 Mar 2024 00:16:41 GMT
< x-amzn-requestid: 054a6434-6eb0-4610-92e1-fd77a317e664
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubDomains
< access-control-allow-headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
< x-frame-options: DENY
< x-amz-apigw-id: VDnYgHgLIAMElQA=
< x-content-type-options: nosniff
< retry-after: 1
< x-amzn-trace-id: Root=1-65fe1f69-26d8c28c2abab25137d85c7f;Parent=69d830715498a759;Sampled=0;lineage=18704a40:0
< x-cache: Miss from cloudfront
< via: 1.1 100e7eca600d702a8613a94cb0899fe8.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO5-C3
< x-amz-cf-id: o9WTqnNBsr37DembN2tod_naZHaLUgkekDHODKVSNgQeANGZA4Yj9g==
* HTTP error before end of send, stop sending
<
* Ignoring the response-body
* Connection #0 to host service.daniel.dev.singlecell.gi.ucsc.edu left intact
* Issue another request to this URL: 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ=='
* Found bundle for host: 0x6000002b8150 [can multiplex]
* Re-using existing connection with host service.daniel.dev.singlecell.gi.ucsc.edu
* [HTTP/2] [3] OPENED stream for https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ==
* [HTTP/2] [3] [:method: PUT]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: service.daniel.dev.singlecell.gi.ucsc.edu]
* [HTTP/2] [3] [:path: /manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ==]
* [HTTP/2] [3] [user-agent: curl/8.4.0]
* [HTTP/2] [3] [accept: */*]
> PUT /manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ== HTTP/2
> Host: service.daniel.dev.singlecell.gi.ucsc.edu
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/2 403
< content-type: application/json
< content-length: 42
< date: Sat, 23 Mar 2024 00:16:41 GMT
< x-amzn-requestid: d19b4b60-51a7-4e9b-86f5-9b16c8f03b11
< x-amzn-errortype: MissingAuthenticationTokenException
< x-amz-apigw-id: VDnYlGLDoAMEQxQ=
< x-cache: Error from cloudfront
< via: 1.1 100e7eca600d702a8613a94cb0899fe8.cloudfront.net (CloudFront)
< x-amz-cf-pop: SFO5-C3
< x-amz-cf-id: HiJJ8UU8bkwUIXS_kC8DZm6LCQc3yKqf_XvIjO3Z49x-XrhjdLhU5w==
* HTTP error before end of send, stop sending
<
* Connection #0 to host service.daniel.dev.singlecell.gi.ucsc.edu left intact
{"message":"Missing Authentication Token"}daniel@Crispin ~ $
daniel@Crispin ~ $

daniel@Crispin ~ $ curl -X 'PUT' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ=='
{"message":"Missing Authentication Token"}daniel@Crispin ~ $
daniel@Crispin ~ $

daniel@Crispin ~ $ curl -X 'GET' 'https://service.daniel.dev.singlecell.gi.ucsc.edu/manifest/files/k8Qgu0WHYIqGlc33bFjQxUZ861cjApimBzV_wN2zkoIFHVkAAQ=='

Download the manifest in cmd.exe with `curl` using:

curl.exe --location --fail "https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-dev-storage-daniel.us-east-1/manifests/1e8f0fb7-bed9-5ff2-b44d-76920544f46f.ad14b32b-dea0-555e-b745-8ebe5bb0958d.curlrc?response-content-disposition=attachment%3Bfilename%3D%22hca-manifest-1e8f0fb7-bed9-5ff2-b44d-76920544f46f.ad14b32b-dea0-555e-b745-8ebe5bb0958d.curlrc%22&AWSAccessKeyId=ASIARZFZ7W77RWXBGBB4&Signature=dvMIfBes%2F3FBI7OAThWGF7VTEbI%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEDkaCXVzLWVhc3QtMSJIMEYCIQDE0jicKhaiK13VXwfFyQ%2F72kSAaop7ThO6CkNTJTq%2FxgIhANWs%2BDOqKeesuNdwJB1JrAnY06Hubwz6CfvGK7KXcyGlKo0DCFEQBBoMMTIyNzk2NjE5Nzc1IgzXdS%2B9lviMR3XZad4q6gLcMCG87TkAHx3gBbRqb%2F9tnlBq2PQBURlKEq0nc4oQG0k0Pg1gTz%2F22Dx9mvscmYKZPcQg1Zo3vcESnvQ9R0BP94PCco3qlnIs5yMnaEPt%2F8l3WjwY6PLqNYCmlNazkLwScTTJYUD7XlhVJiY6unSSTBCrR2roqfhm%2FpbuILpY0G9NScQ29D8S9SOKbPMGN%2FXdZn5wBwBuT4XMpM0Bag0Z33MUlvZcEu5comtSAm79jsdCAR0hsSBSyAl0u1X8jYq%2FApfmkvxovYfL2z5LRs4eVtoWMrqgf%2Bkf3EAdWrpVYit7tGqvEr%2Fbn7GE9FRU2I9pqhYiRhagoe%2BK3wC3PPZ7gWtiTFCc5vdc1qS%2FickulXnX0tIyPZpB%2BxmsuPpq0E0%2FQlV%2BRsminPmWcaclzOP0koXo5G6tFktM3exzycVhzpb8FLZvXu7DlqS8qKwoo7YLB6UoYV2WB6y5vJIJHx6AQk7lZqpltzIK%2BTD%2FvfivBjqcAQ29hh8uMFzAsC0rwf3bT1nw86yqf4g0Mnz%2BPbr%2BzWls1Dmp%2Fs4wjiTSF%2BdBXJd04YNF7llPBodI2KX4kPQkffi3R1IKoy7tfvZhBEU2iKl4OBK7ysmBsYJZ1l5%2FlZud2t6AP78bp4Vrunaihqb5ihbQushsDKXROBy8I6Xp0RXmoAo6IeUBGvdHTTG3QloClUE12Qx3R0B0cJM%2FKg%3D%3D&Expires=1711156604" | curl.exe --fail-early --continue-at - --retry 15 --retry-delay 10 --config -

Download the manifest in bash with `curl` using:

curl --location --fail 'https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-dev-storage-daniel.us-east-1/manifests/1e8f0fb7-bed9-5ff2-b44d-76920544f46f.ad14b32b-dea0-555e-b745-8ebe5bb0958d.curlrc?response-content-disposition=attachment%3Bfilename%3D%22hca-manifest-1e8f0fb7-bed9-5ff2-b44d-76920544f46f.ad14b32b-dea0-555e-b745-8ebe5bb0958d.curlrc%22&AWSAccessKeyId=ASIARZFZ7W77RWXBGBB4&Signature=dvMIfBes%2F3FBI7OAThWGF7VTEbI%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEDkaCXVzLWVhc3QtMSJIMEYCIQDE0jicKhaiK13VXwfFyQ%2F72kSAaop7ThO6CkNTJTq%2FxgIhANWs%2BDOqKeesuNdwJB1JrAnY06Hubwz6CfvGK7KXcyGlKo0DCFEQBBoMMTIyNzk2NjE5Nzc1IgzXdS%2B9lviMR3XZad4q6gLcMCG87TkAHx3gBbRqb%2F9tnlBq2PQBURlKEq0nc4oQG0k0Pg1gTz%2F22Dx9mvscmYKZPcQg1Zo3vcESnvQ9R0BP94PCco3qlnIs5yMnaEPt%2F8l3WjwY6PLqNYCmlNazkLwScTTJYUD7XlhVJiY6unSSTBCrR2roqfhm%2FpbuILpY0G9NScQ29D8S9SOKbPMGN%2FXdZn5wBwBuT4XMpM0Bag0Z33MUlvZcEu5comtSAm79jsdCAR0hsSBSyAl0u1X8jYq%2FApfmkvxovYfL2z5LRs4eVtoWMrqgf%2Bkf3EAdWrpVYit7tGqvEr%2Fbn7GE9FRU2I9pqhYiRhagoe%2BK3wC3PPZ7gWtiTFCc5vdc1qS%2FickulXnX0tIyPZpB%2BxmsuPpq0E0%2FQlV%2BRsminPmWcaclzOP0koXo5G6tFktM3exzycVhzpb8FLZvXu7DlqS8qKwoo7YLB6UoYV2WB6y5vJIJHx6AQk7lZqpltzIK%2BTD%2FvfivBjqcAQ29hh8uMFzAsC0rwf3bT1nw86yqf4g0Mnz%2BPbr%2BzWls1Dmp%2Fs4wjiTSF%2BdBXJd04YNF7llPBodI2KX4kPQkffi3R1IKoy7tfvZhBEU2iKl4OBK7ysmBsYJZ1l5%2FlZud2t6AP78bp4Vrunaihqb5ihbQushsDKXROBy8I6Xp0RXmoAo6IeUBGvdHTTG3QloClUE12Qx3R0B0cJM%2FKg%3D%3D&Expires=1711156604' | curl --fail-early --continue-at - --retry 15 --retry-delay 10 --config -
daniel@Crispin ~ $

hannes-ucsc commented 5 months ago

OK, looks like our options are

to add support for POST as an alternative to PUT and rely on curl emulating legacy UAs by changing from POST to GET when following a 301 redirect or
to keep using PUT but return a 303 (not a 301) which the standards say should be followed with a GET

The latter option is complicated by the fact that it would be backwards incompatible and require CCs cooperation in adjusting the client. They've been reluctant when we asked them to switch from GET to PUT for the initial manifest request. Alternatively, we could sniff the user-agent and tweak the response accordingly. However, curl was only recently modified to switch from PUT to GET on 303.

All in all, I think we should implement the first alternative.

dsotirho-ucsc commented 5 months ago

Assignee to implement first alternative outlined in comment above.

DataBiosphere / azul

Can't use curl to download a single manifest in one invocation #5918