Song client testing : URL retrieval occasionally works, Manifest download and view not working,

[edsu7]

Findings

• ICGC legacy
  • URL retrieval works depending on object_id, however was unable to run curl/wget on resulting url
  • Download works on object_id (single or #1f) but not on manifest (using same object ids)
  • View/Mount did not work
  • Generally View/Mount/Download issues had the same error of "Amazon S3 error: FORBIDDEN: If you are trying to connect to the AWS S3 Repository, you need to be running ths ICGC client on an EC2 VM instance"

Initialize w/ docker

sudo apt-get update

sudo apt install docker.io
sudo chmod 666 /var/run/docker.sock

docker pull overture/score

Set ENV

legacy_accessToken=

docker run \
    --name test-env \
    -d \
    -u $(id -u):$(id -g) \
    -it \
    -e ACCESSTOKEN=${legacy_accessToken} \
    --mount \
    type=bind,source="$(pwd)",target=/output overture/score

URL retrieval Works but not wget:

docker exec \
test-env \
sh -c \
"bin/score-client url --object-id ddcdd044-adda-5f09-8849-27d6038f8ccd"

docker exec \
test-env \
sh -c \
wget "https://s3-external-1.amazonaws.com/oicr.icgc/data/ddcdd044-adda-5f09-8849-27d6038f8ccd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220217T164337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAIDAITD7JYHCF4URQ%2F20220217%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=bf85a6c8c7c5c2855e0d20b8c4a4e4933d3189472b4e3cda8bf2bdac53eb9396"

Log

--2022-02-17 16:44:44--  https://s3-external-1.amazonaws.com/oicr.icgc/data/ddcdd044-adda-5f09-8849-27d6038f8ccd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220217T164337Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAIDAITD7JYHCF4URQ%2F20220217%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=bf85a6c8c7c5c2855e0d20b8c4a4e4933d3189472b4e3cda8bf2bdac53eb9396
Resolving s3-external-1.amazonaws.com (s3-external-1.amazonaws.com)... 52.216.146.233
Connecting to s3-external-1.amazonaws.com (s3-external-1.amazonaws.com)|52.216.146.233|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-02-17 16:44:44 ERROR 403: Forbidden.

Example of URL retrieval not working :

bin/score-client url --object-id afa653ea-5178-50f2-b146-817b9b044b59

Log

Resolving URL for object: afa653ea-5178-50f2-b146-817b9b044b59                                                                                                                  
ERROR: Command error: java.io.IOException: Storage client error: {"timestamp":"2022-02-17T16:52:58.536+0000","status":404,"error":"Not Found","message":"Entity afa653ea-5178-50f2-b146-817b9b044b59 is not registered on the server.","path":"/download/afa653ea-5178-50f2-b146-817b9b044b59"}

Please check the log for detailed error messages

Download via object-ID

docker exec \
    test-env \
    sh -c \
    "bin/score-client \
    --profile collab \
    download \
    --object-id afa653ea-5178-50f2-b146-817b9b044b59 \
    --output-dir /output"

docker exec \
    test-env \
    sh -c \
    "bin/score-client \
    --profile collab \
    download \
    --object-id 94a513d3-ec31-574e-9ff3-27b091ff0d8d \
    --output-dir /output"

docker exec \
    test-env \
    sh -c \
    "bin/score-client \
    --profile collab \
    download \
    --object-id ddcdd044-adda-5f09-8849-27d6038f8ccd \
    --output-dir /output"

docker exec \
    test-env \
    sh -c \
    "bin/score-client \
    --profile collab \
    download \
    --object-id 99fb9877-8ab2-5763-be33-4250ff84b265\
    --output-dir /output"

Bulk Download via object-ID

docker exec \
    test-env \
    sh -c \
    "bin/score-client \
    --profile collab \
    download \
    --object-id \
    c511b0d5-8bd4-5ff8-ac5a-bbd1f201d480 \
    b2f70bd4-dfd1-5584-8a7a-bdaef1ef0276 \
    5f5e5c24-6ae5-5711-85f3-7c00537dbe3a \
    b261ac56-b5fa-568a-87c2-a7119ccbffb7 \
    --output-dir /output"

Query for manifest

docker exec test-env sh -c "bin/score-client manifest --manifest 4be4e654-c402-4cdb-a875-53f5f5b26cdf"

View / Manifest Download

docker exec test-env sh -c "bin/score-client view --object-id ace274bb-059e-55f2-875d-56c18705fe41 --query 1:1-10000"
docker exec test-env sh -c "bin/score-client download --manifest 4be4e654-c402-4cdb-a875-53f5f5b26cdf --output-dir /output"

Log (same error shared by both cmds)

ERROR: Command error: java.io.IOException: Amazon S3 error: FORBIDDEN: If you are trying to connect to the AWS S3 Repository, you need to be running ths ICGC client on an EC2 VM instance<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>TNQB5ZVETWDVNXVE</RequestId><HostId>GpUyQQJI3yVsfrEChe9FRkqojhaRMkur1Lym/vNZX/rJSJlFkc7ok/A6j7SGVlpmM2tWFaIwvPo=</HostId></Error>

Please check the log for detailed error messages

[edsu7]

Error was on my end. missed "--profile collab" in certain commands. Commands "view","download --manifest","mount" work when "--profile collab" is provided.