MUTO-INTL cannot upload with Score

[rosibaj]

user command used:

ubuntu@jw32-sf5:~$ ./score-client-5.1.0/bin/score-client upload --manifest PD41424b_manifest.txt
Uploading object: '/lustre/scratch123/mutographs/molecular-data-submission/PD41424b/PD41424b.bam' using the object id 5ddae072-6a63-5c4f-910b-7ba665e300b9
ERROR: Command error: java.lang.IllegalStateException: There is already an upload in progress that was started elsewhere with upload id 2~5jTWefxR_macr3dFgh_hYk1ajx-Nwcv. Aborting local upload.

Also this error: Access refused by repository. Ensure client is running as part of cloud.

Example:

Log file: client.log

Verified that these are the URLs that they are using:

• https://submission-song.rdpc.cancercollaboratory.org
• https://submission-score.rdpc.cancercollaboratory.org

[rosibaj]

March 01, 2021

• Test completed against production Song/Score
• Confirmed I am not on vpn by https://platform-ui.qa.argo.cancercollaboratory.org/ (it did not load)
• Steps 1-4 worked: https://docs.icgc-argo.org/docs/submission/submitting-molecular-data
• Step 5 (publishing the analysis) failed with this error:
• SONG_SERVER_ERROR[unknown.error @ 1614618063104]: Request processing failed; nested exception is error="access_denied", error_description="Error requesting access token."

  "errorId" : "unknown.error",
  "httpStatusCode" : 500,
  "httpStatusName" : "INTERNAL_SERVER_ERROR",
  "requestUrl" : "http://submission-song.rdpc.cancercollaboratory.org/studies/TEST-INTL/analysis/publish/74a379a4-1954-43f6-a379-a41954a3f693?ignoreUndefinedMd5=false",
  "datetime" : "2021-03-01T12:01:03.104-05:00[America/New_York]",
  "timestamp" : 1614618063104,
  "message" : "Request processing failed; nested exception is error=\"access_denied\", error_description=\"Error requesting access token.\"",
  "debugMessage" : "[ROOT_CAUSE] -> org.springframework.web.client.HttpClientErrorException$Forbidden: 403 null",

[rosibaj]

Update:

• We need to take a look at the bucket
• verify if there are any objects in there in some weird intermediate state

[rosibaj]

Debugged with @andricDu:

• Checked the bucket. The metadata of file parts was there, but was not verified as "completed". Unsure why this happened, but the error that score was throwing was accurate. There was a "stuck" upload so it refused to start a second duplicate upload. We have deleted the offending upload metadata from the bucket and asked the data submitter to try again.
• In addition to this, there were several configuration issues. Dusan has fixed these, and i was able to successfully publish the analysis, representing a completed e2e test of data upload to the production-submission-song.

  curl --location --request GET 'https://submission-song.rdpc.cancercollaboratory.org/studies/TEST-INTL/analysis/74a379a4-1954-43f6-a379-a41954a3f693' \

I had re-asked the data submitter to attempt the upload a third time. Waiting on confirmation of results from the MUTO-INTL data submitter.

[andricDu]

@rosibaj sorry to just clarify, the file was not there, but there was a record of all the parts in the upload metadata. It looked like it uploaded based on the upload meta data, but it just wasn't there.

[rosibaj]

@andricDu updated summary!

[rosibaj]

MUTO-INTL tried again (I re-submitted the JSON file with sing and then used the new analysisId to do the uploading, however, it still returned the same errors)

• same initial error provided There is already an upload in progress that was started elsewhere with upload id 2~5jTWefxR_macr3dFgh_hYk1ajx-Nwcv. Aborting local upload.
• Also still seeing Access refused by repository. Ensure client is running as part of cloud.

Troubleshooting:

• generated a signed S3 URL and asked them to download it to verify connection
• waiting on confirmation.

[rosibaj]

Verified that they can download the S3 url object

[rosibaj]

Clsoing this issue as they had successful uploads from their system. The issue is not on our end.