search

openproblems-bio / neurips2021_multimodal_viash

MIT License
18 stars 7 forks source link

S3 Bucket Pull from Generate Submission doesn't work without AWS Creds #17

Closed dburkhardt closed 3 years ago

dburkhardt commented 3 years ago

When running the ./generate_submission.sh script:

######################################################################
##            Generating submission files using nextflow            ##
######################################################################
N E X T F L O W  ~  version 21.04.1
Launching `openproblems-bio/neurips2021_multimodal_viash` [fabulous_watson] - revision: 1ae5ae17b4 [0.4.0]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [d40bafe6ad]
WARN: It appears you have never run this project before -- Option `-resume` is ignored
[-        ] process > method:method_process -
[-        ] process > method:method_process -
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: V8RZMPP7311QDM2Q; S3 Extended Request ID: ynXO8NxhUgX9Q3G2at2wzTgpm3WW01jQBTJHAyyqgPq2nbW91sxw9W8pmlxIIWEf6unYa+IdEvs=)

The issue is that when accessing data from a public bucket, you need to add the --no-sign-request to aws s3 ls s3://neurips2021-multimodal-public-datasets/

dburkhardt commented 3 years ago

Currently testing to see if aws s3 sync s3://neurips2021-multimodal-public-datasets/ output/ --no-sign-request first fixes this

rcannood commented 3 years ago

Problem should be solved by first syncing from aws to local disk and then running nextflow. Can you confirm that it works? ^^

rcannood commented 3 years ago

@dburkhardt Is it working for you?

dburkhardt commented 3 years ago

Works now!