[v3] Two of my datasets cant be directly mounted with goofys as suggested by the guidelines

data-preservation-programs / slingshot

Official public repository for feedback and data collection in Filecoin Slingshot

https://slingshot.filecoin.io

68 stars 250 forks source link

[v3] Two of my datasets cant be directly mounted with goofys as suggested by the guidelines #543

Closed benjaminh83 closed 2 years ago

benjaminh83 commented 2 years ago

Purpose of this GitHub issue

[ ] Issue with Slingshot deal engine
[x] Issue with AWS Buckets
[ ] Feedback for program design

Description

We are not able to mount the following datasets directly with goofys:

coupled-model-intercomparison-project-6, size 1.66 PiB, region us-west-2, file count 40,584,114
encyclopedia-of-dna-elements-encode, size 1014.2 TiB, region us-west-2, file count 1,220,750

The third dataset (covid-19-genome-sequence-dataset, size 1.1 PiB, region us-east-1, file count 10,245,257) works fine with goofys.

Urgency/priorization

[ ] Low: non-blocking, nice to have
[x] Medium: non-blocking, eventually must have
[ ] High: blocking (i.e., I cannot get deal proposals when expected)

benjaminh83 commented 2 years ago

Running the command goofys cmip6-pds /mnt/external-buckets/slingshotv3/cmip6-pds gives us the following error:

Jul 13 11:50:40 staging1 /usr/local/bin/goofys[68273]: s3.ERROR code=NoCredentialProviders msg=no valid providers in chain. Deprecated.#012#011For verbose messaging see aws.Config.CredentialsChainVerboseErrors, err=<nil>#012
Jul 13 11:50:40 staging1 /usr/local/bin/goofys[68273]: main.ERROR Unable to access 'cmip6-pds': NoCredentialProviders: no valid providers in chain. Deprecated.#012#011For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Jul 13 11:50:40 staging1 /usr/local/bin/goofys[68273]: main.FATAL Mounting file system: Mount: initialization failed

liuziba commented 2 years ago

@benjaminh83 could you try this patch and see if it works? https://github.com/liuziba/goofys/commit/fe378e8b6205e1afc6a32b449838a07578e31410

dkkapur commented 2 years ago

@benjaminh83 did this work for you?

benjaminh83 commented 2 years ago

@liuziba @dkkapur Sorry for not getting back on this. So we ended up deciding not to use Goofys and just use native S3 instead.