search

nanopore-wgs-consortium / NA12878

Data and analysis for NA12878 genome on nanopore
Other
372 stars 93 forks source link

bam download difficulty #68

Open bgphd opened 5 years ago

bgphd commented 5 years ago

I've tried downloading the 2018 Nanopore NA12878 bam file to no avail.

I have tried the https:// method twice on two different computers (one mac, one workstation PC), using two different connections (wired), but it always fails out with partial downloads.

I have also installed the Amazon CLI, but I am getting this:

D:>cd nanopore

D:\nanopore>aws s3 cp s3://nanopore-human-wgs/rel3-nanopore-wgs-288418386-FAB39088.fastq.gz . fatal error: Unable to locate credentials

D:\nanopore>

mattloose commented 5 years ago

You need to configure your aws client - have you completed the instructions as described here? https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

bgphd commented 5 years ago

Hello, it is super easy to configure the client using the Windows installation package, viz:

D:\nanopore>aws --version aws-cli/1.16.181 Python/3.6.0 Windows/10 botocore/1.12.171

D:\nanopore>

the python method on Mac is still very buggy (at least in 10.13.6).

Bert

On Fri, Jun 21, 2019 at 1:17 PM mattloose notifications@github.com wrote:

You need to configure your aws client - have you completed the instructions as described here? https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nanopore-wgs-consortium/NA12878/issues/68?email_source=notifications&email_token=AKPPWL6Z2WVDIGONRYNDZPDP3UZNNA5CNFSM4H2UQ5L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJPQSI#issuecomment-504559689, or mute the thread https://github.com/notifications/unsubscribe-auth/AKPPWL7KAWWQW3HLKR7BCDLP3UZNNANCNFSM4H2UQ5LQ .

bgphd commented 5 years ago

Oh, so my configuration is not the issue...

On Fri, Jun 21, 2019 at 1:17 PM mattloose notifications@github.com wrote:

You need to configure your aws client - have you completed the instructions as described here? https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nanopore-wgs-consortium/NA12878/issues/68?email_source=notifications&email_token=AKPPWL6Z2WVDIGONRYNDZPDP3UZNNA5CNFSM4H2UQ5L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJPQSI#issuecomment-504559689, or mute the thread https://github.com/notifications/unsubscribe-auth/AKPPWL7KAWWQW3HLKR7BCDLP3UZNNANCNFSM4H2UQ5LQ .

bgphd commented 5 years ago

Hi,

I tried Adam Phillippy's suggestion this morning after two failed attempts to https:// sequences

Using AWS CLI I did the following and got the following results ( a different error than I had been getting, perhaps because I first entered some credentials, as shown below). Yes, my CLI is working properly (I think). Suggestions appreciated.

D:\nanopore>aws configure AWS Access Key ID [None]: AKIA6F4YYBA AWS Secret Access Key [None]: pg2HtYj8hi7IWPKw Default region name [None]: us-west-2 Default output format [None]:

D:\nanopore\bam>aws --no-sign-request s3 cp s3://nanopore-human-wgs/rel3-nanopore-wgs288418386-FAB39088.fastq.gz . fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

D:\nanopore\bam>aws --no-sign-request s3 cp s3://nanopore-human-wgs/rel5-guppy-0.3.0-chunk10k.sorted.bam . fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

D:\nanopore\bam>aws --version aws-cli/1.16.181 Python/3.6.0 Windows/10 botocore/1.12.171

On Fri, Jun 21, 2019 at 1:17 PM mattloose notifications@github.com wrote:

You need to configure your aws client - have you completed the instructions as described here? https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nanopore-wgs-consortium/NA12878/issues/68?email_source=notifications&email_token=AKPPWL6Z2WVDIGONRYNDZPDP3UZNNA5CNFSM4H2UQ5L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJPQSI#issuecomment-504559689, or mute the thread https://github.com/notifications/unsubscribe-auth/AKPPWL7KAWWQW3HLKR7BCDLP3UZNNANCNFSM4H2UQ5LQ .

mattloose commented 5 years ago

OK - I still think this is an issue with your client configuration. You default region name is us-west-2 but the data are held on eu-west-1 - there are some reports on the internet for an issue on the amazon client when you try and copy cross regions resulting in 400 errors as you are seeing. I suggest looking in to your region configuration.

mattloose commented 5 years ago

Also - from the links you are giving, it looks as though you are trying to download all the fastq datasets - note that we have just made rel6 available which are the latest data.

bgphd commented 5 years ago

Hi!

I have never gotten this to work :-{

It either times out or gives an error message.

I found some chromosome X assemblies elsewhere (presumbably from this same project, because it was Nanopore on NA12878) and successfully used that!

Now I see several more human genomes are available via HELEN and I would like to secure the bam files or at least the bam files of chromosome X. How can I do this?

Bert Gold

On Mon, Jun 24, 2019 at 11:38 AM Bert Gold bert.gold@billiontoone.com wrote:

Hi,

I tried Adam Phillippy's suggestion this morning after two failed attempts to https:// sequences

Using AWS CLI I did the following and got the following results ( a different error than I had been getting, perhaps because I first entered some credentials, as shown below). Yes, my CLI is working properly (I think). Suggestions appreciated.

D:\nanopore>aws configure AWS Access Key ID [None]: AKIA6F4YYBA AWS Secret Access Key [None]: pg2HtYj8hi7IWPKw Default region name [None]: us-west-2 Default output format [None]:

D:\nanopore\bam>aws --no-sign-request s3 cp s3://nanopore-human-wgs/rel3-nanopore-wgs288418386-FAB39088.fastq.gz . fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

D:\nanopore\bam>aws --no-sign-request s3 cp s3://nanopore-human-wgs/rel5-guppy-0.3.0-chunk10k.sorted.bam . fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

D:\nanopore\bam>aws --version aws-cli/1.16.181 Python/3.6.0 Windows/10 botocore/1.12.171

On Fri, Jun 21, 2019 at 1:17 PM mattloose notifications@github.com wrote:

You need to configure your aws client - have you completed the instructions as described here? https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nanopore-wgs-consortium/NA12878/issues/68?email_source=notifications&email_token=AKPPWL6Z2WVDIGONRYNDZPDP3UZNNA5CNFSM4H2UQ5L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJPQSI#issuecomment-504559689, or mute the thread https://github.com/notifications/unsubscribe-auth/AKPPWL7KAWWQW3HLKR7BCDLP3UZNNANCNFSM4H2UQ5LQ .