search

wwood / kingfisher-download

Easier download/extract of FASTA/Q read data and metadata from the ENA, NCBI, AWS or GCP.
https://wwood.github.io/kingfisher-download
GNU General Public License v3.0
246 stars 38 forks source link

Failed to authenticate #9

Closed a00101 closed 1 year ago

a00101 commented 3 years ago

I got error below. How could I do that?


python ena-fast-download.py ERR962744
03/12/2021 02:27:29 PM INFO: Using aspera ssh key file: $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh
03/12/2021 02:27:29 PM INFO: Querying ENA for FTP paths for ERR962744..
03/12/2021 02:28:13 PM INFO: Downloading 1 FTP read set(s): ftp.sra.ebi.ac.uk/vol1/fastq/ERR962/ERR962744/ERR962744.fastq.gz
03/12/2021 02:28:13 PM INFO: Running command: ascp -T -l 300m -P33001  -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR962/ERR962744/ERR962744.fastq.gz .
ascp: failed to authenticate, exiting.

Session Stop  (Error: failed to authenticate)
Traceback (most recent call last):
  File "ena-fast-download.py", line 115, in <module>
    subprocess.check_call(cmd,shell=True)
  File "/root/anaconda2/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ascp -T -l 300m -P33001  -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/ERR962/ERR962744/ERR962744.fastq.gz .' returned non-zero exit status 1
wwood commented 3 years ago

Hi,

Looks to me like potentially you $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh file doesn't exist? Have you run this tool successfully before?

a00101 commented 3 years ago

Thanks for your rapid reply. yes. it worked well before and openssh file exsits. but it occurred intermittently.

wwood commented 3 years ago

Hi,

I just pushed a commit that fixed an issue different to the one you were getting, and now your command seems to work for me. Is there still an issue for you? I'm not sure I can be much help with intermittent issues since they are most likely on the ENA side.

Thanks.

a50044758 commented 3 years ago

i also meet this problem when i download data form ENA, "ascp: failed to authenticate, exiting. Session Stop (Error: failed to authenticate)". It was fine before, and this problem suddenly happened in one day. I copy the KEY to my file and used command "ascp -i asperaweb_id_dsa.openssh -l 300M -T -P33001 -k1 era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz ." Can anyone can help me ?

wwood commented 3 years ago

Hi @a50044758 is this still an issue? Running this command works for me:

kingfisher get -r SRR5005053 -m ena-ascp
AssumeAssume commented 3 years ago

Hi @a50044758 is this still an issue? Running this command works for me:

kingfisher get -r SRR5005053 -m ena-ascp

Hello, @wwood , I've checked the openssh file, it's ok, but still met the issue with the same error :

kingfisher get -r SRR5005053 -m ena-ascp
10/26/2021 11:24:21 PM INFO: Kingfisher v0.0.1-dev
10/26/2021 11:24:21 PM INFO: Attempting download method ena-ascp ..
10/26/2021 11:24:21 PM INFO: Using aspera ssh key file: $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh
10/26/2021 11:24:21 PM INFO: Querying ENA for FTP paths for SRR5005053..
10/26/2021 11:24:33 PM INFO: Downloading 2 FTP read set(s): ftp.sra.ebi.ac.uk/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz, ftp.sra.ebi.ac.uk/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_2.fastq.gz
10/26/2021 11:24:33 PM INFO: Running command: ascp -T -l 300m -P33001 -k 2 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz .
10/26/2021 11:24:45 PM WARNING: Error downloading from ENA with ASCP: Command ascp -T -l 300m -P33001 -k 2 -i $HOME/.aspera/connect/etc/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz . returned non-zero exit status 1.
STDERR was: b'ascp: failed to authenticate, exiting.\n'STDOUT was: b'\r\nSession Stop  (Error: failed to authenticate)\n'
10/26/2021 11:24:45 PM WARNING: Method ena-ascp failed
Traceback (most recent call last):
  File "/analysis2/software/kingfisher-download/bin/kingfisher", line 280, in <module>
    main()
  File "/analysis2/software/kingfisher-download/bin/kingfisher", line 232, in main
    kingfisher.download_and_extract(
  File "/analysis2/software/kingfisher-download/bin/../kingfisher/__init__.py", line 37, in download_and_extract
    download_and_extract_one_run(run, **kwargs)
  File "/analysis2/software/kingfisher-download/bin/../kingfisher/__init__.py", line 276, in download_and_extract_one_run
    raise Exception("No more specified download methods, cannot continue")
Exception: No more specified download methods, cannot continue

I'm not sure if it is a common question for using ASPERA to download from EBI/ENA. I've been using ascp to download data for over two years, but recently it failed. Could you still download from EBI using ascp?

Thanks.

Alright, it is our network problem...... Using another WLAN it worked.

wwood commented 3 years ago

Thanks @AssumeAssume good that you got to the bottom of it, and fixed by changing network. Do you know what the difference between the two WLANs is?

I've also posted a (somewhat vague) FAQ entry on this topic. If you have time, are you able to check the aspera logs in the network that is failing please? For me it was in /var/log/syslog. https://github.com/wwood/kingfisher-download#failed-to-authenticate-with-ascp

Thanks in advance. ben

AssumeAssume commented 3 years ago

Hello, ben @wwood

Do you know what the difference between the two WLANs is?

I'm not certainly sure where this two WLAN differ.

If you have time, are you able to check the aspera logs in the network that is failing please?

Sure. I checked the log file in /var/log/messages:

Oct 26 23:24:33 localhost ascp[83585]: LOG Configuration: using v2 configuration file "/xxxxx/.aspera/connect/etc/aspera.conf", user -
Oct 26 23:24:33 localhost ascp[83585]: LOG Initializing FASP version 3.9.1.168302, license max rate=(unlimited), account no.=1, license no.=1 product=6
Oct 26 23:24:33 localhost ascp[83585]: LOG Configured symlink actions: create=1, follow=1, follow_wide=0, skip=0
Oct 26 23:24:33 localhost ascp[83585]: LOG Aspera Connect version 3.9.8.176272
Oct 26 23:24:33 localhost ascp[83585]: LOG (access key) Not present
Oct 26 23:24:34 localhost ascp[83585]: LOG [asssh] remote host-key fingerprint 3eb78b35f5a01fdf131b20255d689196dcae1efa
Oct 26 23:24:35 localhost ascp[83585]: LOG [asssh] authentication succeeded, proceeding.
Oct 26 23:24:35 localhost ascp[83585]: LOG [libssh2] 0.886546 Failure Event: -22 - Unable to complete request for channel-process-startup
Oct 26 23:24:35 localhost ascp[83585]: ERR Failed to start remote command, libssh2 result -22
Oct 26 23:24:45 localhost ascp[83585]: ERR [asssh] channel eof wait rc=-9
Oct 26 23:24:45 localhost ascp[83585]: ERR [ascp] Failed to start remote command
Oct 26 23:24:45 localhost ascp[83585]: ERR failed to authenticate
Oct 26 23:24:45 localhost ascp[83585]: LOG FASP Session Start uuid=07920b99-46a9-4d0f-b16d-1281d1c96287 op=recv status=failed errcode=89 errstr="SSH authentication failed" source=/vo
l1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz dest=. source_prefix=- local=0.0.0.0:33001 peer=193.62.193.135:33001 tcp_port=33001 os="Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP"
 ver=3.9.1.168302 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000e
Oct 26 23:24:45 localhost ascp[83585]: LOG FASP Session Params uuid=07920b99-46a9-4d0f-b16d-1281d1c96287 userid=0 user="era-fasp" targetrate=300000000 minrate=0 rate_policy=fair cipher=none resume=2 create=0 ovr=1 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=0 prepostcmd=- tcp_mode=no rtt_auto=yes cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_delay exclude_older/newer_than=0/0
Oct 26 23:24:45 localhost ascp[83585]: LOG FASP Session Stop uuid=07920b99-46a9-4d0f-b16d-1281d1c96287 op=recv status=failed errcode=89 errstr="SSH authentication failed" source=/vol1/fastq/SRR500/003/SRR5005053/SRR5005053_1.fastq.gz dest=. source_prefix=- local=0.0.0.0:33001 peer=193.62.193.135:33001 tcp_port=33001 os="Linux 4.18.0-147.8.1.el8_1.x86_64 #1 SMP" ver=3.9.1.168302 lic=6:1:1 peeros="-" peerver=- peerlic=0:0:0 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000e
Oct 26 23:24:45 localhost ascp[83585]: LOG FASP Session Params uuid=07920b99-46a9-4d0f-b16d-1281d1c96287 userid=0 user="era-fasp" targetrate=300000000 minrate=0 rate_policy=fair cipher=none resume=2 create=0 ovr=1 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=0 prepostcmd=- tcp_mode=no rtt_auto=yes cookie="-" vl_proto_ver=1  peer_vl_proto_ver=0 vl_local=0 vlink_remote=0 vl_sess_id=0 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_delay exclude_older/newer_than=0/0
Oct 26 23:24:45 localhost ascp[83585]: LOG FASP Session Statistics [Receiver] id=07920b99-46a9-4d0f-b16d-1281d1c96287 delay=0ms rex_delay=0ms ooo_delay=0ms solicited_rex=0.00% rcvd_rex=0.00% rcvd_dups=0.00% ave_xmit_rate 0.00Kbps effective=0.00% effective_rate=0.00Kbps (detail: good_blks 0 bl_total 0 bl_orig 0 bl_rex 0 dup_blks 0 dup_last_blks 0 drop_blks_xnf 0) (sndr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rcvr ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (rex  ctl: sent 0 rcvd 0 lost 0 lost 0.00%) (progress: tx_bytes 0 file_bytes 0 tx_time 0) rex_xmit_blks 0  xmit_total 0 rex_xmit_pct 0.00%

The libssh2 results -22, maybe it's the point ?

wwood commented 3 years ago

Thanks. Maybe you are right about the -22. For reference here's the log of one that works

Oct 27 16:28:19 b2 ascp[731075]: LOG Configuration: using v2 configuration file "/home/ben/.aspera/connect/etc/aspera.conf", user -
Oct 27 16:28:19 b2 ascp[731075]: LOG Initializing FASP version 3.9.6.178596, license max rate=(unlimited), account no.=1, license no.=1 product=6
Oct 27 16:28:19 b2 ascp[731075]: LOG Configured symlink actions: create=1, follow=1, follow_wide=0, skip=0
Oct 27 16:28:19 b2 ascp[731075]: LOG Aspera Connect version 3.10.0.180973
Oct 27 16:28:20 b2 ascp[731075]: LOG (access key) Not present
Oct 27 16:28:22 b2 ascp[731075]: LOG [asssh] remote host-key fingerprint 3eb78b35f5a01fdf131b20255d689196dcae1efa
Oct 27 16:28:24 b2 ascp[731075]: LOG [asssh] authentication succeeded, proceeding.
Oct 27 16:28:24 b2 ascp[731075]: LOG AUTH [CLIENT] op=receive auth_method=client_full_access sess_auth=pass sess_root=unset access_key=NONE remote_user=era-fasp remote_host=fasp.sra.ebi.ac.uk system_user=ben token=- storage=(no docroot)
Oct 27 16:28:26 b2 ascp[731075]: LOG Xattrs-preserving modes for starters: --preserve-xattrs=none, --remote-preserve-xattrs=none; --preserve-acls=none, --remote-preserve-acls=none
Oct 27 16:28:26 b2 ascp[731075]: LOG FASP Session Start uuid=ade2f318-6ac5-4d51-abb6-3d3183a58ef1 op=recv status=started source=/vol1/fastq/SRR121/066/SRR12118866/SRR12118866_1.fastq.gz dest=. source_prefix=- local=131.181.167.16:38886 peer=193.62.193.135:33001 tcp_port=33001 os="Linux 5.13.0-20-generic #20-Ubuntu SMP F" ver=3.9.6.178596 lic=6:1:1 peeros="Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP " peerver=3.9.3.174419 peerlic=10:13043:63869 proto_sess=20003 proto_udp=20000 proto_bwmeas=20000 proto_data=2000e
Oct 27 16:28:26 b2 ascp[731075]: LOG FASP Session Params uuid=ade2f318-6ac5-4d51-abb6-3d3183a58ef1 userid=0 user="era-fasp" targetrate=300000000 minrate=0 rate_policy=fair cipher=none resume=2 create=0 ovr=1 times=0 precalc=no mf=0 mf_path=- mf_suffix=.aspera-inprogress partial_file_suffix=- files_encrypt=no files_decrypt=no file_csum=NONE dgram_sz=0 prepostcmd=- tcp_mode=no rtt_auto=yes cookie="-" vl_proto_ver=1  peer_vl_proto_ver=1 vl_local=0 vlink_remote=0 vl_sess_id=22363 srcbase=- rd_sz=0 wr_sz=0 cluster_num_nodes=1 cluster_node_id=0 cluster_multi_session_threshold=-1 range=0-0 keepalive=no test_login=no proxy_ip=- net_rc_alg=alg_delay exclude_older/newer_than=0/0
Oct 27 16:28:27 b2 ascp[731075]: LOG Measured pMTU: 1492 Bytes, start_brtt: 282 ms
Oct 27 16:28:27 b2 ascp[731075]: LOG datagram size 1448B, block size 1408B, path MTU 1492B
Oct 27 16:28:28 b2 ascp[731075]: LOG Receiver bl t/o/r/d/ts/c/h=0/0/0/0/0/1/1 rex_rtt l/h/s/o=0/0/282/8 ooo_rtt l/h/s/o=0/0/282/8 rate_rtt b/l/h/s/r/f=282/0/0/282/0/2 ctl bm/bs=0/0 rex n/s/q/v/a/r=0/0/0/0/0/0 bl l/d/o/r/a/x/dl/df/dm/ds=0/0/0/0/0/0/0/0/0/0 disk l/h/b=0/0/0 vlink lq/lo/rq/ro=0/0/0/0 rate t/m/c/n/vl/vr/r=300000000/0/184980000/184980000/300000000/300000000/300000000 prog t/f/e=0/0/0 rcvD=0
Oct 27 16:28:28 b2 ascp[731075]: LOG Receiver DS Qs ds/n/rq/ao/ap/rd/ru/no/po/pc/do/tv/sv=0/0/0/0/0/0/0/0/0/0/0/0/0 Rs i/o=0/0 mgmt backlog i/s/n =
Oct 27 16:28:32 b2 ascp[731075]: LOG FASP Transfer Start uuid=ade2f318-6ac5-4d51-abb6-3d3183a58ef1 op=recv status=started file="SRR12118866_1.fastq.gz" size=4117481 start_byte=0 rate=300.00Mbps loss=0.00 rexreqs=0 overhead=0 mtime="2020-08-21 00:42:43"
Oct 27 16:28:37 b2 ascp[731075]: LOG FASP Transfer Stop uuid=ade2f318-6ac5-4d51-abb6-3d3183a58ef1 op=recv status=success file="SRR12118866_1.fastq.gz" size=4117481 start_byte=0 rate=6.66Mbps elapsed=4.95s loss=0.00 rexreqs=0 overhead=0 mtime="2020-08-21 00:42:43"
yulchen810 commented 9 months ago

Hi, I want to upload fastq folder to ncbi, I used command ssh-keygen -f ./private.openssh to create key, and used command **ascp -i <mypath/to/key_file> -QT -l100m -k1 -d <path/to/folder/containing files> subasp@upload.ncbi.nlm.nih.gov:uploads/*gmail.com and get the following error: Key passphrase: ascp: failed to authenticate, exiting.

Session Stop (Error: failed to authenticate)

at home directory, there is private.openssh and private.openssh.pub file, there is no ~/.aspera/connect/.. directory. does anyone know what happened?

wwood commented 9 months ago

Hi @yulchen810 upload isn't supported by kingfisher, so I won't be able to be much help beyond this comment. From memory though, you will need to get the key from NCBI, not generate it yourself.

yulchen810 commented 9 months ago

Hi @yulchen810 upload isn't supported by kingfisher, so I won't be able to be much help beyond this comment. From memory though, you will need to get the key from NCBI, not generate it yourself.

Hi, I've resolved this problem. Thank you.