search

IGS / portal_client

Python-based client for downloading data made available through portals powered by the GDC-based portal system..
MIT License
17 stars 17 forks source link

How to obtain an HMP account for FASP downloads? #9

Closed zhanxw closed 5 years ago

zhanxw commented 5 years ago

I want to download via FASP protocol and need a username. Can you please show where to apply for it? Thanks.

victor73 commented 5 years ago

Which portal are you interested in getting a username for? The portal_client tool is designed to work with multiple projects/portals, so this is a necessary bit of information...

jacknicoludis commented 5 years ago

I have the same question. I'm trying to download files from the HMP Project Data Portal. I requested a password from hmpdacc.org but when I try to transfer files, I get the error that none of the urls succeeded. Here is my command-line submission and a section of the output when running in debug mode: 

portal_client --manifest ~/path-to-file/hmp_cart_4364ccfe46.tsv --destination ~/path-to-file/data --endpoint-priority FASP,HTTP --user jacknicoludis --debug
2019-09-11 12:53:17,267 - aspera - DEBUG - Command: ascp -T -v -l 300M jacknicoludis@aspera.ihmpdcc.org:/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom /Users/jacknicoludis/Dropbox/insight/drugmetabolism/HMP-T2D-abundances/data/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom.partial
2019-09-11 12:53:17,267 - aspera - DEBUG - In get_ascp_env.
2019-09-11 12:53:17,268 - aspera - INFO - Setting ASPERA_SCP_PASS environment variable.
2019-09-11 12:53:17,273 - aspera - INFO - Beginning transfer.
2019-09-11 12:53:20,621 - aspera - INFO - Invocation of ascp complete. Return code: 1.
2019-09-11 12:53:20,622 - aspera - ERROR - Aspera authentication failure.
2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - ERROR - Aspera transfer failed.
2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - DEBUG - Returning error
Skipping file ID 76612bd9a41885add4f6b0b768365c93 as none of the URLs ['FASP'] succeeded.

I just requested my account earlier today so it is possible that it hasn't been approved...

zhanxw commented 5 years ago

I also want to obtain data from hmpdacc. Is there a way to create a user name so that I can use portal_client to download data?

Best, Xiaowei

On Wed, Sep 11, 2019 at 3:01 PM jacknicoludis notifications@github.com wrote:

I have the same question. I'm trying to download files from the HMP Project Data Portal. I requested a password from hmpdacc.org but when I try to transfer files, I get the error that none of the urls succeeded. Here is my command-line submission and a section of the output when running in debug mode:

portal_client --manifest ~/path-to-file/hmp_cart_4364ccfe46.tsv --destination ~/path-to-file/data --endpoint-priority FASP,HTTP --user jacknicoludis --debug

2019-09-11 12:53:17,267 - aspera - DEBUG - Command: ascp -T -v -l 300M jacknicoludis@aspera.ihmpdcc.org:/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom /Users/jacknicoludis/Dropbox/insight/drugmetabolism/HMP-T2D-abundances/data/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom.partial 2019-09-11 12:53:17,267 - aspera - DEBUG - In get_ascp_env. 2019-09-11 12:53:17,268 - aspera - INFO - Setting ASPERA_SCP_PASS environment variable. 2019-09-11 12:53:17,273 - aspera - INFO - Beginning transfer. 2019-09-11 12:53:20,621 - aspera - INFO - Invocation of ascp complete. Return code: 1. 2019-09-11 12:53:20,622 - aspera - ERROR - Aspera authentication failure. 2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - ERROR - Aspera transfer failed. 2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - DEBUG - Returning error Skipping file ID 76612bd9a41885add4f6b0b768365c93 as none of the URLs ['FASP'] succeeded.

I just requested my account earlier today so it is possible that it hasn't been approved...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/IGS/portal_client/issues/9?email_source=notifications&email_token=AABGRCAQ32GO2HUJBKK6QSTQJFFA7A5CNFSM4IPWVE6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6PW6CY#issuecomment-530542347, or mute the thread https://github.com/notifications/unsubscribe-auth/AABGRCBV4MKIRQKKUUWFLC3QJFFA7ANCNFSM4IPWVE6A .

victor73 commented 5 years ago

One can download data from the HMP project anonymously (without a username/password) by simply using the HTTP endpoint exclusively. This is because the HMP data has been made available over HTTP. However, when using the Aspera protocol (the FASP endpoint), portal_client will prompt for login credentials. An account (username and password) is required in this case. One can request an account online through the HMPDACC website's registration page, but the account will not work until you receive an account approval notice via email.

To download anonymously:

portal_client --manifest ~/path-to-file.tsv --destination /path/to/destination/ --endpoint-priority HTTP --debug
jacknicoludis commented 5 years ago

This did not work. When I try to do it using the HTTP endpoint-priority, I get an error like this for all files:

No valid URL found in the manifest for file ID 9dc112963819ad139611820d22900771
2019-09-12 15:46:13,362 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.

My manifest contains files from the T2D dataset.

zhanxw commented 5 years ago

Http does not work for me either. I have applied for an account on the portal website but have heard back for a long time. Is there any way I can try?

Sent from my iPhone

On Sep 12, 2019, at 5:49 PM, jacknicoludis notifications@github.com wrote:

This did not work. When I try to do it using the HTTP endpoint-priority, I get an error like this for all files:

No valid URL found in the manifest for file ID 9dc112963819ad139611820d22900771 2019-09-12 15:46:13,362 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint. My manifest contains files from the T2D dataset.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

victor73 commented 5 years ago

@jacknicoludis and @zhanxw, can you both post a few lines of your manifest files please? There should be some URLs in there with http:// links...

jacknicoludis commented 5 years ago

Here is a sample:

file_id md5 size    urls    sample_id
76612bd9a41885add4f6b0b76835fc51    0c3315b09b64e939408987b224b78ef3    184000  fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J45191_1_ST_T0_B0_0120_ZOZOW1T-6046_APAJ2.biom  d57eb430d669de8329be1769d4fe660b
9dc112963819ad139611820d228cb2f2    8c40ead778d7a5b5698f5ca1512adffb    69000   fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J79976_1_ST_T0_B0_0120_ZUPAQVU-10_B64B9.biom    5a950f27980b5d93e4c16da1249c555c
9dc112963819ad139611820d228c6b7e    8a65e6ffe3ec010643ec506a72dac1a1    43000   fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J79856_1_ST_T0_B0_0120_ZL9BTWF-6012_B86HB.biom  5a950f27980b5d93e4c16da12498f3b2
9dc112963819ad139611820d22a1d656    0e371e0cca834f9c6658e571b1105057    34000   fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J79953_1_ST_T0_B0_0120_ZL9BTWF-6033_B64B9.biom  5a950f27980b5d93e4c16da1249bad47
9dc112963819ad139611820d22928e5d    97dad57eb036d349d07270423f81eacb    103000  fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J79859_1_ST_T0_B0_0120_ZOZOW1T-7030_B86HB.biom  5a950f27980b5d93e4c16da124990e68
76612bd9a41885add4f6b0b7683546f3    5c83b03463fe7cced8f3d53d19de2865    196000  fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J45211_1_ST_T0_B0_0120_ZOZOW1T-6042-1_APATM.biom
victor73 commented 5 years ago

This manifest file only has FASP urls, so in that case, the suggested approach of using just the HTTP endpoint won't work. Therefore, you'll need to wait for your account approval to be completed. Can you describe the steps you used to generate this manifest? There should have been some HTTP urls included...

jacknicoludis commented 5 years ago

Every path in my manifest contains FASP urls, because the T2D dataset only contains FASP urls. I've successfully downloaded HTTP files for other datasets, but the FASP ones still require approval I guess. Now I gotta figure out how to get approval more quickly!

zhanxw commented 5 years ago

Is there any way we can convert the fasp protocol and url to http?

@jacknicoludis, Please post if you have hints to get account approval, as I have not heard back after online account registration for a while. Thanks.

Sent from my iPhone

On Sep 13, 2019, at 9:45 AM, notifications@github.com wrote:

Every path in my manifest contains FASP urls, because the T2D dataset only contains FASP urls. I've successfully downloaded HTTP files for other datasets, but the FASP ones still require approval I guess. Now I gotta figure out how to get approval more quickly!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jacknicoludis commented 5 years ago

Today I will try to manually replace the "fasp" of the manifest to "http" but I don't have high hope for that. I also tried out installing the Aspera Client separately (as they suggest on one part of the website) but I still needed a username, so that failed.

I found this page that has some datasets available for download: https://www.hmpdacc.org/hmp/resources/data_browser.php

I'm also trying to get approval, though I haven't been able to find any contact information on their website

jacknicoludis commented 5 years ago

I got access to HMP!

zhanxw commented 5 years ago

Can you share your experience in getting an account?

I registered on 6/18 but have not heard feedbacks since then.

Sent from my iPhone

On Sep 13, 2019, at 3:35 PM, jacknicoludis notifications@github.com wrote:

I got access to HMP!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

victor73 commented 5 years ago

Closing as nothing more to be done here.