general gene regulatory network link not working with wget

RubenVanEsch commented 2 months ago

Hi LINGER team, paper looked really cool! So wanted to try and run on some of my own data and tried to download the general GRN using the wget code provided:

wget --load-cookies /tmp/cookies.txt "https://drive.usercontent.google.com/download?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://drive.usercontent.google.com/download?id=1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS" -O data_bulk.tar.gz && rm -rf /tmp/cookies.txt

however, I get the following error message: <!DOCTYPE html><html><head><title>Google Drive - Quota exceeded</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><style nonce="xMKJdxGwxulflmiIEboRhQ">.goog-link-button{position:relative;color:#15c;text-decoration:underline;cursor:pointer}.goog-link-button-disabled{color:#ccc;text-decoration:none;cursor:default}body{color:#222;font:normal 13px/1.4 arial,sans-serif;margin:0}.grecaptcha-badge{visibility:hidden}.uc-main{padding-top:50px;text-align:center}#uc-dl-icon{display:inline-block;margin-top:16px;padding-right:1em;vertical-align:top}#uc-text{display:inline-block;max-width:68ex;text-align:left}.uc-error-caption,.uc-warning-caption{color:#222;font-size:16px}#uc-download-link{text-decoration:none}.uc-name-size a{color:#15c;text-decoration:none}.uc-name-size a:visited{color:#61c;text-decoration:none}.uc-name-size a:active{color:#d14836;text-decoration:none}.uc-footer{color:#777;font-size:11px;padding-bottom:5ex;padding-top:5ex;text-align:center}.uc-footer a{color:#15c}.uc-footer a:visited{color:#61c}.uc-footer a:active{color:#d14836}.uc-footer-divider{color:#ccc;width:100%}.goog-inline-block{position:relative;display:-moz-inline-box;display:inline-block}* html .goog-inline-block{display:inline}*:first-child+html .goog-inline-block{display:inline}sentinel{}</style><link rel="icon" href="//ssl.gstatic.com/docs/doclist/images/drive_2022q3_32dp.png"/></head><body><div class="uc-main"><div id="uc-text"><p class="uc-error-caption">Sorry, you can't view or download this file at this time.</p><p class="uc-error-subcaption">Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.</p></div></div><div class="uc-footer"><hr class="uc-footer-divider"></div></body></html>

going to the drive link on my laptop and downloading also does not seem to work, so both options seem to be blocked

withermatt commented 2 months ago

Hi, I am having the same issue downloading the general grn. wget --load-cookies /tmp/cookies.txt "https://drive.usercontent.google.com/download?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://drive.usercontent.google.com/download?id=1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS' -O- | sed -rn 's/.confirm=([0-9A-Za-z_]+)./\1\n/p')&id=1lAlzjU5BYbpbr4RHMlAG DOh9KWdCMQpS" -O data_bulk.tar.gz && rm -rf /tmp/cookies.txt

My output is: --2024-05-07 13:19:54-- https://drive.usercontent.google.com/download?export=download&confirm=&id=1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS Resolving drive.usercontent.google.com (drive.usercontent.google.com)... 142.251.211.225, 2607:f8b0:400a:804::2001 Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|142.251.211.225|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2004 (2.0K) [text/html]

When I unzip, tar -xzf data_bulk.tar.gz

My output is: gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now

Downloading from firefox initiates (file size 19.4 GB) but ends up failing after about 75% completion. UPDATE: I retried to download from google drive in firefox and it was successful. Must have timed out or something my first try.

RubenVanEsch commented 2 months ago

@withermatt for the wget, you get the same error, but the extension of your file is wrong because the download failed. If you change the extension to .txt and then open it with vim or something you will see the same message

amssyqy commented 2 months ago

Thanks for using LINGER~ There are 2 more options to download the data.

wget --no-check-certificate -O data_bulk.tar.gz "https://drive.usercontent.google.com/download?id=1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS&export=download&authuser=0&confirm=t&uuid=1dd434ea-bada-46dd-80bc-3c80945066cd&at=APZUnTW25jguC0k7Lljqk6Bo71nF%3A1705074384034"
Open the link in Chrome: https://drive.google.com/file/d/1lAlzjU5BYbpbr4RHMlAGDOh9KWdCMQpS/view?usp=sharing We are checking the reason why the original command does not work. Maybe one reason is 'Too many users have viewed or downloaded this file recently.' We are trying to solve this problem. Thank you~

RubenVanEsch commented 1 month ago

That new wget link works great! Though it might have also just been the time-out being lifted by trying it a week later

Durenlab / LINGER

general gene regulatory network link not working with wget #31