wkentaro / gdown

Google Drive Public File Downloader when Curl/Wget Fails
MIT License
4.18k stars 348 forks source link

Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

Open alexklwong opened 4 years ago

alexklwong commented 4 years ago

I want to download a 16GB zip file on Google Drive using:

gdown https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8

I have already set the permissions on the file to ``Anyone with the link''

But I get this error:

Permission denied: https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8 Maybe you need to change permission over 'Anyone with the link'?

shidalao commented 1 year ago

My solution is (same as above more or less)

  1. Click "Make A Copy" on google drive. It will create a copy of the file (either it's yours or other people's) to your drive.
  2. Set the permission to "Anyone with the link" as "Viewer".
  3. Now you can download as usual.

This works for single files or a folder. The problem is that the same error will occur if you are downloading a folder containing too many files. In that case, you can delete the folder and do the above process again, in this way, google will recognize them as different files so it doesn't err like "Too many users have viewed or downloaded".

The drawback of this is that it requires manual operation once the error occurs again.

IMO this is related to google drive not gdown so there is no point to do those pip install stuff

Google Driver now does not have "make a copy" feature, but "make a shortcut" instead, in this case, you can't have own copy of the target, which is very sad.

ArgoHA commented 1 year ago

Updated to the latest version (gdown-4.7.1), tried using gdown "id...&confirm=t" and nothing worked for me (47gb zip file):

Access denied with the following error:

    Cannot retrieve the public link of the file. You may need to change
    the permission to 'Anyone with the link', or have had many accesses.

You may still be able to access the file from the browser:

     https://drive.google.com/uc?id=...&confirm=t

upd: worked for me after I changed permission type from 'editor' back to 'viewer'

upd2: same command failed on another machine. The 'cure' was just to wait for 10 mins and same command worked. idn why

dt990pro commented 12 months ago

Updated to the latest version (gdown-4.7.1), tried using gdown "id...&confirm=t" and nothing worked for me (47gb zip file):

Access denied with the following error:

  Cannot retrieve the public link of the file. You may need to change
  the permission to 'Anyone with the link', or have had many accesses.

You may still be able to access the file from the browser:

   https://drive.google.com/uc?id=...&confirm=t

upd: worked for me after I changed permission type from 'editor' back to 'viewer'

upd2: same command failed on another machine. The 'cure' was just to wait for 10 mins and same command worked. idn why

same issue here.

MikeBivins commented 12 months ago

very strange that this is still an issue for almost entire nodejs time. Ive also had some encounters every once a while not being able to proceed just accept it. If it is indeed the cookie and its behaviour at warning etc. why is there no automated option still so

grypesc commented 11 months ago

Still not solved 👎🏻

z3d3m0n commented 11 months ago

same here

davyzhang commented 10 months ago

It looks like google is trying to protect its bandwidth. After some time with downloading big files, it will be very slow and shutting down access later. Temporary solution I found : https://github.com/ericyd/gdrive-copy

Copy to your own folder, reshare it, coulde bypass this check

zingfrid commented 9 months ago

But the thing is that I'm able to download the file from the browser, but not with gdown. So, is there are possibility for gdown to simulate 'browser behavior' and pretend to be a browser for google?

JiaHeng-DLUT commented 8 months ago

For anyone who is still struggling with this problem, please refer to https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work or 解决命令行下载Google Drive大文件出现的问题.

e96031413 commented 8 months ago

Older version gdown also helps.

pip install gdown==4.6.0

I finally solved this issue by using 4.6.0 gdown to download files.

aasyedd1 commented 8 months ago

@e96031413 worked like a charm. thanks alot

neeeal commented 8 months ago
pip install gdown==4.6.0

actually solved my issue. thanks

wkentaro commented 8 months ago

This PR #300 should help this issue in some regard. I also added FAQ to the README. https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-still-cant-download

Releasing v5.0.0 soon.

pmeier commented 7 months ago

When hitting large files that GDrive wants extra confirmation before it starts the download, e.g. https://drive.google.com/uc?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK, we currently hit this branch:

https://github.com/wkentaro/gdown/blob/ae122f428a0a5e72ec76193b797b3750435f8f25/gdown/download.py#L31-L35

The relevant corresponding HTML looks like

<form id="download-form" action="https://drive.usercontent.google.com/download" method="get">
    <input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/>
    <input type="hidden" name="id" value="1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK">
    <input type="hidden" name="confirm" value="t">
    <input type="hidden" name="uuid" value="ce320bab-e9f0-4d0d-a9d8-8cede0161915">
</form>

The branch above just extracts https://drive.usercontent.google.com/download from it and this later fails other checks as this doesn't provide a file.

What we should do here is to extract the other parts of the form as well and append them as query. So for this example https://drive.usercontent.google.com/download?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK&confirm=t&uuid=ce320bab-e9f0-4d0d-a9d8-8cede0161915. As you can see this link also includes confirm=t that multiple people in this thread have already mentioned. I'll send a PR to fix this.

pmeier commented 7 months ago

@wkentaro see #308. Could you please have a look?

wkentaro commented 7 months ago

Hi everyone.

I've just merged #308 by @pmeier, which adds extra pattern to parse the url to download.

Since I've never seen that pattern, I suspect it is a regional issue (I'm accessing from Japan, @pmeier is accessing from Germany). But at least It should help people who have the same issue as him.

vanloc1808 commented 6 months ago

@wkentaro Hi, I observed that only downgrading to 4.6.0 helps, new version does not help :(

wkentaro commented 6 months ago

Can you please give me an example url?

vanloc1808 commented 6 months ago

https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link Sure, here is it

vanloc1808 commented 6 months ago

I tried both

gdown --fuzzy 'https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link'

and

gdown https://drive.google.com/uc?id=1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb

But it only works for me with gdown==4.6.0

wkentaro commented 6 months ago

@vanloc1808 It seems working for me with the latest.

image

Maybe the cookies file is not helping? If so remove it by rm ~/.cache/gdown/cookies.txt

vanloc1808 commented 6 months ago

@wkentaro it works for me, thank you!

Junyi42 commented 6 months ago

Older version gdown also helps.

pip install gdown==4.6.0

I finally solved this issue by using 4.6.0 gdown to download files.

Restarting the terminal after the downgrading works for me!

AndrewLt commented 6 months ago

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

julienboussard commented 6 months ago

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Yeah this happened to me as well several times (with different versions). I'm not sure why, but changing the version (sometimes to a newer one, sometimes to an older one) seem to work, but always with a different version. Maybe it clears something and allows gdown to download again? I usually try versions 4.6.0, 5.1.0, or the latest (pip install --upgrade gdown). I also remove cookies.txt, and I end up (randomly) with a version that works. Hope this helps

ArianAmani commented 6 months ago

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Weirdly this worked for me too. And I was using the default gdown (pip install gdown) on colab using gdown --id ID without a problem until just a few days ago. Thanks!

ain-soph commented 3 months ago

@wkentaro I'm still suffering this issue today. (gdown version 5.2.0)

I'm trying to download my own uploaded dataset (10G) https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

And it raises the error

gdown https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

Failed to retrieve file url:

        Too many users have viewed or downloaded this file recently. Please
        try accessing the file again later. If the file you are trying to
        access is particularly large or is shared with many people, it may
        take up to 24 hours to be able to view or download the file. If you
        still can't access a file after 24 hours, contact your domain
        administrator.

You may still be able to access the file from the browser:

        https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

but Gdown can't. Please check connections and **permissions.**

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

wkentaro commented 3 months ago

@ain-soph It works for me now, so looks like too many requests?

image

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

We can't do this because the scope of gdown is without logging in.

this is another workaround: https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-i-still-cant-download

cleanerspam commented 1 month ago
#!/bin/bash

# Prompt for access token
read -p "Enter your Google Drive access token: " ACCESS_TOKEN

# Define the input file containing URLs
INPUT_FILE="file_urls.txt"

# Define the output directory
OUTPUT_DIR="/mnt/harddrive/gdrive"

# Check if the input file exists
if [[ ! -f "$INPUT_FILE" ]]; then
    echo "Input file '$INPUT_FILE' not found!"
    exit 1
fi

# Check if the output directory exists
if [[ ! -d "$OUTPUT_DIR" ]]; then
    echo "Output directory '$OUTPUT_DIR' does not exist!"
    exit 1
fi

# Loop through each line in the input file
while IFS= read -r URL; do
    # Extract the file ID from the URL
    FILE_ID=$(echo "$URL" | grep -oP 'd/\K[^/]+')

    # Check if FILE_ID was extracted successfully
    if [[ -z "$FILE_ID" ]]; then
        echo "Failed to extract file ID from URL: $URL"
        continue
    fi

    # Get the file metadata to retrieve the original name
    FILE_METADATA=$(curl -s -H "Authorization: Bearer $ACCESS_TOKEN" "https://www.googleapis.com/drive/v3/files/$FILE_ID?fields=name")
    ORIGINAL_NAME=$(echo "$FILE_METADATA" | jq -r '.name')

    # Check if the original name was retrieved successfully
    if [[ -z "$ORIGINAL_NAME" ]]; then
        echo "Failed to retrieve original name for file ID: $FILE_ID"
        continue
    fi

    # Use curl to download the file to the specified output directory
    OUTPUT_FILE="$OUTPUT_DIR/$ORIGINAL_NAME"
    echo "Downloading file with ID: $FILE_ID as '$ORIGINAL_NAME' to '$OUTPUT_FILE'"
    curl -H "Authorization: Bearer $ACCESS_TOKEN" -C - "https://www.googleapis.com/drive/v3/files/$FILE_ID?alt=media" -o "$OUTPUT_FILE"

    # Check if the download was successful
    if [[ $? -eq 0 ]]; then
        echo "Downloaded: $OUTPUT_FILE"
    else
        echo "Failed to download file with ID: $FILE_ID"
    fi
done < "$INPUT_FILE"

To anybody reading this

  1. Install jq from your distro package manager
  2. Change the output_location in script to something of your liking
  3. Save the above code as download_drive_files.sh and
  4. enter all links of files in a text file named file_urls.txt in the same directory as the script
  5. Get AccessToken by following guide below
Going to OAuth 2.0 Playground https://developers.google.com/oauthplayground/
In the Select the Scope box, paste https://www.googleapis.com/auth/drive.readonly
Click Authorize APIs and then Exchange authorization code for tokens
Copy the Access token

This will save all the files that were mentioned in the file_urls.txt , just make there is only one url per line.