search

wader / goutubedl

Go wrapper for youtube-dl and yt-dlp
https://pkg.go.dev/github.com/wader/goutubedl?tab=doc
MIT License
94 stars 26 forks source link

download from youtube: [youtube] qCyb-D0XTXI: Sign in to confirm you’re not a bot. #205

Open TeddyKahwaji opened 1 month ago

TeddyKahwaji commented 1 month ago

Summary

Youtube recently updated thier apis and it's breaking lots of yt-dlp applications

Running your package on any cloud instance will result in:

download from youtube: [youtube] qCyb-D0XTXI: Sign in to confirm you’re not a bot.

I'd like if we could integrate youtube-oauth2 within this package so we can bypass this issue

wader commented 1 month ago

Hey! this looks similar to #187? could you try using the netrc optiin?

TeddyKahwaji commented 1 month ago

Tried using the netrc option however it isn't working. The YouTube-oauth2 documentation states that the install has to be a yt-dlp nightly, which doesn't seem to currently work with this package

wader commented 1 month ago

Hmm in the linked issue i think nightly is used. The go module has no checks on which version of yt-dlp your using so should work unless there is some breaking change in nightly

TeddyKahwaji commented 1 month ago

I had some issue with the module and the nightly version but it might of been a broken nightly build. I'll try it out again with a different version and share my results later.

Out of curiosity in regard to .netrc file, my server runs on the cloud does the .netrc file still get detected if it's generated during runtime?

wader commented 1 month ago

👍 dont think that should be a problem. If inconvinient maybe you want to contribute a username/password PR?

TeddyKahwaji commented 1 month ago

I'm not seeing this injecting itself even with manually modifying source code to include username and password and using a .netrc file

for context this is how I have the docker file set up for getting these packages 

RUN curl -L https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/download/2024.09.26.232938/yt-dlp -o /usr/local/bin/yt-dlp \
    && chmod a+x /usr/local/bin/yt-dlp

RUN mkdir -p ~/.config/yt-dlp/plugins/

# Install yt-dlp OAuth2 plugin
RUN curl -L https://github.com/coletdjnz/yt-dlp-youtube-oauth2/releases/download/v2024.9.14/yt-dlp-youtube-oauth2.zip -o /usr/local/bin/yt-dlp-oauth2.zip && \
    unzip /usr/local/bin/yt-dlp-oauth2.zip -d ~/.config/yt-dlp/plugins/ && \
    chmod -R a+x ~/.config/yt-dlp/plugins/ && \
    rm /usr/local/bin/yt-dlp-oauth2.zip
wader commented 1 month ago

Ok but you can make it work using the cli directly?

wader commented 1 month ago

Hmm seems like the plugin is not found. If i add an additional directory, as the yt-dlp docs says https://github.com/yt-dlp/yt-dlp#installing-plugins, quite confusing, it works:

FROM debian:latest

RUN apt-get update && apt-get -y install curl unzip python3

RUN curl -L https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/download/2024.09.27.232842/yt-dlp -o /usr/local/bin/yt-dlp \
    && chmod a+x /usr/local/bin/yt-dlp

RUN mkdir -p ~/.config/yt-dlp/plugins/

# Install yt-dlp OAuth2 plugin
RUN curl -L https://github.com/coletdjnz/yt-dlp-youtube-oauth2/releases/download/v2024.9.14/yt-dlp-youtube-oauth2.zip -o /usr/local/bin/yt-dlp-oauth2.zip && \
    mkdir -p ~/.config/yt-dlp-plugins/something/ && \
    unzip /usr/local/bin/yt-dlp-oauth2.zip -d ~/.config/yt-dlp-plugins/something/ && \
    chmod -R a+x ~/.config/yt-dlp/plugins/ && \
    rm /usr/local/bin/yt-dlp-oauth2.zip

RUN find $HOME

RUN yt-dlp --verbose

Log:

$ docker build --progress=plain .
...
#9 [6/7] RUN find $HOME
#9 0.145 /root
#9 0.145 /root/.bashrc
#9 0.145 /root/.profile
#9 0.145 /root/.config
#9 0.145 /root/.config/yt-dlp
#9 0.145 /root/.config/yt-dlp/plugins
#9 0.145 /root/.config/yt-dlp-plugins
#9 0.145 /root/.config/yt-dlp-plugins/something
#9 0.145 /root/.config/yt-dlp-plugins/something/yt_dlp_plugins
#9 0.145 /root/.config/yt-dlp-plugins/something/yt_dlp_plugins/extractor
#9 0.145 /root/.config/yt-dlp-plugins/something/yt_dlp_plugins/extractor/youtubeoauth.py
#9 DONE 0.1s

#10 [7/7] RUN yt-dlp --verbose

...
#10 0.433 [debug] Extractor Plugins: oauth2 (YoutubeClipIE), oauth2 (YoutubeConsentRedirectIE), oauth2 (YoutubeFavouritesIE), oauth2 (YoutubeIE), oauth2 (YoutubeMusicSearchURLIE), oauth2 (YoutubeNotificationsIE), oauth2 (YoutubeSearchDateIE), oauth2 (YoutubeSearchIE), oauth2 (YoutubeSearchURLIE), oauth2 (YoutubeTabIE)
#10 0.433 [debug] Plugin directories: ['/root/.config/yt-dlp-plugins/something/yt_dlp_plugins']
...
TeddyKahwaji commented 1 month ago

Hey thanks for your reply; so I got that working in my docker container the other day and I'm able to execute it when entering the shell, I've go vendored this module and allowed for a username & password option. The issue I'm running into is the 

[youtube+oauth2] To give yt-dlp access to your account, go to https://www.google.com/device and enter code XXX-YYY-ZZZ

Which outputs in the shell does not output in this modules DebugLog not sure if I'm missing something

wader commented 1 month ago

Hmm looks like that message should be printed to stdout unless in quiet mode (which i don't think is used) https://github.com/yt-dlp/yt-dlp/blob/5945fc1945a4001537072e39f03725f944437834/yt_dlp/YoutubeDL.py#L646C51-L646C56, so i would expect it to work with DebugLog. Do you see other log messages? and to you see plugins gets loaded when using the go module? maybe that gets logged to stderr, then you would have to use StderrFn: func(cmd *exec.Cmd) io.Writer { return os.Stderr } etc

TeddyKahwaji commented 1 month ago

So here is the current output

Seems like the username & password is not being accepted? 

spicetunes-1  | 2024-09-29T18:04:20.577Z        INFO    goutubedl/goutubedl.go:399      cmd [/usr/local/bin/yt-dlp --ignore-errors --no-call-home --restrict-filenames --batch-file - --dump-single-json --username oauth2 --password "" --no-playlist]
spicetunes-1  | Reading URLs from STDIN - EOF (Ctrl+D) to end:
spicetunes-1  | WARNING: [youtube] Login with password is not supported for this website. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies
spicetunes-1  | 2024-09-29T18:04:22.889Z        INFO    goutubedl/goutubedl.go:697      cmd [/usr/local/bin/yt-dlp --ignore-errors --no-call-home --newline --restrict-filenames --output - --load-info /tmp/ydls3358470594/info.json --username oauth2 --password  -f best -x]
TeddyKahwaji commented 1 month ago

Here is my docker set up for reference 

FROM golang:1.23 AS builder

WORKDIR /usr/src/app

COPY go.mod go.sum ./

RUN go mod download

COPY . .

RUN go build -o /go/bin/app ./cmd

FROM golang:1.23

WORKDIR /usr/src/app

RUN useradd -m docker && echo "docker:docker" | chpasswd && adduser docker sudo

RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates ffmpeg && \
    apt-get clean autoclean && \
    rm -rf /var/lib/apt/lists/*

RUN curl -L https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/download/2024.09.27.232842/yt-dlp -o /usr/local/bin/yt-dlp \
    && chmod a+x /usr/local/bin/yt-dlp

COPY --from=builder /go/bin/app /go/bin/app

EXPOSE 8080

CMD ["/go/bin/app"]
wader commented 1 month ago

Ah you probably have to add -v for goutubedl know if there is any [debug] Extractor Plugins: oauth2 lines. I would make sure that happens first.

TeddyKahwaji commented 1 month ago 
INFO    cmd/main.go:141 Bot has connected
spicetunes-1  | 2024-09-29T18:30:07.637Z        INFO    goutubedl/goutubedl.go:400      cmd [/usr/local/bin/yt-dlp --ignore-errors --verbose --no-call-home --restrict-filenames --batch-file - --dump-single-json --username "oauth2" --password "" --no-playlist]
spicetunes-1  | [debug] Command-line config: ['--ignore-errors', '--verbose', '--no-call-home', '--restrict-filenames', '--batch-file', '-', '--dump-single-json', '--username', 'PRIVATE', '--password', 'PRIVATE', '--no-playlist']
spicetunes-1  | Reading URLs from STDIN - EOF (Ctrl+D) to end:
spicetunes-1  | [debug] Batch file urls: ['https://www.youtube.com/watch?v=b65MoVwANq4']
spicetunes-1  | [debug] Encodings: locale utf-8, fs utf-8, pref utf-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
spicetunes-1  | [debug] yt-dlp version nightly@2024.09.27.232842 from yt-dlp/yt-dlp-nightly-builds [5945fc194] (zip)
spicetunes-1  | [debug] Python 3.11.2 (CPython aarch64 64bit) - Linux-6.10.4-linuxkit-aarch64-with-glibc2.36 (OpenSSL 3.0.14 4 Jun 2024, glibc 2.36)
spicetunes-1  | [debug] exe versions: ffmpeg 5.1.6-0 (setts), ffprobe 5.1.6-0
spicetunes-1  | [debug] Optional libraries: sqlite3-3.40.1
spicetunes-1  | [debug] Proxy map: {}
spicetunes-1  | [debug] Request Handlers: urllib
spicetunes-1  | [debug] Loaded 1838 extractors
spicetunes-1  | WARNING: [youtube] Login with password is not supported for this website. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies
wader commented 1 month ago

Ok so oauth2 plugin is not found when yt-dlp is executed via goutubedl?

TeddyKahwaji commented 1 month ago

oh it's working now 

spicetunes-1  | 2024-09-29T18:37:42.411Z        INFO    cmd/main.go:141 Bot has connected
spicetunes-1  | 2024-09-29T18:37:48.858Z        INFO    goutubedl/goutubedl.go:398      cmd [/usr/local/bin/yt-dlp --verbose --no-call-home --restrict-filenames --batch-file - --dump-single-json --username "oauth2" --password "" --no-playlist]
spicetunes-1  | [debug] Command-line config: ['--verbose', '--no-call-home', '--restrict-filenames', '--batch-file', '-', '--dump-single-json', '--username', 'PRIVATE', '--password', 'PRIVATE', '--no-playlist']
spicetunes-1  | Reading URLs from STDIN - EOF (Ctrl+D) to end:
spicetunes-1  | [debug] Batch file urls: ['https://www.youtube.com/watch?v=b65MoVwANq4']
spicetunes-1  | [debug] Encodings: locale utf-8, fs utf-8, pref utf-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
spicetunes-1  | [debug] yt-dlp version nightly@2024.09.27.232842 from yt-dlp/yt-dlp-nightly-builds [5945fc194] (zip)
spicetunes-1  | [debug] Python 3.11.2 (CPython aarch64 64bit) - Linux-6.10.4-linuxkit-aarch64-with-glibc2.36 (OpenSSL 3.0.14 4 Jun 2024, glibc 2.36)
spicetunes-1  | [debug] exe versions: ffmpeg 5.1.6-0 (setts), ffprobe 5.1.6-0
spicetunes-1  | [debug] Optional libraries: sqlite3-3.40.1
spicetunes-1  | [debug] Proxy map: {}
spicetunes-1  | [debug] Request Handlers: urllib
spicetunes-1  | [debug] Extractor Plugins: oauth2 (YoutubeClipIE), oauth2 (YoutubeConsentRedirectIE), oauth2 (YoutubeFavouritesIE), oauth2 (YoutubeIE), oauth2 (YoutubeMusicSearchURLIE), oauth2 (YoutubeNotificationsIE), oauth2 (YoutubeSearchDateIE), oauth2 (YoutubeSearchIE), oauth2 (YoutubeSearchURLIE), oauth2 (YoutubeTabIE)
spicetunes-1  | [debug] Plugin directories: ['/root/.config/yt-dlp-plugins/something/yt_dlp_plugins']
spicetunes-1  | [debug] Loaded 1838 extractors
spicetunes-1  | [debug] YouTube OAuth2 plugin version 2024.09.14
spicetunes-1  | [youtube+oauth2] Extracting URL: https://www.youtube.com/watch?v=b65MoVwANq4
spicetunes-1  | [youtube+oauth2] b65MoVwANq4: Downloading webpage
spicetunes-1  | [youtube+oauth2] b65MoVwANq4: Downloading ios player API JSON
spicetunes-1  | [youtube+oauth2] b65MoVwANq4: Downloading web creator player API JSON
spicetunes-1  | [youtube+oauth2] b65MoVwANq4: Downloading player b0557ce3
spicetunes-1  | [debug] Saving youtube-nsig.b0557ce3 to cache
TeddyKahwaji commented 1 month ago

however i'm not seeing this pop up 

[youtube+oauth2] To give yt-dlp access to your account, go to https://www.google.com/device and enter code XXX-YYY-ZZZ

Which is what is linked in their readme

wader commented 1 month ago

You mean pop up as an actual pop up windows or your not seeing the log line? but your using username/password for oauth2 now?

TeddyKahwaji commented 1 month ago

Oh I got it working so I was passing username and password incorrectly here is the output now 

picetunes-1  | [debug] Plugin directories: ['/root/.config/yt-dlp-plugins/something/yt_dlp_plugins']
spicetunes-1  | [debug] Loaded 1838 extractors
spicetunes-1  | [debug] YouTube OAuth2 plugin version 2024.09.14
spicetunes-1  | [youtube+oauth2] oauth2: Initializing OAuth2 Authorization Flow
spicetunes-1  | [youtube+oauth2] To give yt-dlp access to your account, go to  https://www.google.com/device  and enter code  LXDZ-XLVS

All is good ty ty

wader commented 1 month ago

👍 Ok, so did you end up with any change you would like to contribute? just send a PR if so

TeddyKahwaji commented 1 month ago

I don't believe I have the permissions to send PR

TeddyKahwaji commented 1 month ago

can't push my branch 

 goutubedl git:(update-options) git push origin update-options 
remote: Permission to wader/goutubedl.git
wader commented 1 month ago

can't push my branch

 goutubedl git:(update-options) git push origin update-options 
remote: Permission to wader/goutubedl.git

You need to create a fork and then push the branch to your fork, then create a PR

wader commented 1 month ago

Close issue? let me know if you have problems creating a PR

TeddyKahwaji commented 1 month ago

Will do, i'm actually noticing when the code runs on a vps, it'll error out on and give that sign in issue prior to yt-dlp-oauth2 being able to pop up. 

VwANq4: Sign in to confirm you’re not a bot. This helps protect our community. Learn more
2024-10-02T00:29:27.247307+00:00 app[worker.1]:   File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 741, in extract
2024-10-02T00:29:27.247308+00:00 app[worker.1]:     ie_result = self._real_extract(url)
2024-10-02T00:29:27.247308+00:00 app[worker.1]:                 ^^^^^^^^^^^^^^^^^^^^^^^
2024-10-02T00:29:27.247309+00:00 app[worker.1]:   File "/usr/local/bin/yt-dlp/yt_dlp/extractor/youtube.py", line 4468, in _real_extract
2024-10-02T00:29:27.247309+00:00 app[worker.1]:     self.raise_no_formats(reason, expected=True)
2024-10-02T00:29:27.247310+00:00 app[worker.1]:   File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 1275, in raise_no_formats
2024-10-02T00:29:27.247310+00:00 app[worker.1]:     raise ExtractorError(msg, expected=expected, video_id=video_id)
2024-10-02T00:29:27.247311+00:00 app[worker.1]: 
2024-10-02T00:29:27.286020+00:00 app[worker.1]: {"level":"warn","ts":1727828967.2859464,"caller":"music/music.go:88","msg":"error playing audio","guild_id":"819412972924305409","error":"downloading result: attempting to download from youtube: [youtube] b65MoVwANq4: Sign in to confirm you’re not a bot. This helps protect our community. Learn more"}
wader commented 1 month ago

Maybe a yt-dlp bug?