search

herowinb / live-dl

Web GUI for Youtube-dl and Bash Script auto download/record upcoming live stream on Youtube and Twitch.
https://hub.docker.com/r/herowinb/live-dl
GNU Affero General Public License v3.0
29 stars 5 forks source link

Can't detect livestream #23

Closed challi1337 closed 1 year ago

challi1337 commented 1 year ago

Describe the bug Doesn't seem to find livestream (livestream is indeed live)

To Reproduce

  1. in auto.sh nohup /bin/bash /usr/src/app/live-dl https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ --debug
  2. See below for issue 
    
logs:
[Wed Jul 19 17:06:37 -01 2023] DEBUG: Live-dl is up to date
[Wed Jul 19 17:06:37 -01 2023] URL (URI) to process           : https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ
[Wed Jul 19 17:06:37 -01 2023] Mode                           : download  
[Wed Jul 19 17:06:37 -01 2023] Output base dir                : /youtube-dl/Downloads
[Wed Jul 19 17:06:37 -01 2023] Separate folder for membership : true
[Wed Jul 19 17:06:37 -01 2023] Remove Title                   : false
[Wed Jul 19 17:06:37 -01 2023] Archive chat                   : false - Mode: 
[Wed Jul 19 17:06:37 -01 2023] Keyword filter                 : false
[Wed Jul 19 17:06:37 -01 2023] Run interval (when loop)       : 30
[Wed Jul 19 17:06:37 -01 2023] Long interval (when loop)      : 60
[Wed Jul 19 17:06:37 -01 2023] Skip embedding metadata        : true
[Wed Jul 19 17:06:37 -01 2023] Skip email notification        : true
[Wed Jul 19 17:06:37 -01 2023] Skip Slack notification        : true
[Wed Jul 19 17:06:37 -01 2023] Skip Telegram notification     : true
[Wed Jul 19 17:06:37 -01 2023] Skip Discord notification      : true
[Wed Jul 19 17:06:37 -01 2023] Callback                       : false
[Wed Jul 19 17:06:37 -01 2023] Address Pool                   : false
[Wed Jul 19 17:06:37 -01 2023] Email provider                 : ses
[Wed Jul 19 17:06:37 -01 2023] One-time execution             : false
[Wed Jul 19 17:06:37 -01 2023] No log mode                    : 
[Wed Jul 19 17:06:37 -01 2023] YouTube URL detected
[Wed Jul 19 17:06:37 -01 2023] YouTube video detected (alt)
[Wed Jul 19 17:06:37 -01 2023] Checking https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ/live...
[Wed Jul 19 17:06:37 -01 2023] DEBUG: start func_check_state
[Wed Jul 19 17:06:37 -01 2023] DEBUG: Cookies using: ./config/cookies.txt
[Wed Jul 19 17:06:37 -01 2023] DEBUG: start func_check_newlivestream_ytb
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Can not detect living stream, checking schedule streams
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Can not detect scheduled stream which scheduled time less than 2h
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Channel page is offline with no JSON extracted
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Found long interval setting 60
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Not valid at the moment, continue running
[Wed Jul 19 17:06:38 -01 2023] DEBUG: Stream unavailable, retry in 60s...
[Wed Jul 19 17:07:38 -01 2023] DEBUG: start func_check_state
[Wed Jul 19 17:07:38 -01 2023] DEBUG: Cookies using: ./config/cookies.txt
[Wed Jul 19 17:07:38 -01 2023] DEBUG: start func_check_newlivestream_ytb
[Wed Jul 19 17:07:38 -01 2023] DEBUG: Can not detect living stream, checking schedule streams
[Wed Jul 19 17:07:38 -01 2023] DEBUG: Can not detect scheduled stream which scheduled time less than 2h
[Wed Jul 19 17:07:39 -01 2023] DEBUG: Channel page is offline with no JSON extracted
[Wed Jul 19 17:07:39 -01 2023] DEBUG: Found long interval setting 60
[Wed Jul 19 17:07:39 -01 2023] DEBUG: Not valid at the moment, continue running
[Wed Jul 19 17:07:39 -01 2023] DEBUG: Stream unavailable, retry in 60s...
[Wed Jul 19 17:08:39 -01 2023] DEBUG: start func_check_state
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Cookies using: ./config/cookies.txt
[Wed Jul 19 17:08:39 -01 2023] DEBUG: start func_check_newlivestream_ytb
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Can not detect living stream, checking schedule streams
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Can not detect scheduled stream which scheduled time less than 2h
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Channel page is offline with no JSON extracted
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Found long interval setting 60
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Not valid at the moment, continue running
[Wed Jul 19 17:08:39 -01 2023] DEBUG: Stream unavailable, retry in 60s...

Desktop (please complete the following information):

herowinb commented 1 year ago

I run on my docker it still work, https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ/live still redirect to https://www.youtube.com/watch?v=7FG1kbnxdqw so it must work like before. I really don't know what happened with your run. Can you re-run it again ?


[Wed Jul 19 18:13:50 UTC 2023] DEBUG: Live-dl is up to date
[Wed Jul 19 18:13:50 UTC 2023] URL (URI) to process           : https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ
[Wed Jul 19 18:13:50 UTC 2023] Mode                           : download - Using YTARCHIVE
[Wed Jul 19 18:13:50 UTC 2023] Output base dir                : /youtube-dl/VTuber Recordings
[Wed Jul 19 18:13:50 UTC 2023] Separate folder for membership : true
[Wed Jul 19 18:13:50 UTC 2023] Remove Title                   : false
[Wed Jul 19 18:13:50 UTC 2023] Archive chat                   : false - Mode: simple
[Wed Jul 19 18:13:50 UTC 2023] Keyword filter                 : false
[Wed Jul 19 18:13:50 UTC 2023] Run interval (when loop)       : 30
[Wed Jul 19 18:13:50 UTC 2023] Long interval (when loop)      : 900
[Wed Jul 19 18:13:50 UTC 2023] Skip embedding metadata        : true
[Wed Jul 19 18:13:50 UTC 2023] Skip email notification        : false
[Wed Jul 19 18:13:50 UTC 2023] Skip Slack notification        : true
[Wed Jul 19 18:13:50 UTC 2023] Skip Telegram notification     : false
[Wed Jul 19 18:13:50 UTC 2023] Skip Discord notification      : false
[Wed Jul 19 18:13:50 UTC 2023] Callback                       : false
[Wed Jul 19 18:13:50 UTC 2023] Address Pool                   : false
[Wed Jul 19 18:13:50 UTC 2023] Email provider                 : mailgun
[Wed Jul 19 18:13:50 UTC 2023] One-time execution             : false
[Wed Jul 19 18:13:50 UTC 2023] No log mode                    : false
[Wed Jul 19 18:13:50 UTC 2023] YouTube URL detected
[Wed Jul 19 18:13:50 UTC 2023] YouTube video detected (alt)
[Wed Jul 19 18:13:50 UTC 2023] Checking https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ/live...
[Wed Jul 19 18:13:50 UTC 2023] DEBUG: start func_check_state
[Wed Jul 19 18:13:50 UTC 2023] DEBUG: Cookies using: ./config/cookies.txt
[Wed Jul 19 18:13:50 UTC 2023] DEBUG: start func_check_newlivestream_ytb
[Wed Jul 19 18:13:50 UTC 2023] DEBUG: Can not detect living stream, checking schedule streams
[Wed Jul 19 18:13:50 UTC 2023] DEBUG: Can not detect scheduled stream which scheduled time less than 2h
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Got valid cURL metadata
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Is live: true
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Playability: OK
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Playability Reason:
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Streamability ID: 7FG1kbnxdqw
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Scheduled Start Time:
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Channel: 배라소니
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Title: 너 재울때까지 안잘꺼야
[Wed Jul 19 18:13:51 UTC 2023] cURL check seems goes live now, continue checking...
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: No keyword filter run, mark as downloadable...
[Wed Jul 19 18:13:51 UTC 2023] DEBUG: Current state for first check: live
[Wed Jul 19 18:13:51 UTC 2023] Re-checking via yt-dlp...
[Wed Jul 19 18:13:51 UTC 2023] Get real video url: https://www.youtube.com/watch?v=7FG1kbnxdqw
[Wed Jul 19 18:13:53 UTC 2023] DEBUG: Got valid yt-dlp metadata
[Wed Jul 19 18:13:53 UTC 2023] DEBUG: Channel name mapping not found, consider adding it in config.yml
[Wed Jul 19 18:13:53 UTC 2023] DEBUG: I will use 배라소니 for folder name, embedding metadata, etc.
[Wed Jul 19 18:13:53 UTC 2023] DEBUG: Telegram channel not found, consider adding it in config.yml
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Got state: live
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Current state for second check: live
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Found long interval setting 900
[Wed Jul 19 18:13:54 UTC 2023] Prepare downloading...
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Download platform: YouTube
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Download type: channel
[Wed Jul 19 18:13:54 UTC 2023] DEBUG: Download mode: download

ID:        7FG1kbnxdqw
Title:     너 재울때까지 안잘꺼야
URL:       https://www.youtube.com/watch?v=7FG1kbnxdqw
Author:    배라소니
Channel:   https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ
Date:      20230719
Thumbnail: https://i.ytimg.com/vi/7FG1kbnxdqw/maxresdefault_live.jpg
Filename:  20230719 너 재울때까지 안잘꺼야 (7FG1kbnxdqw).1
Location:  /youtube-dl/VTuber Recordings/배라소니/
Hostname:  cb55ea3019ed

[Wed Jul 19 18:13:54 UTC 2023] No specific channel set for current user, skip sending Telegram notification.
[Wed Jul 19 18:13:54 UTC 2023] Sending email notification...
[Wed Jul 19 18:13:54 UTC 2023] Sending Discord notification... ```
challi1337 commented 1 year ago

I reset image and ran again, same result sadly :/

[Wed Jul 19 17:19:19 -01 2023] DEBUG: Live-dl is up to date [Wed Jul 19 17:19:19 -01 2023] URL (URI) to process : https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ [Wed Jul 19 17:19:19 -01 2023] Mode : download
[Wed Jul 19 17:19:19 -01 2023] Output base dir : /youtube-dl/Downloads [Wed Jul 19 17:19:19 -01 2023] Separate folder for membership : true [Wed Jul 19 17:19:19 -01 2023] Remove Title : false [Wed Jul 19 17:19:19 -01 2023] Archive chat : false - Mode: [Wed Jul 19 17:19:19 -01 2023] Keyword filter : false [Wed Jul 19 17:19:19 -01 2023] Run interval (when loop) : 30 [Wed Jul 19 17:19:19 -01 2023] Long interval (when loop) : 60 [Wed Jul 19 17:19:19 -01 2023] Skip embedding metadata : true [Wed Jul 19 17:19:19 -01 2023] Skip email notification : true [Wed Jul 19 17:19:19 -01 2023] Skip Slack notification : true [Wed Jul 19 17:19:19 -01 2023] Skip Telegram notification : true [Wed Jul 19 17:19:19 -01 2023] Skip Discord notification : true [Wed Jul 19 17:19:19 -01 2023] Callback : false [Wed Jul 19 17:19:19 -01 2023] Address Pool : false [Wed Jul 19 17:19:19 -01 2023] Email provider : ses [Wed Jul 19 17:19:19 -01 2023] One-time execution : false [Wed Jul 19 17:19:19 -01 2023] No log mode : [Wed Jul 19 17:19:19 -01 2023] YouTube URL detected [Wed Jul 19 17:19:19 -01 2023] YouTube video detected (alt) [Wed Jul 19 17:19:19 -01 2023] Checking https://www.youtube.com/channel/UCDvazhEPkMkIsDJGZsGw1zQ/live... [Wed Jul 19 17:19:19 -01 2023] DEBUG: start func_check_state [Wed Jul 19 17:19:19 -01 2023] DEBUG: Cookies using: ./config/cookies.txt [Wed Jul 19 17:19:19 -01 2023] DEBUG: start func_check_newlivestream_ytb [Wed Jul 19 17:19:20 -01 2023] DEBUG: Can not detect living stream, checking schedule streams [Wed Jul 19 17:19:20 -01 2023] DEBUG: Can not detect scheduled stream which scheduled time less than 2h [Wed Jul 19 17:19:20 -01 2023] DEBUG: Channel page is offline with no JSON extracted [Wed Jul 19 17:19:20 -01 2023] DEBUG: Found long interval setting 60 [Wed Jul 19 17:19:20 -01 2023] DEBUG: Not valid at the moment, continue running [Wed Jul 19 17:19:20 -01 2023] DEBUG: Stream unavailable, retry in 60s...

edit: I tried download same URL +/live with youtube-dlp and works fine so I'm not blocked by youtube

herowinb commented 1 year ago

Can you help me to check this:

challi1337 commented 1 year ago

In browser it goes directly to the stream.

On incognito browser this happens: image

edit: since the screenshot was related to cookies, I was thinking and I had to relog into my google account recently, I just got a new cookie file and now it works fine!

Unrelated: do you think it could be possible for live-dl support --live-from-start argument that yt-dlp has?

Issue can be closed

herowinb commented 1 year ago

Look like it required some basic cookies in your region. I will check if I can make it bypass this required cookies but not sure it could work.

Unrelated: do you think it could be possible for live-dl support --live-from-start argument that yt-dlp has?

Live-dl now using yt-dlp and --live-from-start at default, same as ytarchive option. It just don't work on Twitch.

herowinb commented 1 year ago

You don't need a cookies.txt with login accounts. You can extract a basic cookies (just info your browser using accepted the ToS) follow this steps, I'm using Firefox but it should be the same on Chrome:

Make sure in your cookies.txt file have this line .youtube.com TRUE / TRUE <time> CONSENT PENDING+<number>

kyngs commented 2 months ago

You don't need a cookies.txt with login accounts. You can extract a basic cookies (just info your browser using accepted the ToS) follow this steps, I'm using Firefox but it should be the same on Chrome:

* Install add-on cookies.txt, enable add-on in private mode.

* Open browser in private mode, go to Youtube and accept ToS. Extract `cookies.txt` and replace cookies file in config folder. Make sure you not replace the `cookies_membership.txt`

Make sure in your cookies.txt file have this line .youtube.com TRUE / TRUE <time> CONSENT PENDING+<number>

How often do you have to repeat this process?

herowinb commented 2 months ago

You don't need a cookies.txt with login accounts. You can extract a basic cookies (just info your browser using accepted the ToS) follow this steps, I'm using Firefox but it should be the same on Chrome:

* Install add-on cookies.txt, enable add-on in private mode.

* Open browser in private mode, go to Youtube and accept ToS. Extract `cookies.txt` and replace cookies file in config folder. Make sure you not replace the `cookies_membership.txt`

Make sure in your cookies.txt file have this line .youtube.com TRUE / TRUE <time> CONSENT PENDING+<number>

How often do you have to repeat this process?

Tbh, I don't know since I don't live in EU so I don't have this issue. Did you have problem because cookies.txt expired in short time (under 1h) ?

kyngs commented 2 months ago

Not yet, I just did this, but I'm wondering cuz I'm pretty sure those cookies won't last forever.