search

madiele / vod2pod-rss

Vod2Pod-RSS converts a YouTube or Twitch channel into a podcast with ease. It creates a podcast RSS that can be listened to directly inside any podcast client. VODs are transcoded to MP3 on the fly and no server storage is needed!
MIT License
199 stars 5 forks source link

streams no longer available right the way #131

Closed Eisaichen closed 8 months ago

Eisaichen commented 8 months ago

app version: 1.1.0

Environment: Docker / Linux vod2pod 1.1.0

when playing the new feed on a client like an RSS reader or Google Podcast for the first time, server will not respond. This log will show up on the server when the client clicks play: 2023-11-23T21:32:57.800Z INFO [vod2pod_rss::server] processing transcode at 320k for https://www.youtube.com/watch?v=fvJaGKNGKN4 4 minutes later (depending on the length of the video), this log showed up indicating the stream was ready: 2023-11-23T21:37:19.812Z INFO [vod2pod_rss::transcoder] generated ffmpeg command: And this problem did not exist a couple of months ago.

another minor issue is, the environment variable for setting the bitrate seems not working. In the readme.md, the name of that environment variable is "BITRATE", however, in the docker-compose.yaml, is "MP3_BITRATE". No matter which is the right one, the server will transcode to 320k regardless.

logs ``` 2023-11-23T21:32:50.618Z INFO [app] app version 1.1.0 2023-11-23T21:32:50.619Z INFO [app] listening on http://0.0.0.0:8080 2023-11-23T21:32:50.619Z INFO [actix_server::builder] starting 12 workers 2023-11-23T21:32:50.619Z INFO [actix_server::server] Actix runtime found; starting in Actix runtime 2023-11-23T21:32:57.800Z INFO [vod2pod_rss::server] processing transcode at 320k for https://www.youtube.com/watch?v=fvJaGKNGKN4 2023-11-23T21:37:19.812Z INFO [vod2pod_rss::transcoder] generated ffmpeg command: < MORE THAN 4 MINUTES LATER ffmpeg -ss 0 -i https://rr1---sn-a5meknsy.googlevideo.com/videoplayback?expire=1700797033&ei=CcZfZYTXMf67sfIP7emBQA&ip=99.113.211.182&id=o-AMcrWha1mF-lrej7F4OLKU2zPMGUB59KYwLjb_3-URTd&itag=251&source=youtube&requiressl=yes&mh=CL&mm=31%2C26&mn=sn-a5meknsy%2Csn-o097znsl&ms=au%2Conr&mv=m&mvi=1&pl=18&initcwndbps=842500&spc=UWF9f62sKa6lv09a_XZYMUJTIsjdn8c&vprv=1&svpuc=1&mime=audio%2Fwebm&gir=yes&clen=2837460&dur=162.561&lmt=1700622448980383&mt=1700774929&fvip=5&keepalive=yes&fexp=24007246&c=ANDROID&txp=5532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ANLwegAwRgIhANLzrq73JU8mvWSt0qzozYywEYz1ES7pQACQDqI9itvpAiEApKK_GZdcZlruUKmD55st9dsNZ2wyRAOd3OqkUCRilZg%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AM8Gb2swRgIhANnfrTFQXL5U6iiUSCVGpouOk85q9SNDLjIl8WSC68iLAiEA4fIuhaK5kfNJYslW3V4QXfF2oOC-KDo6PPSuI39UhHE%3D -acodec libmp3lame -ab 320k -f mp3 -bufsize 9600 -maxrate 9600k -timeout 300 -hide_banner -loglevel error pipe:stdout 2023-11-23T21:37:19.812Z INFO [vod2pod_rss::transcoder] streaming to client 2023-11-23T21:37:20.316Z INFO [vod2pod_rss::transcoder] connection to client dropped, stopping transcode ```
env ``` PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=vod2pod-6f67f6b946-dsrlv REDIS_ADDRESS=redis.core.svc.cluster.local REDIS_PORT=6379 YT_API_KEY=************************** TZ=America/Los_Angeles BITRATE=192 MP3_BITRATE=192 TRANSCODE=true SUBFOLDER=/ RUST_LOG=INFO HOME=/root ```
madiele commented 8 months ago

Thanks for the segnalation of the wrong env in the Readme, fixed it, the correct name is MP3_BITRATE, I will investigate the fact that it does not work for you.

For the 4 minutes delay I doubt it's just yt-dlp being slow for some reason, my instance does not have this problem, maybe youtube is breaking something, I'll give it a look when I have time though.

madiele commented 8 months ago

I'll write here a checklist of todo to improve logging to make troubleshooting envs and bottleneck problems easily. Once done I'll ping you so you can try again with the new logs

@Eisaichen In the meanwhile does this issue happen only on 1.1.0 for you? If you use the precompiled container image you can try the previous version by changing latest to 1.0.5, in 1.1.0 I changed lots of the internals so I don't exclude a new bug from the rewrite

Eisaichen commented 8 months ago

I'll write here a checklist of todo to improve logging to make troubleshooting envs and bottleneck problems easily. Once done I'll ping you so you can try again with the new logs

  • [ ] print all settings on startup (only show if set or unset for sensitive infos)
  • [ ] print a log before each external dep. call (yt-dlp, apis, etc.)

@Eisaichen In the meanwhile does this issue happen only on 1.1.0 for you? If you use the precompiled container image you can try the previous version by changing latest to 1.0.5, in 1.1.0 I changed lots of the internals so I don't exclude a new bug from the rewrite

Thank you so much for the fast reply! For the bitrate issue, I didn't change a thing, but it's somehow fixed now, sorry for bothering you. I don't know when the setting lost effect because I always set the bitrate to 320k in the past, I only changed it recently to troubleshoot the delay issue.

And about the delay issue, is complicated. v1.0.5 was running well, until a couple of months ago, it completely stopped working even though it's the same v1.0.5. errors with "URL parsing" alike. I didn't change anything because it was running like a charm before, and the docker image is also the same, so I think YouTube changed something and can only be fixed by update the components like yt-dlp. later, I found you updated the docker-compose file in the master branch and released a v1.1.0 beta image. So I upgarde to v1.1.0 beta with the new podtube removed compose file. This fixes the previous issue and makes vod2pod usable again. but also introducing the delay issue. Now I'm running the image with latest tag and the issue persists.

Here are the logs and my config, please notice I set the log level to debug, so it's really long. (Client were Google Podcast Web) logs.txt

config ``` initContainers: - name: init-redis image: busybox:1.28 command: ['sh', '-c', "until nc -z redis.core.svc.cluster.local 6379; do echo waiting for redis; sleep 2; done"] containers: - name: vod2pod image: docker.io/madiele/vod2pod-rss imagePullPolicy: "Always" ports: - containerPort: 8080 name: http env: - name: TZ value: "America/Los_Angeles" - name: BITRATE value: "192" - name: TRANSCODE value: "true" - name: SUBFOLDER value: "/" - name: RUST_LOG value: "DEBUG" - name: REDIS_ADDRESS value: "redis.core.svc.cluster.local" - name: REDIS_PORT value: "6379" - name: YT_API_KEY valueFrom: secretKeyRef: name: yt-api-key key: key # - name: TWITCH_SECRET # value: "${TWITCH_SECRET:-}" # - name: TWITCH_CLIENT_ID # value: "${TWITCH_CLIENT_ID:-}" enableServiceLinks: false ```

Every headers from the Internet will be forwarded to vod2pod with the following begin added.

    forceSTSHeader: true
    stsIncludeSubdomains: false
    stsPreload: true
    stsSeconds: 31536000
madiele commented 8 months ago

Thanks for the debug logs! By checking the logs it does seem like the culprit is yt-dlp taking too long to fetch the stream url, to verify this can you try to lunch this command from inside the container by attaching to it using bash?

sudo docker exec -it <container> /bin/bash

The when inside do

yt-dlp -f bestaudio --get-url https://www.youtube.com/watch?v=BWcp9-4TLVQ

If this takes 4 minutes we have the problem, if it does try to see if removing -f or changing its value improves things (it's a pretty recent addition)

Eisaichen commented 8 months ago

Just tried, it will take 4-5 minutes to get the URL, and removing the -f flag didn't help. there no error occurred, just slow. I also noticed that Metube also takes much longer to start a download mission than usual. so, probably Youtube did something? Maybe we can add a new feather to automatically generate mp3 for the new feed and cache it, with some options to tweak that?

madiele commented 8 months ago

Unfortunately generating and caching the url ahead of time is not super clean as a solution, because the generated url has an expiry date, actually the generated url is already cached once generated for 5 hours, so if you try to query the same video again and it should not wait 4 minutes any more (if it does it's a bug).

A temporary workaround for you could be to write a side script to query the transcode url you need ahead of time to put them in the cache, probably chatgpt is able to code something like this pretty easily in python.

I'm open add a feature for caching ahead of time every x hours but before committing I would like to know if other people have the same issue as you, to make sure it's not something specific to your setup (mostly because it's harder for me to maintain feature that I personally don't use, so I want to be sure it's needed). I'll leave the issue open and see if someone check in.

In the meanwhile I suggest you to install yt-dlp on your personal pc or another server and check if it's still slow, maybe the ip you query youtube from is rate limited by youtube or something. Also I live in Italy so it's possible youtube is testing something in your region

Eisaichen commented 8 months ago

After spending a lot of hours on this today, I finally figured it out, It was ipv6. apparently, yt-dlp will try to contact YouTube with ipv6 first, but my environment didn't allow containers to use ipv6, so the yt-dlp wasted all the time waiting for the ipv6 connction time out. The problem was solved immediately after I permitted ipv6 traffic. So sorry for letting you look into this, the problem is on my end after all.

madiele commented 8 months ago

Glad it was solved! Enjoy your podcasts ✌️