Error (409) when parsing YouTube account or playlist

[pmd2016]

Brilliant work, this! Was really looking for a solution like this!

I just started the service on a Raspberry Pi, but I can't get it to work. Every time I submit a YouTube account or playlist I get this error message:

2023-05-07T18:19:01.758Z ERROR [app] could not translate rss
2023-05-07T18:19:01.759Z ERROR [app] could not parse rss, reason: unable to parse feed: no root element

I'm not sure where to look to solve this issue, do you have an idea where I could start?

[madiele]

can you pass me the url you are trying to input? if you don't want to share it in public send it to my email madiele92@gmail.com

also do you have a youtube api key set?

[pmd2016]

It doesn't really matter what url I'm trying, I tried with several accounts, for example: https://www.youtube.com/@DisneyNL, https://www.youtube.com/@Formula1.

Yes, I set up a YT API key. The API container is stopped immediately after setting it up. Is that supposed to happen?

[madiele]

The API container is stopped immediately after setting it up. Is that supposed to happen?

yes

I'll see if I can replicate it and report back

[pmd2016]

O, wait! It just started working!

[madiele]

I was able to replicate your issue by setting an invalid youtube apikey, so my guess it was some problem related to that.

I should definitely try to improve error messaging to check if the apikey works though!

feel free to close the issue, or I will close it myself in a few days if the problem does not shows up.

[pmd2016]

Yeah, I think it had something to do with the API key.

I'll close this issue!

[BubiBalboa]

I also get the 409 error when I try to use an API key. Works fine without it, limitations aside.

Just to make sure I'm doing it right: Are we supposed to paste the key after the = ? Or are we replacing everything that comes after the = with the API key?

[madiele]

This is the correct syntax (using random characters instead of the api key)

- YT_API_KEY=djsnidmsoeksnsjsodnedox9

[BubiBalboa]

Thanks for quick response!

Okay, I changed it and did the "sudo docker compose up -d" thing. When everything is green I should be good to go, right?

I tried it once and got a new error, which is fun.

"Error 400: relative URL without a base"

Tried it again and got 409 again. Any ideas what could be going on?

Edit: I tried generating a new API key just to double check since the other key was old and rarely used. Still 409 :(

[madiele]

I'm thinking your api key is not working right maybe it does not have the right permissions, try to get a new one.

Also do a "sudo docker compose pull" too, to download any updates if are there (I released a new version few days ago)

If nothing works, set the rust logs setting on the docker compose to "DEBUG" and post here the logs sudo docker compose logs (check if you need to remove any sensitive data before posting it)

[BubiBalboa]

Both the old and the new key are unrestricted. Are there other permissions I need to check?

I freshly installed Docker to try your app, so it should be the latest version.

I have Docker running in a container in Proxmox. I don't think this should be problem but I'm mentioning it just in case. It works fine without the API key.

I'm working on getting the logs.

[madiele]

Try this settings

[madiele]

Also give me an example of the inputs you are trying, I will test them in my own instance

[BubiBalboa]

I got the 409 error again. I really thought the change to the Youtube API would to the trick. :/ I waited at least ten minutes for the change to take effect. And I double checked that I'm using the correct key in the config file. I hope I can rule out easy errors.

I'm just posting random channels into the text box. "https://www.youtube.com/@TomScottGo" for example.

[madiele]

Can confirm the link works on my end with the api key...

can you post your docker-compose file with sensitive data obscured so that I can see if something is weird?

[BubiBalboa]

He are some logs, not sure if that is detailed enough. I don't think the memory thing is the problem.

root@pDocker:~/vod2pod-rss# docker compose logs -t -f --tail 5
vod2pod-rss-vod2pod-1  | 2023-09-10T19:14:08.883512192Z 2023-09-10T19:14:08.883Z ERROR [app] could not parse rss, reason: unable to parse feed: no root element
vod2pod-rss-vod2pod-1  | 2023-09-10T19:14:36.727569556Z 2023-09-10T19:14:36.715Z INFO  [vod2pod_rss::provider::youtube] trying to convert youtube channel url https://www.youtube.com/@patrickwillemspresents1220
vod2pod-rss-vod2pod-1  | 2023-09-10T19:14:36.727638329Z 2023-09-10T19:14:36.716Z INFO  [vod2pod_rss::provider::youtube] conversion not in cache, using yt-dlp for conversion...
vod2pod-rss-vod2pod-1  | 2023-09-10T19:14:41.077220917Z 2023-09-10T19:14:41.076Z ERROR [app] could not translate rss
vod2pod-rss-vod2pod-1  | 2023-09-10T19:14:41.077261706Z 2023-09-10T19:14:41.076Z ERROR [app] could not parse rss, reason: unable to parse feed: no root element
vod2pod-rss-podtube-1  | 2023-09-10T19:14:03.025979802Z 2023-09-10 19:14:03,025 [ERROR] 500 GET /youtube/channel/UCQHX6ViZmPsWiYSFAyS0a3Q (172.18.0.3) 296.91ms
vod2pod-rss-podtube-1  | 2023-09-10T19:14:08.881151722Z 2023-09-10 19:14:08,880 [ERROR] Error Downloading Channel: Forbidden
vod2pod-rss-podtube-1  | 2023-09-10T19:14:08.883933271Z 2023-09-10 19:14:08,883 [ERROR] 500 GET /youtube/channel/UCXuqSBlHAE6Xw-yeJA0Tunw (172.18.0.3) 325.30ms
vod2pod-rss-podtube-1  | 2023-09-10T19:14:41.074957200Z 2023-09-10 19:14:41,074 [ERROR] Error Downloading Channel: Forbidden
vod2pod-rss-podtube-1  | 2023-09-10T19:14:41.077984236Z 2023-09-10 19:14:41,077 [ERROR] 500 GET /youtube/channel/UCMAwBUlaxyqZY-2hVac8qgg (172.18.0.3) 781.02ms
twitchToPodcastRSS     | 2023-09-10T19:17:58.209309410Z     raise Exception("Twitch API client id env variable is not set.")
twitchToPodcastRSS     | 2023-09-10T19:17:58.209315473Z Exception: Twitch API client id env variable is not set.
twitchToPodcastRSS     | 2023-09-10T19:17:58.209732839Z [2023-09-10 20:17:58 +0100] [8] [info] Worker exiting (pid: 8)
twitchToPodcastRSS     | 2023-09-10T19:17:58.311808344Z [2023-09-10 20:17:58 +0100] [6] [INFO] Shutting down: Master
twitchToPodcastRSS     | 2023-09-10T19:17:58.312038225Z [2023-09-10 20:17:58 +0100] [6] [INFO] Reason: Worker failed to boot.
vod2pod-rss-redis-1    | 2023-09-10T13:31:10.054013299Z 1:C 10 Sep 2023 13:31:10.052 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
vod2pod-rss-redis-1    | 2023-09-10T13:31:10.054080772Z 1:C 10 Sep 2023 13:31:10.052 # Redis version=6.2.13, bits=64, commit=00000000, modified=0, pid=1, just started
vod2pod-rss-redis-1    | 2023-09-10T13:31:10.054094658Z 1:C 10 Sep 2023 13:31:10.052 # Configuration loaded
vod2pod-rss-redis-1    | 2023-09-10T13:31:10.062872486Z 1:M 10 Sep 2023 13:31:10.054 # Server initialized
vod2pod-rss-redis-1    | 2023-09-10T13:31:10.062907939Z 1:M 10 Sep 2023 13:31:10.054 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

[madiele]

Wait, where is the podtube service block?

If you check the docker-compose.yml in the repo you will see it, you have to add it back if you want the api key to work

[madiele]

I'm reopening the issue for the time being

[BubiBalboa]

Yeah, my bad on the yml file. I never scrolled down that far in the nano editor. I have now set the logs to debug as well. Didn't see that before. I only changed the YT API key though.

version: "3.9"

services:
  api_keys:
    image: alpine
    environment:
    #add your api-keys here (put them after the "=" sign or use an .env file)
      # Set your YouTube API key
      - YT_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      # Set your Twitch secret
      - TWITCH_SECRET=${TWITCH_SECRET:-}
      # Set your Twitch client ID
      - TWITCH_CLIENT_ID=${TWITCH_CLIENT_ID:-}

  vod2pod:
    extends: api_keys
    image: madiele/vod2pod-rss
    depends_on:
      - redis
    restart: unless-stopped
    ports:
      - "1234:8080" #change from 80 to another port if you already use the port 80 on your host
    environment:
      - TZ=Europe/London #set if you want the logs to have you timezone
      - MP3_BITRATE=192 #bitrate in kilobits of the mp3 transcode
      - TRANSCODE=true #put to false if you only need feed generation
      - SUBFOLDER=/ #for reverse proxies, ex: "/" -> access the app at mywebsite.com ; "vod2pod" -> access at mywebsite>
      - RUST_LOG=DEBUG #INFO #set to DEBUG if you are having problems than open a github issue with the logs, use "sudo>
      - TWITCH_TO_PODCAST_URL=ttprss #don't edit this
      - PODTUBE_URL=http://podtube:15000 #don't edit this
      - REDIS_ADDRESS=redis #don't edit this
      - REDIS_PORT=6379 #don't edit this

  ttprss: # you can remove this section and save some storage if you don't need twitch
    extends: api_keys
    image: madiele/twitch_to_podcast_rss
    restart: unless-stopped
    container_name: twitchToPodcastRSS
    environment:
        - TZ=Europe/London
        - TRANSCODE=False #keep this to false, the vod2pod one is the one you should change

  podtube: #remove this section if you don't need improved youtube support
    extends: api_keys
    image: ftawesome/podtube
    restart: unless-stopped
    environment:
      - YT_AUTOLOAD_NEWEST_AUDIO=False

  redis:
    image: "redis:6.2"
    command: redis-server --save 20 1 --loglevel warning
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      interval: 10s
      timeout: 3s
      retries: 5

[madiele]

The config seems correct, by the logs the 500 forbidden on podtube service indicate that is indeed that youtube api is not happy with your api key.

I'll take some wild shots at it

• Check that you did not accidentally add a space/tab to the end/begin of the key in the compose file
• Check if when you paste it some characters get lost (might happen if you use vim for example, nano is better when pasting critical stuff)
• redo a key from scratch
• are you using a vpn or hosting? Maybe youtube is blocking your traffic due to location, shared ip or something

Let me know if you did find a solution, feel free to ask any extra questions you have though ✌️

[BubiBalboa]

No VPN. I'm watching Youtube as I write this with no problems.

Yeah, I think I'll start from scratch and see if that does the trick. Those yaml files are so sensitive to the slightest syntax errors and I don't want to hunt for an extra space somewhere in this file. It's totally possible I messed something up when I first edited the file since I use nano inside a browser window using copy and paste and so on.

I hope that will work. Sorry for wasting your time and thanks for this cool app.

[madiele]

BTW you can also use an .env file, use the default docker-compose.yml from this repo and on the same folder of it do

echo 'YT_API_KEY=auzuwbbsixjsns' > .env 

It's an alternative way to set the api keys that I use mostly so that I can keep the docker-compose.yml file from leaking my api key when developing 😝 but can be used by anyone too (maybe I'll actually write it as the preferred way in the documentation later, it's probably less error prone)

[BubiBalboa]

Okay, I just redid everything. Went a lot faster doing it the second time. :)

Same error though, unfortunately. I was careful not to mess up the config file.

Just a thought but can you try pasting your API key in your config file instead of using the .env file? Maybe that's why it works for you?

[BubiBalboa]

I think there is something wrong with my API key after all.

On the Google API dashboard I have a 100% error rate for the method "youtube.api.v3.V3DataChannelService.List"

I'll keep reading the docs to see what could be the problem.

[madiele]

Can confirm that it works even if set in the config file.

I would suggest you check your Google api dashboard for the usage of the api key, maybe you made to many requests and you've been rate limited (in that case tomorrow the problem might disappear out of the blue)

I'm running out of other possibilities, but I leaning towards youtube api key problem more and more.

Though in the case you've been rate limited maybe podtube or vod2pod might have a bug somewhere and send to many request in some weird case, I'm still not ruling out a bug being there somewhere that causes youtube to temporarily say no to the key, that would explain how the original user got it to work out of the blue.

[BubiBalboa]

I have 72 requests (seems too high) for the method "youtube.api.v3.V3DataChannelService.List" with a 100% error rate. That's the only method that has been used. I don't think they rate limit you so soon. And shouldn't at least some requests gotten through in the start so the error rate should be lower than 100% ?

I'll try again tomorrow hopefully it fixes itself.

[madiele]

I think there is something wrong with my API key after all.

On the Google API dashboard I have a 100% error rate for the method "youtube.api.v3.V3DataChannelService.List"

I'll keep reading the docs to see what could be the problem.

Good! That means the api key is correctly setted at least! Otherwise you would not see the logs of your errors.

Rate limiting might be an explanation, bad permission another, many options, but you can rule out the bad config

[madiele]

I have 72 requests (seems too high) for the method "youtube.api.v3.V3DataChannelService.List" with a 100% error rate. That's the only method that has been used. I don't think they rate limit you so soon. And shouldn't at least some requests gotten through in the start so the error rate should be lower than 100% ?

I'll try again tomorrow hopefully it fixes itself.

It is low (expecially if not in the same minute) for it to be rate limited, but maybe Google has weird rules for it, we'll see tomorrow, if it still not gone maybe try calling that api yourself from postman, you'll probably get a more specific reason for the forbidden

You can find the settings for the call inside the podtube repo (link in the honorable mentions section of the readme)

[BubiBalboa]

I didn't fix itself unfortunately. What I meant with "too high" is that I'm pretty sure I didn't try to use the app more than maybe 30 times. I guess your program does a few retries when it encounters an error? I agree that this number of calls shouldn't trigger a rate limit.

I just tried using a VPN from a few different countries in case my IP has been blacklisted or something. Didn't work either.

I'll see if I can figure what Google's problem is.

[madiele]

My app does 1 api call (cached if no new episode are there, but since you were deploying a lot is possible the cache got invalidated) to get the first 50 videos durations, and podtobe makes anywhere from 1 to 3 api calls on its own, so the number is pretty reasonable, but it's pretty low in volume anyway, and I know of no similar issue (I know a good number of people are using the app judging by how much the image is pulled)

[madiele]

Very weird, you could try changing account maybe, I don't know 🤷‍♂️

[BubiBalboa]

I feel it has got to do something with authentication since I get 403 forbidden on the dashboard. But I don't think the called method needs any extra auth beyond the API key. It's weird.

[BubiBalboa]

It was the quota! I just tried an request from a Python program and it returned 403 with the message "The request cannot be completed because you have exceeded your quota."

I guess I'll just leave the app alone for a few days and hope that resets the quota. I'd never thought so few requests would trigger a rate limit.

[madiele]

I use the app daily for many feed with 30 minutes refresh and I never got hit. Weird, maybe they have some rules to dinamically adjust them? Don't know. Anyway good that you found the issue, since it does not seem a problem with the app I will close the issue again, thanks for reporting back!

[BubiBalboa]

Okay, I fixed it. In case anyone has the same problem:

The main issue was that I was using a 4 year old, unused Google developer project. I had created an API key at that time. I tried using that key and it didn't work. I thought maybe the key is too old somehow and created a new key, which didn't work either.

I had the right idea but my first solution was wrong. You need to create a whole new project or it will not work. Google sets the quotas for old, unused projects to zero. There are supposedly ways to reactivate an old project but creating a new one is faster and works as well.

So check if your daily requests quota has been set to zero if you are getting 403 "quota exceeded" errors.