search

meeb / tubesync

Syncs YouTube channels and playlists to a locally hosted media server
GNU Affero General Public License v3.0
1.83k stars 119 forks source link

No metadata was returned by youtube-dl #393

Open KandyMan1990 opened 1 year ago

KandyMan1990 commented 1 year ago

I've just attempted to set up TubeSync on my Synology NAS using docker and it is failing to download anything. I've looked into the logs and came across this:

2023-07-18 14:33:32,863 [tubesync/DEBUG] [youtube:tab] Extracting URL: https://www.youtube.com/channel/UCDrqiuwNRbEahL1UEB0hkKQ/videos 2023-07-18 14:33:32,863 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading webpage 2023-07-18 14:33:52,882 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (1/3)... 2023-07-18 14:33:52,882 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading webpage 2023-07-18 14:34:12,898 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (2/3)... 2023-07-18 14:34:12,898 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading webpage 2023-07-18 14:34:32,916 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (3/3)... 2023-07-18 14:34:32,916 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading webpage 2023-07-18 14:34:52,933 [tubesync/WARNING] [youtube:tab] Unable to download webpage: <urlopen error [Errno -3] Temporary failure in name resolution> (caused by URLError(gaierror(-3, 'Temporary failure in name resolution'))). Giving up after 3 retries 2023-07-18 14:34:52,934 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading API parameters API JSON 2023-07-18 14:35:12,947 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (1/3)... 2023-07-18 14:35:12,948 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading API parameters API JSON 2023-07-18 14:35:32,969 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (2/3)... 2023-07-18 14:35:32,970 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading API parameters API JSON 2023-07-18 14:35:52,988 [tubesync/WARNING] [youtube:tab] <urlopen error [Errno -3] Temporary failure in name resolution>. Retrying (3/3)... 2023-07-18 14:35:52,989 [tubesync/DEBUG] [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ/videos: Downloading API parameters API JSON 2023-07-18 14:36:13,014 [tubesync/ERROR] ERROR: [youtube:tab] UCDrqiuwNRbEahL1UEB0hkKQ: Unable to download API page: <urlopen error [Errno -3] Temporary failure in name resolution> (caused by URLError(gaierror(-3, 'Temporary failure in name resolution'))) Rescheduling Index media from source "beyblade" Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner func(*args, **kwargs) File "/app/sync/tasks.py", line 159, in index_source_task videos = source.index_media() ^^^^^^^^^^^^^^^^^^^^ File "/app/sync/models.py", line 517, in index_media response = indexer(self.index_url) ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/sync/youtube.py", line 60, in get_media_info raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was ' sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/channel/UCDrqiuwNRbEahL1UEB0hkKQ/videos": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.

I'm not sure if this is an error in the latest build or on error on my end. If anyone could help me I'd really appreciate it, I have a feeling it's probably a set up issue. I've tried restarting docker and the issue persists.

meeb commented 1 year ago

The actual error here buried in your traceback is "Temporary failure in name resolution". That would imply that tubesync can't resolve the host "www.youtube.com". Something would appear to be broken with your DNS resolver in your containers. The channel URL looks fine to me otherwise.

KandyMan1990 commented 1 year ago

I've figured it out, just needed to allow the docker network through the Synology firewall, thanks.

Unrelated but I'm seeing the container is using over 5.5gb of ram for downloading 522 videos. Is that normal? Before it had even started downloading them, just indexing and thumbnails took it up to that much usage, I've seen it as high as 5.75gb. I had to shut down a bunch of other containers to free up some ram as the Synology only has 8gb total. Even though a bunch of videos are downloaded it's still staying around 5.6gb, should it not be coming back down as the downloads finish?

meeb commented 1 year ago

That does seem like way too much RAM allocated. Some other people have reported some memory issues previously but I've never been able to replicate it to find the cause. Can you see what in the container is using RAM?

KandyMan1990 commented 1 year ago

I restarted the container after that large download was complete as the system was maxed out. I currently have 6 playlists being watched, all of which finished downloading last night, and currently it is sat at 1.87gb ram usage

image

KandyMan1990 commented 1 year ago

If you want to try and replicate it, you can see in my original post the channel I was attempting to download, maybe that's a good test case?

meeb commented 1 year ago

The issue is the current (jank) worker system. On my system it releases RAM after downloads regardless of what channels I add. It seems in some deployments it doesn't and the RAM usage continues to increase over time. Generally as I was going to replace the workers at some point I didn't spend that much time looking into. Your process list looks OK, even if the gunicorn workers do seem to be allocating way more RAM than I would expect.

mavrik132 commented 9 months ago

I am having a similar problem.

When I add a channel ID for some channels, I get the same "No metadata was returned"... error. The traceback shows "[tubesync/ERROR] ERROR: [youtube:tab] @bigclivedotcom: Unable to download API page: HTTP Error 404: Not Found (caused by <HTTPError 404: Not Found>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U"

The interesting part is I get a 404 if I access the URL displayed by TubeSync manually in chrome, but if I remove the "c/" from the URL, chrome will load the page.

alexander-henkes commented 8 months ago

I am having a similar problem.

When I add a channel ID for some channels, I get the same "No metadata was returned"... error. The traceback shows "[tubesync/ERROR] ERROR: [youtube:tab] @bigclivedotcom: Unable to download API page: HTTP Error 404: Not Found (caused by <HTTPError 404: Not Found>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U"

The interesting part is I get a 404 if I access the URL displayed by TubeSync manually in chrome, but if I remove the "c/" from the URL, chrome will load the page.

On which device do you use Tubesync? If you have used docker compose, a new network will be created, which may need to be enabled in the firewall. Here is the example on a Synology NAS. image image

Alternatively, you can also integrate Tubesync into the bridge network. Then usually no firewall approval is necessary. Add "network_mode: bridge" to your compose.yaml. image

meeb commented 8 months ago

@alexander-henkes thanks for the input, however a 404 error isn't likely to be a network issue. The 404 error is returned by YouTube so @mavrik132 's container can connect out and has networking.

@mavrik132 likely you are experiencing an issue with some channels not being channels, discussed here: https://github.com/meeb/tubesync/issues/389 https://github.com/meeb/tubesync/issues/299 etc.