Matched format but not downloading

[Extarys]

Video: https://www.youtube.com/watch?v=RMxB7zA-e4Y

I see in a red box: Media cannot be downloaded because it has no formats which match the source requirements.

In available formats:

 ID: 249 , audio:opus @50.656k / 48000Hz
ID: 250 , audio:opus @59.357k / 48000Hz
ID: 251 , audio:opus @106.796k / 48000Hz (matched) <<<===
ID: 140 , audio:mp4a.40.2 @129.477k / 44100Hz
ID: 160 , 144p (256x144), fps:30, video:avc1.4d400c @37.686k
ID: 278 , 144p (256x144), fps:30, video:vp9 @73.996k
ID: 133 , 240p (426x240), fps:30, video:avc1.4d4015 @72.452k
ID: 242 , 240p (426x240), fps:30, video:vp9 @109.107k
ID: 134 , 360p (640x360), fps:30, video:avc1.4d401e @134.389k
ID: 243 , 360p (640x360), fps:30, video:vp9 @229.64k
ID: 135 , 480p (854x480), fps:30, video:avc1.4d401f @214.676k
ID: 244 , 480p (854x480), fps:30, video:vp9 @406.179k
ID: 136 , 720p (1280x720), fps:30, video:avc1.4d401f @339.962k
ID: 247 , 720p (1280x720), fps:30, video:vp9 @787.016k (matched) <<<===
ID: 137 , 1080p (1920x1080), fps:30, video:avc1.640028 @1304.274k
ID: 248 , 1080p (1920x1080), fps:30, video:vp9 @1558.936k
ID: 271 , 1440p (2560x1440), fps:30, video:vp9 @4914.483k
ID: 313 , 2160p (3840x2160), fps:30, video:vp9 @15184.846k
ID: 18 , 360p (640x360), fps:30, video:avc1.42001E @441.672k , audio:mp4a.40.2 @k / 44100Hz
ID: 22 , 720p (1280x720), fps:30, video:avc1.64001F @468.948k , audio:mp4a.40.2 @k / 44100Hz 

Combined: no match
Audio: 251 (exact match)
Video: 247 (exact match) 

I'm not sure I understand why it doesn't download. In the logs I see (not for the same video though as the line disappeared quite rapidly):

2021-04-09 01:04:52,896 [tubesync/INFO] Media: Technology Connections / TbHBHhZOglw has a valid publishing date, marking to be unskipped

2021-04-09 01:04:52,993 [tubesync/INFO] Scheduling task to download thumbnail for: The touch lamp; a neat idea, and older than you'd think! from: https://i.ytimg.com/vi_webp/TbHBHhZOglw/maxresdefault.webp

2021-04-09 01:04:53,203 [tubesync/INFO] Saved 276865 bytes of metadata for: Technology Connections / 7993c665-445f-4c0e-b053-63eb1252ae51

2021-04-09 01:04:57,264 [tubesync/DEBUG] [youtube] Bi_Tp1H9CDs: Downloading webpage

2021-04-09 01:04:58,058 [tubesync/INFO] Media: Technology Connections / Bi_Tp1H9CDs has a valid publishing date, marking to be unskipped

2021-04-09 01:04:58,150 [tubesync/INFO] Scheduling task to download thumbnail for: Retroreflectors; they're everywhere, and they cheat physics (sort of) from: https://i.ytimg.com/vi_webp/Bi_Tp1H9CDs/maxresdefault.webp

2021-04-09 01:04:58,378 [tubesync/INFO] Saved 292154 bytes of metadata for: Technology Connections / e1cb8e47-c433-4ba9-9f0c-e5ae096462b9

I want to say a huge thank you for this! I don't download a lot but I like to rewatch those type of videos from time to time, and nice to watch it on kodi/jellyfin :)

[meeb]

I think the video actually will download, I noticed this myself earlier but hadn't got around to making an issue for it yet. I'm pretty sure this is just a template logic bug showing that error message when it's actually not meant to be shown rather than an actual error. I'll dig into it though. Thanks for the detail but report!

[Extarys]

Ok thanks! I cannot confirm yet if it does work as it's still gathering data and didn't start any download yet. I'm also waiting for this channel to complete before adding anything else.

Glad you liked the detailed issue :smile: :hugs:

Late edit: downloads are working even if the error is there so :+1:

[Casuallynoted]

For me, the error has shown but nothing downloads. I've only had two videos actually pull, while the rest sit there and nothing happens. The server has been sitting for about two weeks with no downloads. Settings for the channel are just 1080p vp9.

[meeb]

Is there anything in your container logs @Casuallynoted ? Any actual Python errors would be in there and it would be pretty difficult to diagnose your issue without more information.

[Jimmni]

I'm having this issue too. Every video on every channel I add gets marked as "No matching formats" and " Media cannot be downloaded because it has no formats which match the source requirements." whatever settings I choose.

It does seem to be finding valid formats though. One for example says:

Combined: no match Audio: 251 (exact match) Video: 247 (exact match)

After a while this gets added:

"Media is marked to be skipped and will not be downloaded."

[meeb]

Thanks for the confirmation @Jimmni - in the interim if the (exact match) is visible and the media is downloaded you can ignore the template incorrectly saying it'll be skipped for now.

[meeb]

@Extarys I've just pushed a tweak which should hopefully fix this. You will need to update to :latest and then edit and save the source to trigger a re-evaluation of some fields to see it. Let me know if this resolves it for you.

@Jimmni the "skipped" flag is a different issue, the skip flag can only either be set manually with the "skip" button or if you have set a maximum time window to download media for a source and it's now exceeded that time window (e.g. you've added a source, it has a new media item, the media item gets downloaded, you've set the source to not download media older than one week, after one week the media item will be marked to be skipped and you'll see that message).

I can clarify the message to better explain the behaviour, but if the above situation matches what you've set up in the source then it's not a bug really that's working as expected, just not that obviously.

[Jimmni]

I updated to the latest today but all media is still being marked as "Media cannot be downloaded because it has no formats which match the source requirements."

It did try to download some on the last version but I'd made a mistake with my paths and they appear to have gone into a void. I'm waiting on it to try again today with the latest version of tubesync and (hopefully) fixed paths. Hopefully it'll still download and just be saying the error like you mentioned above.

Just thought I'd mention that whatever you pushed a few days ago didn't fix the error message for me.

Edit: Files did indeed download now, it's just the error message.

[meeb]

"Media cannot be downloaded because it has no formats which match the source requirements." is a different issue to the "Media is marked to be skipped and will not be downloaded.". Which one are you still getting @Jimmni ?

[Jimmni]

Previously I was getting both, but after reinstalling with different paths I'm now only getting "Media cannot be downloaded because it has no formats which match the source requirements." As you said, media is actually downloading, but that message is still shown on every item.

[meeb]

Can you paste a YouTube URL where you have this occurring please? I'll check its metadata.

[Jimmni]

It's happened for every single video on each channel I've added. I definitely can't rule out something I've done wrong.

Here's a few:

https://www.youtube.com/watch?v=7rIJNT7dCmE https://www.youtube.com/watch?v=qDS-xw_ECm8 https://www.youtube.com/watch?v=4p0fRlCHYyg

My settings are:

Desired format | 1080p (video:VP9, audio:OPUS) Fallback | Get next best resolution or codec instead

TubeSync version 0.9.1 (installed with :latest) with youtube-dl version 2021.05.16 and FFmpeg version 4.3.2-static.

Let me know if there's anything else you need!

As I said this seems to be cosmetic, all three did download (though an hour after they said they would despite my timezone being correct). TubeSync is doing its job like a champ - thanks!

[meeb]

Are all of the videos that you get the notice on "exact match" for the video stream by any chance? It is cosmetic, but obviously still a bug I need to fix at some point too :)

[Jimmni]

As far as I can tell every single one is an exact match for both audio and video.

[meeb]

Thanks, helpful. I'll look into it and might open a new issue later.

[MatthK]

I'm having this issue as well.

Downloading media for "Blippi Explores Construction Vehicles Part 1 | Machines For Kids | Educational Videos For Kids" Task will run at 2021-08-24 22:07:09 still waiting for a download today.

ID: 249 , audio:opus @52.928k / 48000Hz
ID: 250 , audio:opus @68.601k / 48000Hz
ID: 140 , audio:mp4a.40.2 @129.478k / 44100Hz
ID: 251 , audio:opus @133.406k / 48000Hz (matched)
ID: 394 , 144p (256x144), fps:25, video:av01.0.00M.08 @69.451k
ID: 160 , 144p (256x144), fps:25, video:avc1.4d400c @81.992k
ID: 278 , 144p (256x144), fps:25, video:vp9 @91.625k
ID: 395 , 240p (426x240), fps:25, video:av01.0.00M.08 @145.851k
ID: 133 , 240p (426x240), fps:25, video:avc1.4d4015 @152.865k
ID: 242 , 240p (426x240), fps:25, video:vp9 @193.914k
ID: 134 , 360p (640x360), fps:25, video:avc1.4d401e @290.648k
ID: 396 , 360p (640x360), fps:25, video:av01.0.01M.08 @304.734k
ID: 243 , 360p (640x360), fps:25, video:vp9 @352.847k
ID: 135 , 480p (854x480), fps:25, video:avc1.4d401e @441.496k
ID: 397 , 480p (854x480), fps:25, video:av01.0.04M.08 @555.157k
ID: 244 , 480p (854x480), fps:25, video:vp9 @592.388k
ID: 136 , 720p (1280x720), fps:25, video:avc1.4d401f @825.189k
ID: 398 , 720p (1280x720), fps:25, video:av01.0.05M.08 @1027.848k
ID: 247 , 720p (1280x720), fps:25, video:vp9 @1062.605k (matched)
ID: 399 , 1080p (1920x1080), fps:25, video:av01.0.08M.08 @1783.429k
ID: 248 , 1080p (1920x1080), fps:25, video:vp9 @2339.784k
ID: 137 , 1080p (1920x1080), fps:25, video:avc1.640028 @3037.349k
ID: 18 , 360p (640x360), fps:25, video:avc1.42001E @697.231k , audio:mp4a.40.2 @k / 44100Hz
ID: 22 , 720p (1280x720), fps:25, video:avc1.64001F @954.293k , audio:mp4a.40.2 @k / 44100Hz 

[meeb]

Anything in the container logs for this download @MatthK ?

[MatthK]

Hmm, the only thing I could find is the following line, which matches the Media at the top of the page:

chown: changing ownership of '/downloads/video/Blippi/20210814_blippi-educational-videos-for-kids_blippi-explores-construction-vehicles-part-1-machines-for-kids-educational-video_JuRl6d8JzdE_720p-vp9-opus.mkv.f247.part': Operation not permitted

I do have the media on a mounted drive on my NAS.

[meeb]

That would strongly imply it's a permissions issue. TubeSync itself doesn't contain any code that invokes chown so I would suspect that would be part of youtube-dl. If the download "fails" it's likely going to abort the process and your download will just eventually be marked as failed to download.

[MatthK]

It used to work perfectly fine in this configuration. And I am not aware that anything has changed to the configuration. And it just downloaded some other files into the same directory.

[meeb]

Edited reply again. Can you drop into TubeSync container as the application user:

$ docker exec -ti --user app tubesync bash

and run this as a test:

$ youtube-dl --verbose --merge-output-format=mkv --format 247+251 --output /downloads/test.mkv https://www.youtube.com/watch\?v\=JuRl6d8JzdE

That's pretty much exactly what TubeSync does over the API and will download the file to your same network share.

[SmartPhoneLover]

Any progress about that special behaviour? I'm using latest (0.9.1) from (unRAID), I'm experiencing the same thing.

As an example, this is one of the channels which from I got some of the errors: https://www.youtube.com/channel/UC_0NfufarVw04vDfWFm8z_Q

In that case, is different from others that I have (both audio and video matches, but fails to download it locally actually): Combined: no match Audio: ### (exact match) Video: ### (no match)

So, in that case, the resolution of the video is smaller than the configured one. Example: I have configured the channel to download videos at 480p (minimum). And, selecting 'Get next best resolution or codec instead' from its options, to allow downloading those videos with smaller resolutions (ie: 360p).

[meeb]

Are you using :0.9.1 or :latest? If you're not using :latest try it please. :0.9.1 is pretty old now. The documentation has been updated to advise using :latest a while ago.

[SmartPhoneLover]

Hello, oh. Fast reply (actively working on the project 😉🤝)

I'm using the 'latest' tag, not a specific one. But it shows as '0.9.1' from the bottom of the TubeSync webpage (GUI): TubeSync version 0.9.1 with youtube-dl version 2021.06.06 and FFmpeg version 4.3.2-static

Note: One more thing I would like to request, if possible, for future releases. I tried to know the version by another way (CMD: docker image inspect), but it did not show any version label from the image. Could it be added? Because, when we (unRAID users) configure the tag as 'latest', using 'docker image inspect' is the easiest way to check for the real image version that's behind the 'latest' tag. cmd: _docker image inspect --format '{{json .}}' "imageid" | jq -r '. | {Id: .Id, Digest: .Digest, RepoDigests: .RepoDigests, Labels: .Config.Labels}'

[meeb]

Yeah sure, I'll add a version label in the next release.

As for your download error, what's in the container logs for that failing download? There will be a more specific error.

[SmartPhoneLover]

Sorry for the late reply, but I was testing some things, also migrating all my setup to MariaDB (as discussed in another thread days ago). Here are the logs of one of the channels I have added (others have similar issues, or didn't download anything).

https://pastebin.com/QV0DQ1Fs

As shown in the log, the video is downloaded, but the conversion seems to fail (ffmpeg). So, it gets stuck.

https://imgur.com/a/V9EQJh4

This is the channel (playlist actually) that was failing, from the log.

https://www.youtube.com/playlist?list=PLDawMIcxuUkpdZX-ernQQfI2SLMIbc4PM

Note: The log collects a portion of the errors refered to a specific channel, and after running the 'reset-tasks' command (CLI).

---

Another thing I would like to discuss and ask for...

I already tried, with some of my other channels, to download the videos that failed to download, manually (JDownloader, using same video/audio codec, resolution, extension and filename). Then, moving them to where the TubeSync stores the corresponding videos from these channels.

Well, some of these channels, are successfully indexed and now there are no videos waiting to be downloaded. Fine! But, others like the one from the log (above) keeps failing, even if TubeSync seems to fallback to the right video/audio ID.

[meeb]

I've had a look and it seems your failed downloads are due to YouTube returning 404s for that particular videos DASH parts. For example:

[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 2 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 3 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 4 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 5 of 10) ...
... etc.

It seems the metadata returned by yt-dlp says the format is valid, then can't download it when specified as a format. This is either a bug in yt-dlp or with the actual format on YouTube is broken. I imagine other clients just fall back to a different format more gracefully. Either way, I don't think this is a TubeSync issue although I might be able to more gracefully handle the error.

As for downloading the videos manually, that works fine but you'll need to import them into TubeSync. This just lets TubeSync know the video file is already downloaded so it doesn't try and download it again.

https://github.com/meeb/tubesync/blob/main/docs/import-existing-media.md

[SmartPhoneLover]

It would be great to see if TubeSync can handle that particular scenario as best as possible. Thanks!

Talking about importing media inside TubeSync manually, the documentation says that the filename of the video must contain the unique ID for each one. But, I have managed (without knowing it actually, as I didn't read it before) to import all the missing videos from other channels just naming them with what the titles of the videos contained without including IDs (some still failing to detect them), and after running 'reset-tasks', TubeSync managed to show all the videos that were missing due to: not matching codec/audio. If TubeSync requires the ID to be included on the filename, why some of the channels were successfully recognized (all videos)? Just to understand what could happent here, as the rule was not followed in that case, but the result was the same.

I have a question regarding to renaming the videos to allow TubeSync register missing ones. I use TubeSync to download channels and playlists that will be available to be watched directly from Jellyfin. So, in Jellyfin, I would have to rename all the videos that were manually added (ID+title) to not show the ID, as I search YT content by name. Taking the above into consideration, can I rename filenames from source again deleting IDs and only keeping the title, preventing TubeSync from marking them as not downloaded again when it performs scans for new videos automatically? Or I must keep the ID on its filenames after running 'import-existing-media'?

Also, what is the difference between executing the cmds 'reset-tasks' and 'import-existing-media' in that particular case (importing videos manually), if as said before, some of the channels were successfully re-indexed and all its videos were shown as downloaded.

[meeb]

The import-existing-media uses the media unique key (for YouTube videos this is the unique video ID) as the detection mechanism for matching a video file to import, as specified here:

https://github.com/meeb/tubesync/blob/main/tubesync/sync/management/commands/import-existing-media.py#L44

There is no other way to match a video file on import other than by video ID so there is probably another explanation for what you've encountered with your own file importing attempts. If you had the video ID as a directory name it would be matched as well I've just noticed so I probably need to patch that.

If you rename the file after importing it the file will still be marked as "downloaded", however the file association within TubeSync will be broken. Internally TubeSync stores an absolute path to the downloaded video, which if you rename it won't exist any more. You might not care about this of course.

reset-tasks is a completely different command, what that does is simply delete all tasks entirely and then call .save() on every source, there are signals that reschedule any required missing tasks on save so this just basically completely resets tasks. It doesn't affect media on disk at all.

[SmartPhoneLover]

Ok, thank you for the explanation.

About broken links, yes. I don't care about, as I won't play them from TubeSync, but from Jellyfin.

[meeb]

This should be fixed in :latest. Please open an new issue if you still have problems.