search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
10.66k stars 880 forks source link

[Newgrounds] Gallery-dl only checks for MP4s, but original videos are often in other formats #5709

Closed agitbird closed 6 days ago

agitbird commented 2 weeks ago

Hello.

Please see yt-dlp/yt-dlp#4715 for a good breakdown. Gallery-dl also partially suffers from this issue.

Firstly, Newgrounds seems to store original copies of their videos, from which they generate sampled copies at different standard resolutions as MP4s. The original copy does not need to be MP4. I expect the original copy is always the highest quality version for any video from Newgrounds. To access the original copy, remove the resolution from the Ungrounded URL (e.g., .1080p) and try different video extensions until one works.

Gallery-dl only seems to generate direct Newgrounds video URLs as MP4s (like the video samples), so if the original video is MP4 then that is successfully downloaded. However, if the original video has a different file extension, then attempts to reach that file will fail, and gallery-dl falls back to the biggest sample.

This is porn.

https://www.newgrounds.com/portal/view/933758

gallery-dl -G https://www.newgrounds.com/portal/view/933758
https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.mp4?1717829489
| https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.1080p.mp4?1717829488
| https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.720p.mp4?1717829489
| https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.360p.mp4?1717829489

gallery-dl https://www.newgrounds.com/portal/view/933758
[downloader.http][warning] '404 Not Found' for 'https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.mp4?1717829489'
[download][info] Trying fallback URL #1
* D:\Downloads\gallery-dl\newgrounds\skeptail\newgrounds_933758_Rivet.mp4

But this artist uses .MOV

https://uploads.ungrounded.net/alternate/5768000/5768625_alternate_266643.mov

Compare against this video mentioned in the yt-dlp issue thread where the original is MP4. This is not porn.

gallery-dl -G https://www.newgrounds.com/portal/view/853785
https://uploads.ungrounded.net/alternate/1927000/1927257_alternate_191145.mp4?1660529517
| https://uploads.ungrounded.net/alternate/1927000/1927257_alternate_191145.1080p.mp4?1660529516
| https://uploads.ungrounded.net/alternate/1927000/1927257_alternate_191145.720p.mp4?1660529516
| https://uploads.ungrounded.net/alternate/1927000/1927257_alternate_191145.360p.mp4?1660529517

gallery-dl https://www.newgrounds.com/portal/view/853785
* D:\Downloads\gallery-dl\newgrounds\kevuhn\newgrounds_853785_Cosmo Dep (2022).mp4

Please have gallery-dl download original Newgrounds videos regardless of their file extension. Thank you.

mikf commented 1 week ago

I made it so it now checks mp4, webm, m4v, mov, and mkv extensions for original files by default (https://github.com/mikf/gallery-dl/commit/c6fc0281e8292140a66f148140a4618686dc9d1b).

You can now also specify your own custom list of extensions that should be checked: newgrounds.format