search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.01k stars 10.01k forks source link

[Pornhub] Youtube-dl grabbing more videos than listed on channel (NSFW) #12819

Closed vxbinaca closed 7 years ago

vxbinaca commented 7 years ago

User has 7 videos uploaded but Youtube-dl repeatedly seen 11, also a different random video ripped at start of every run. The first 4 videos of a channel rip aren't from the target channel (see output below).

youtube-dl -v --ignore-config https://www.pornhub.com/users/mary_mariexxx/videos/public

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', 'https://www.pornhub.com/users/mary_mariexxx/videos/public']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.04.17
[debug] Python version 3.5.2 - Linux-4.4.0-72-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 2.8.11-0ubuntu0.16.04.1, avprobe 2.8.11-0ubuntu0.16.04.1, ffmpeg 2.8.11-0ubuntu0.16.04.1, ffprobe 2.8.11-0ubuntu0.16.04.1
[debug] Proxy map: {}
[PornHubUserVideos] mary_mariexxx: Downloading page 1
[PornHubUserVideos] mary_mariexxx: Downloading page 2
[download] Downloading playlist: mary_mariexxx
[PornHubUserVideos] playlist mary_mariexxx: Collected 11 video ids (downloading 11 of them)
[download] Downloading video 1 of 11
[PornHub] ph58caae5842f45: Downloading webpage
[PornHub] ph58caae5842f45: Downloading webpage
[debug] Invoking downloader on 'https://bv.phncdn.com/videos/201703/16/109837182/720P_1500K_109837182.mp4?ipa=204.232.125.50&rs=200&ri=1200&s=1492879829&e=1492887029&h=1b17c01cddf7a7374f17f9fee2d97663'
[download] Resuming download at byte 2349508
[download] Destination: Brazzers - Dirty teen Nicole Bexley loves cock-ph58caae5842f45.mp4
[download] 100% of 86.97MiB in 09:36
[download] Downloading video 2 of 11
[PornHub] ph589b759af1464: Downloading webpage
[PornHub] ph589b759af1464: Downloading webpage
[debug] Invoking downloader on 'https://dv.phncdn.com/videos/201702/08/105407102/720P_1500K_105407102.mp4?ttl=1492887609&ri=1228800&rs=1600&hash=4ae872fb816b0105c038a62927237ff8'
[download] Destination: Brazzers - Jillian Janson needs anal-ph589b759af1464.mp4
[download] 100% of 82.11MiB in 07:05
[download] Downloading video 3 of 11
[PornHub] 1488643063: Downloading webpage
[PornHub] 1488643063: Downloading webpage
[debug] Invoking downloader on 'https://bv.phncdn.com/videos/201103/07/3043498/vl_480P_303.0k_3043498.mp4?ipa=204.232.125.50&rs=48&ri=1000&s=1492880837&e=1492888037&h=07e85c321f1f933725a4ca6fe14da4f3'
[download] Destination: young head-1488643063.mp4
[download] 100% of 3.92MiB in 01:00
[download] Downloading video 4 of 11
[PornHub] 1099822914: Downloading webpage
[PornHub] 1099822914: Downloading webpage
[debug] Invoking downloader on 'https://dv.phncdn.com/videos/201309/19/17554051/vl_480P_595.0k_17554051.mp4?ttl=1492888100&ri=1433600&rs=744&hash=c5340034dbe1461b95a4696897f14616'
[download] Destination: Home Made Blow Jobs 4 - Scene 4-1099822914.mp4
[download] 100% of 56.82MiB in 10:26
[download] Downloading video 5 of 11
[PornHub] ph58ea2d5f11575: Downloading webpage
[PornHub] ph58ea2d5f11575: Downloading webpage
[debug] Invoking downloader on 'https://bv.phncdn.com/videos/201704/09/112601451/720P_1500K_112601451.mp4?ipa=204.232.125.50&rs=200&ri=2700&s=1492881529&e=1492888729&h=0210c37f5aec14986877da4135a7d1cc'
[download] Destination: Redhead tied up and fucked like a slut-ph58ea2d5f11575.mp4
[download] 100% of 449.40MiB in 38:07
[download] Downloading video 6 of 11
[PornHub] ph58e7be44d0eb2: Downloading webpage
[PornHub] ph58e7be44d0eb2: Downloading webpage
[debug] Invoking downloader on 'https://cv.phncdn.com/videos/201704/07/112409341/720P_1500K_112409341.mp4?a5dcae8e1adc0bdaed975f0d67fb5e0527c20903c5bb57a6cad7e6cb50bc41fbb1152c24e90ee001fb80e2adfff13fd1a9d373ee21fa85533e06d6fa1d5a3ac67c3bdc051991814a43ccb9085902a56cf76826916151ab9bf3c1ce1cf3353601f0569df4e8c69d6093ef443bf7d5c31d29ea1c1d106eb4354445e221'
[download] Destination: Redhead teen getting huge cumshot-ph58e7be44d0eb2.mp4
[download] 100% of 7.23MiB in 00:29
[download] Downloading video 7 of 11
[PornHub] ph58dd6fe73f282: Downloading webpage
[PornHub] ph58dd6fe73f282: Downloading webpage
[debug] Invoking downloader on 'https://dv.phncdn.com/videos/201703/30/111567862/720P_1500K_111567862.mp4?ttl=1492891052&ri=1024000&rs=1600&hash=d197d38ce28801b4c71866c5f5cd1a5e'
[download] Destination: Redhead teen deepthroating and gagging on cock-ph58dd6fe73f282.mp4
[download] 100% of 31.09MiB in 02:38
[download] Downloading video 8 of 11
[PornHub] ph58d550a055cc1: Downloading webpage
[PornHub] ph58d550a055cc1: Downloading webpage
[debug] Invoking downloader on 'https://dv.phncdn.com/videos/201703/24/110799752/720P_1500K_110799752.mp4?ttl=1492891213&ri=1433600&rs=1600&hash=1c774ae361c7619e4eaa17be84ade58e'
[download] Destination: Redhead Teen Deepthroat-ph58d550a055cc1.mp4
[download] 100% of 116.01MiB in 10:10
[download] Downloading video 9 of 11
[PornHub] ph58d31e5ddb5f8: Downloading webpage
[PornHub] ph58d31e5ddb5f8: Downloading webpage
[debug] Invoking downloader on 'https://cv.phncdn.com/videos/201703/23/110611552/720P_1500K_110611552.mp4?a5dcae8e1adc0bdaed975f0d67fb5e0527c20903c5bb57a6cad7e6cb50bc41fbb1152c24e90ee001fc80e0a9fff13fd3a672d47e8a967bd36ef2c622317ce5fcf4dfd48819930ec53a3dc5c06d721505da9ab502c07273fe3ae57600e23ff6f8ca39e4f28f48189582425c4a1e1f4bfe7e78f7921f31b8fd88202d3d'
[download] Destination: Red Head Teen BDSM-ph58d31e5ddb5f8.mp4
[download] 100% of 54.35MiB in 04:31
[download] Downloading video 10 of 11
[PornHub] ph58d03bdede49d: Downloading webpage
[PornHub] ph58d03bdede49d: Downloading webpage
[debug] Invoking downloader on 'https://dv.phncdn.com/videos/201703/20/110342782/720P_1500K_110342782.mp4?ttl=1492892101&ri=1228800&rs=1600&hash=8643cc9561cfc412d9b923d58c2b97fb'
[download] Destination: I love getting sloppy when giving head-ph58d03bdede49d.mp4
[download] 100% of 65.46MiB in 05:38
[download] Downloading video 11 of 11
[PornHub] ph58cd4cec70d16: Downloading webpage
[PornHub] ph58cd4cec70d16: Downloading webpage
[debug] Invoking downloader on 'https://bv.phncdn.com/videos/201703/18/110064742/720P_1500K_110064742.mp4?ipa=204.232.125.50&rs=200&ri=1200&s=1492885247&e=1492892447&h=3f529e33a97a0b6c0b95083814904f0e'
[download] Destination: My boyfriend tied me up and fucked me like a good little slut-ph58cd4cec70d16.mp4
[download] 100% of 55.09MiB in 04:36
[download] Finished downloading playlist: mary_mariexxx
PuffingtonToast commented 7 years ago

Problem PornHubUserVideos downloads 4 extra videos not listed in the user's channel at the beginning of the download process.

Cause PH sends related videos as part of the user page, but these related videos are never rendered. The extractor finds all strings matching a video page regex in the given webpage.

Solution Only parse videos in the "videoUList" div by omitting the beginning of the webpage from the extraction process.

vxbinaca commented 7 years ago

Thank you!