live.fc2.com support - Githubissues

johnthecracker commented 5 years ago

[x] I'm reporting a site feature request
[x] I've verified that I'm running youtube-dl version 2019.05.11
[x] I've searched the bugtracker for similar site feature requests including closed ones

Description

live.fc2.com support

Is it possible for youtube-dl to support https://live.fc2.com/adult/ live streams ?

example url: https://live.fc2.com/67780606/

C:\>youtube-dl.exe --verbose --list-formats --no-check-certificate "https://live.fc2.com/67780606/"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--list-formats', '--no-check-certificate', 'https://live.fc2.com/67780606/']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2019.05.11
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 67780606: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 67780606: Downloading webpage
[generic] 67780606: Extracting information
ERROR: Unsupported URL: https://live.fc2.com/67780606/
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp4jj6sgve\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp4jj6sgve\build\youtube_dl\extractor\common.py", line 529, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp4jj6sgve\build\youtube_dl\extractor\generic.py", line 3329, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://live.fc2.com/67780606/

youtube-dl supports live streams from other sites like https://www.trtworld.com/live. https://www.trtworld.com/live saved file is corrupt. I pipe output to a player and can watch the stream:

youtube-dl.exe -f hls-827-0 https://www.trtworld.com/live -o - | ffplay.exe -

m4gnet98 commented 5 years ago

I have an "interest" in this as well. This is not the same as the "video.fc2.com" live.fc2.com and live.fc2.co/adult is the "same"

Link to stream webpage:

https://live.fc2.com/45390712/

Link to playlist:

https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789

The playlist is continuously updating with new .ts files, but youtube-dl doesn't read the updates when they come, you have to start the download again to get the "new" playlist.

Links to some of the .ts files:

https://hls3020-g.live.fc2.com/hls/45390712/32/data/18841.ts?time=1560607563&hash=9ca5a83df2151eee50bcffac2a77650a20659b397aa8e9408c9ceb4ea5db9ade

https://hls3020-g.live.fc2.com/hls/45390712/32/data/18858.ts?time=1560607580&hash=b5aad114e5902c87bd0844bc0824e2fc40813faf4e42d92a3767144789def4a9

https://hls3020-g.live.fc2.com/hls/45390712/32/data/18874.ts?time=1560607595&hash=ae50643387ecbb827ee846d5a457c49e67dee8ac6db9d3146999f38032b57169

https://hls3020-g.live.fc2.com/hls/45390712/32/data/18881.ts?time=1560607603&hash=82eededdcab50e1fc49e23f46f4a6f8901bec3d7cc0e617252191aa49851cc40

I also have problems downloading the playlist with just "youtube-dl link-to-playlist, but using Youtube-DLG works. The Youtube-DLG command is as follows: youtube-dl.exe --newline -i --restrict-filenames -o "D:\tempdrive\%(extractor)s\%(uploader)s\%(title)s-%(id)s.%(ext)s" --ignore-config --hls-prefer-native "https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789" Which does what I described above.

Here's a command prompt log with the --print-traffic command used:

E:\>youtube-dl.exe --newline -i --restrict-filenames -o "D:\tempdrive\%(extractor)s\%(uploader)s\%(title)s-%(id)s.%(ext)s" --ignore-config --hls-prefer-native "https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789" --no-continue --print-traffic
[generic] playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789: Requesting header
send: b'HEAD /hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789 HTTP/1.1\r\nHost: hls.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 13:59:58 GMT
header: Content-Type: application/x-mpegURL
header: Content-Length: 716
header: Connection: close
header: content-encoding: gzip
header: cache-control: no-cache
[generic] playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789: Downloading m3u8 information
send: b'GET /hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789 HTTP/1.1\r\nHost: hls.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 13:59:59 GMT
header: Content-Type: application/x-mpegURL
header: Content-Length: 715
header: Connection: close
header: content-encoding: gzip
header: cache-control: no-cache
[hlsnative] Downloading m3u8 manifest
send: b'GET /hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789 HTTP/1.1\r\nHost: hls.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:00 GMT
header: Content-Type: application/x-mpegURL
header: Content-Length: 715
header: Connection: close
header: content-encoding: gzip
header: cache-control: no-cache
[hlsnative] Total fragments: 12
[download] Destination: D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4
send: b'GET /hls/45390712/32/data/18496.ts?time=1560607218&hash=1f2cde68d58ca6f4087795ebf33bebc43985bc77d0bc025013138c59c8c499da HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:00 GMT
header: Content-Type: video/MP2T
header: Content-Length: 82344
header: Connection: close
header: cache-control: private, max-age=30
[download]   0.1% of ~964.97KiB at Unknown speed ETA 12:29
[download]   0.3% of ~964.97KiB at Unknown speed ETA 04:09
[download]   0.7% of ~964.97KiB at Unknown speed ETA 01:46
[download]   1.6% of ~964.97KiB at 87.95KiB/s ETA 00:59
[download]   3.2% of ~964.97KiB at 179.66KiB/s ETA 00:28
[download]   6.5% of ~964.97KiB at 180.47KiB/s ETA 00:16
[download]   8.3% of ~964.97KiB at 206.73KiB/s ETA 00:12
[download]   8.3% of ~964.97KiB at 206.73KiB/s ETA 00:12
send: b'GET /hls/45390712/32/data/18497.ts?time=1560607219&hash=91ad6e02dce0f68008f4ac3084c4da1e5a556503b9cb2b11e4e81e1ceb19acd6 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:01 GMT
header: Content-Type: video/MP2T
header: Content-Length: 92684
header: Connection: close
header: cache-control: private, max-age=30
[download]   7.9% of ~1.00MiB at 206.73KiB/s ETA 00:21
[download]   8.1% of ~1.00MiB at 206.73KiB/s ETA 00:21
[download]   8.5% of ~1.00MiB at 206.73KiB/s ETA 00:20
[download]   9.3% of ~1.00MiB at 90.60KiB/s ETA 00:19
[download]  10.9% of ~1.00MiB at 186.13KiB/s ETA 00:16
[download]  14.0% of ~1.00MiB at 188.09KiB/s ETA 00:13
[download]  16.7% of ~1.00MiB at 267.84KiB/s ETA 00:11
[download]  16.7% of ~1.00MiB at 267.84KiB/s ETA 00:11
send: b'GET /hls/45390712/32/data/18498.ts?time=1560607220&hash=6bcf81f544a415791868a8e0fbb4122ce97805573929df185cacf128e7731f75 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:03 GMT
header: Content-Type: video/MP2T
header: Content-Length: 102272
header: Connection: close
header: cache-control: private, max-age=30
[download]  15.9% of ~1.06MiB at 267.84KiB/s ETA 00:15
[download]  16.1% of ~1.06MiB at 267.84KiB/s ETA 00:15
[download]  16.4% of ~1.06MiB at  3.43MiB/s ETA 00:15
[download]  17.2% of ~1.06MiB at 85.05KiB/s ETA 00:15
[download]  18.6% of ~1.06MiB at 171.88KiB/s ETA 00:13
[download]  21.6% of ~1.06MiB at 171.95KiB/s ETA 00:12
[download]  25.0% of ~1.06MiB at 183.44KiB/s ETA 00:10
[download]  25.0% of ~1.06MiB at 183.44KiB/s ETA 00:10
send: b'GET /hls/45390712/32/data/18499.ts?time=1560607220&hash=f6bae95688e6091d4b1e7539d55732ad70d94c55c76ac364ef2e494df4d09c6c HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:04 GMT
header: Content-Type: video/MP2T
header: Content-Length: 85540
header: Connection: close
header: cache-control: private, max-age=30
[download]  25.6% of ~1.04MiB at 183.44KiB/s ETA 00:12
[download]  25.8% of ~1.04MiB at 183.44KiB/s ETA 00:12
[download]  26.1% of ~1.04MiB at 183.44KiB/s ETA 00:12
[download]  26.9% of ~1.04MiB at 84.50KiB/s ETA 00:12
[download]  28.4% of ~1.04MiB at 174.62KiB/s ETA 00:11
[download]  31.4% of ~1.04MiB at 178.90KiB/s ETA 00:10
[download]  33.3% of ~1.04MiB at 233.25KiB/s ETA 00:09
[download]  33.3% of ~1.04MiB at 233.25KiB/s ETA 00:09
send: b'GET /hls/45390712/32/data/18500.ts?time=1560607222&hash=81a945a85062ce81a1034285820d584cca08b5651e6ada4977a33edffea60fe1 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:05 GMT
header: Content-Type: video/MP2T
header: Content-Length: 81968
header: Connection: close
header: cache-control: private, max-age=30
[download]  34.1% of ~1.02MiB at 233.25KiB/s ETA 00:10
[download]  34.3% of ~1.02MiB at 233.25KiB/s ETA 00:10
[download]  34.7% of ~1.02MiB at 233.25KiB/s ETA 00:10
[download]  35.4% of ~1.02MiB at 87.60KiB/s ETA 00:10
[download]  37.0% of ~1.02MiB at 177.93KiB/s ETA 00:09
[download]  40.0% of ~1.02MiB at 180.81KiB/s ETA 00:08
[download]  41.7% of ~1.02MiB at 228.43KiB/s ETA 00:07
[download]  41.7% of ~1.02MiB at 228.43KiB/s ETA 00:07
send: b'GET /hls/45390712/32/data/18501.ts?time=1560607223&hash=f0659fa336a7205ffd7de7c9d2e0bff33efbc77fbf7e52d6943bad335da81d9d HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:06 GMT
header: Content-Type: video/MP2T
header: Content-Length: 105844
header: Connection: close
header: cache-control: private, max-age=30
[download]  40.5% of ~1.05MiB at 228.43KiB/s ETA 00:09
[download]  40.7% of ~1.05MiB at 228.43KiB/s ETA 00:09
[download]  41.0% of ~1.05MiB at 228.43KiB/s ETA 00:09
[download]  41.8% of ~1.05MiB at 84.25KiB/s ETA 00:09
[download]  43.3% of ~1.05MiB at 171.25KiB/s ETA 00:08
[download]  46.2% of ~1.05MiB at 173.05KiB/s ETA 00:07
[download]  50.0% of ~1.05MiB at 189.81KiB/s ETA 00:07
[download]  50.0% of ~1.05MiB at 189.81KiB/s ETA 00:07
send: b'GET /hls/45390712/32/data/18502.ts?time=1560607224&hash=84dae1bf5590ed1bac6fe7065e79af81c8593387d1a5c2bcb50de69c424b8ab0 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:07 GMT
header: Content-Type: video/MP2T
header: Content-Length: 121448
header: Connection: close
header: cache-control: private, max-age=30
[download]  47.9% of ~1.10MiB at 189.81KiB/s ETA 00:08
[download]  48.1% of ~1.10MiB at 189.81KiB/s ETA 00:08
[download]  48.4% of ~1.10MiB at 189.81KiB/s ETA 00:08
[download]  49.1% of ~1.10MiB at 91.70KiB/s ETA 00:08
[download]  50.5% of ~1.10MiB at 149.36KiB/s ETA 00:07
[download]  53.4% of ~1.10MiB at 187.03KiB/s ETA 00:07
[download]  58.3% of ~1.10MiB at 233.29KiB/s ETA 00:05
[download]  58.3% of ~1.10MiB at 233.29KiB/s ETA 00:05
send: b'GET /hls/45390712/32/data/18503.ts?time=1560607225&hash=ef796994694aadf19a89cc627283d528259de120e63b91f26490f8a0c9f9bf6e HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:09 GMT
header: Content-Type: video/MP2T
header: Content-Length: 84788
header: Connection: close
header: cache-control: private, max-age=30
[download]  59.3% of ~1.08MiB at 233.29KiB/s ETA 00:06
[download]  59.5% of ~1.08MiB at 233.29KiB/s ETA 00:06
[download]  59.8% of ~1.08MiB at 233.29KiB/s ETA 00:06
[download]  60.6% of ~1.08MiB at 83.09KiB/s ETA 00:06
[download]  62.0% of ~1.08MiB at 169.85KiB/s ETA 00:05
[download]  64.9% of ~1.08MiB at 180.74KiB/s ETA 00:05
[download]  66.7% of ~1.08MiB at 235.53KiB/s ETA 00:04
[download]  66.7% of ~1.08MiB at 235.53KiB/s ETA 00:04
send: b'GET /hls/45390712/32/data/18504.ts?time=1560607226&hash=2f09ed47dd018149bc4ab9dbe20dd658f8cab67e454c0693e1dc6b7717cb8f65 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:10 GMT
header: Content-Type: video/MP2T
header: Content-Length: 84788
header: Connection: close
header: cache-control: private, max-age=30
[download]  67.5% of ~1.07MiB at 235.53KiB/s ETA 00:04
[download]  67.7% of ~1.07MiB at 235.53KiB/s ETA 00:04
[download]  68.1% of ~1.07MiB at 235.53KiB/s ETA 00:04
[download]  68.8% of ~1.07MiB at 89.68KiB/s ETA 00:04
[download]  70.3% of ~1.07MiB at 185.34KiB/s ETA 00:04
[download]  73.2% of ~1.07MiB at 185.08KiB/s ETA 00:03
[download]  75.0% of ~1.07MiB at 219.46KiB/s ETA 00:03
[download]  75.0% of ~1.07MiB at 219.46KiB/s ETA 00:03
send: b'GET /hls/45390712/32/data/18505.ts?time=1560607227&hash=f9b7e85cb34f5b5aa0c628069849dd45c13c44469ed2f6f5f51825e1d2ff9876 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:11 GMT
header: Content-Type: video/MP2T
header: Content-Length: 117312
header: Connection: close
header: cache-control: private, max-age=30
[download]  73.2% of ~1.10MiB at 219.46KiB/s ETA 00:04
[download]  73.4% of ~1.10MiB at 219.46KiB/s ETA 00:04
[download]  73.8% of ~1.10MiB at 219.46KiB/s ETA 00:04
[download]  74.5% of ~1.10MiB at 87.36KiB/s ETA 00:03
[download]  75.9% of ~1.10MiB at 178.47KiB/s ETA 00:03
[download]  78.7% of ~1.10MiB at 180.60KiB/s ETA 00:03
[download]  83.3% of ~1.10MiB at 218.62KiB/s ETA 00:02
[download]  83.3% of ~1.10MiB at 218.62KiB/s ETA 00:02
send: b'GET /hls/45390712/32/data/18506.ts?time=1560607228&hash=af30def3f911e2b9569200b1231553ed5582c418396f92a1a4427e21c64d5de4 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:12 GMT
header: Content-Type: video/MP2T
header: Content-Length: 88736
header: Connection: close
header: cache-control: private, max-age=30
[download]  84.0% of ~1.09MiB at 218.62KiB/s ETA 00:02
[download]  84.2% of ~1.09MiB at 218.62KiB/s ETA 00:02
[download]  84.5% of ~1.09MiB at 218.62KiB/s ETA 00:02
[download]  85.2% of ~1.09MiB at 90.33KiB/s ETA 00:02
[download]  86.7% of ~1.09MiB at 185.56KiB/s ETA 00:01
[download]  89.5% of ~1.09MiB at 187.72KiB/s ETA 00:01
[download]  91.7% of ~1.09MiB at 255.17KiB/s ETA 00:01
[download]  91.7% of ~1.09MiB at 255.17KiB/s ETA 00:01
send: b'GET /hls/45390712/32/data/18507.ts?time=1560607228&hash=23109695223bfb1e3d65ea9201a0e64a9fa7268f1d86cc99572d6ffcd0d54021 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: hls3020-g.live.fc2.com\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Sat, 15 Jun 2019 14:00:13 GMT
header: Content-Type: video/MP2T
header: Content-Length: 128780
header: Connection: close
header: cache-control: private, max-age=30
[download]  89.1% of ~1.12MiB at 255.17KiB/s ETA 00:01
[download]  89.3% of ~1.12MiB at 255.17KiB/s ETA 00:01
[download]  89.7% of ~1.12MiB at 255.17KiB/s ETA 00:01
[download]  90.4% of ~1.12MiB at 86.44KiB/s ETA 00:01
[download]  91.8% of ~1.12MiB at 177.62KiB/s ETA 00:01
[download]  94.5% of ~1.12MiB at 183.40KiB/s ETA 00:00
[download] 100.0% of ~1.12MiB at 239.94KiB/s ETA 00:00
[download] 100.0% of ~1.12MiB at 239.94KiB/s ETA 00:00
[download] 100% of 1.12MiB in 00:14
[ffmpeg] Fixing malformed AAC bitstream in "D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4"

--verbose log:

E:\>youtube-dl.exe --newline -i --restrict-filenames -o "D:\tempdrive\%(extractor)s\%(uploader)s\%(title)s-%(id)s.%(ext)s" --ignore-config --hls-prefer-native "https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789" --no-continue --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--newline', '-i', '--restrict-filenames', '-o', 'D:\\tempdrive\\%(extractor)s\\%(uploader)s\\%(title)s-%(id)s.%(ext)s', '--ignore-config', '--hls-prefer-native', 'https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789', '--no-continue', '--verbose']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2019.05.20
[debug] Python version 3.7.3 (CPython) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg 3.4, ffprobe 4.1.3, rtmpdump 2.3
[debug] Proxy map: {}
[generic] playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789: Requesting header
[generic] playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://hls.live.fc2.com/hls/45390712/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 12
[download] Destination: D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4
[download]   0.1% of ~773.30KiB at 30.38KiB/s ETA 09:25
[download]   0.4% of ~773.30KiB at 88.47KiB/s ETA 03:08
[download]   0.9% of ~773.30KiB at 200.53KiB/s ETA 01:20
[download]   1.9% of ~773.30KiB at 84.61KiB/s ETA 00:44
[download]   4.0% of ~773.30KiB at 168.24KiB/s ETA 00:21
[download]   8.1% of ~773.30KiB at 180.61KiB/s ETA 00:11
[download]   8.3% of ~773.30KiB at 184.21KiB/s ETA 00:11
[download]   8.3% of ~773.30KiB at 184.21KiB/s ETA 00:11
[download]   8.0% of ~821.77KiB at 184.21KiB/s ETA 00:20
[download]   8.2% of ~821.77KiB at 184.21KiB/s ETA 00:19
[download]   8.7% of ~821.77KiB at  3.43MiB/s ETA 00:18
[download]   9.7% of ~821.77KiB at 87.95KiB/s ETA 00:18
[download]  11.6% of ~821.77KiB at 180.71KiB/s ETA 00:14
[download]  15.5% of ~821.77KiB at 183.10KiB/s ETA 00:11
[download]  16.7% of ~821.77KiB at 209.54KiB/s ETA 00:10
[download]  16.7% of ~821.77KiB at 209.54KiB/s ETA 00:10
[download]  16.8% of ~820.30KiB at 209.54KiB/s ETA 00:13
[download]  17.1% of ~820.30KiB at 209.54KiB/s ETA 00:13
[download]  17.5% of ~820.30KiB at 209.54KiB/s ETA 00:13
[download]  18.5% of ~820.30KiB at 85.46KiB/s ETA 00:13
[download]  20.5% of ~820.30KiB at 174.62KiB/s ETA 00:11
[download]  24.4% of ~820.30KiB at 179.47KiB/s ETA 00:09
[download]  25.0% of ~820.30KiB at 192.94KiB/s ETA 00:09
[download]  25.0% of ~820.30KiB at 192.94KiB/s ETA 00:09
[download]  24.2% of ~851.51KiB at 192.94KiB/s ETA 00:12
[download]  24.4% of ~851.51KiB at 192.94KiB/s ETA 00:12
[download]  24.9% of ~851.51KiB at  2.29MiB/s ETA 00:11
[download]  25.8% of ~851.51KiB at 84.98KiB/s ETA 00:11
[download]  27.7% of ~851.51KiB at 174.63KiB/s ETA 00:10
[download]  31.5% of ~851.51KiB at 178.11KiB/s ETA 00:09
[download]  33.3% of ~851.51KiB at 220.80KiB/s ETA 00:08
[download]  33.3% of ~851.51KiB at 220.80KiB/s ETA 00:08
[download]  33.2% of ~859.22KiB at 220.80KiB/s ETA 00:10
[download]  33.4% of ~859.22KiB at 220.80KiB/s ETA 00:10
[download]  33.8% of ~859.22KiB at  3.43MiB/s ETA 00:09
[download]  34.8% of ~859.22KiB at 85.94KiB/s ETA 00:09
[download]  36.6% of ~859.22KiB at 175.61KiB/s ETA 00:08
[download]  40.4% of ~859.22KiB at 179.46KiB/s ETA 00:07
[download]  41.7% of ~859.22KiB at 210.09KiB/s ETA 00:07
[download]  41.7% of ~859.22KiB at 210.09KiB/s ETA 00:07
[download]  41.3% of ~868.40KiB at 210.09KiB/s ETA 00:08
[download]  41.6% of ~868.40KiB at 210.09KiB/s ETA 00:08
[download]  42.0% of ~868.40KiB at 210.09KiB/s ETA 00:08
[download]  43.0% of ~868.40KiB at 86.92KiB/s ETA 00:08
[download]  44.8% of ~868.40KiB at 177.62KiB/s ETA 00:07
[download]  48.5% of ~868.40KiB at 179.45KiB/s ETA 00:06
[download]  50.0% of ~868.40KiB at 215.81KiB/s ETA 00:06
[download]  50.0% of ~868.40KiB at 215.81KiB/s ETA 00:06
[download]  49.8% of ~873.70KiB at 996.98KiB/s ETA 00:07
[download]  50.0% of ~873.70KiB at  2.92MiB/s ETA 00:07
[download]  50.5% of ~873.70KiB at  3.42MiB/s ETA 00:07
[download]  51.4% of ~873.70KiB at 88.24KiB/s ETA 00:06
[download]  53.2% of ~873.70KiB at 180.25KiB/s ETA 00:06
[download]  56.9% of ~873.70KiB at 184.46KiB/s ETA 00:05
[download]  58.3% of ~873.70KiB at 219.64KiB/s ETA 00:05
[download]  58.3% of ~873.70KiB at 219.64KiB/s ETA 00:05
[download]  58.3% of ~876.57KiB at 219.64KiB/s ETA 00:05
[download]  58.5% of ~876.57KiB at 219.64KiB/s ETA 00:05
[download]  58.9% of ~876.57KiB at  3.43MiB/s ETA 00:05
[download]  59.9% of ~876.57KiB at 89.00KiB/s ETA 00:05
[download]  61.7% of ~876.57KiB at 181.77KiB/s ETA 00:05
[download]  65.3% of ~876.57KiB at 186.89KiB/s ETA 00:04
[download]  66.7% of ~876.57KiB at 218.43KiB/s ETA 00:04
[download]  66.7% of ~876.57KiB at 218.43KiB/s ETA 00:04
[download]  65.6% of ~892.27KiB at 218.43KiB/s ETA 00:04
[download]  65.8% of ~892.27KiB at  1.47MiB/s ETA 00:04
[download]  66.3% of ~892.27KiB at  3.43MiB/s ETA 00:04
[download]  67.2% of ~892.27KiB at 85.45KiB/s ETA 00:04
[download]  69.0% of ~892.27KiB at 176.61KiB/s ETA 00:04
[download]  72.6% of ~892.27KiB at 178.51KiB/s ETA 00:03
[download]  75.0% of ~892.27KiB at 213.21KiB/s ETA 00:03
[download]  75.0% of ~892.27KiB at 213.21KiB/s ETA 00:03
[download]  74.3% of ~902.40KiB at 213.21KiB/s ETA 00:03
[download]  74.5% of ~902.40KiB at 213.21KiB/s ETA 00:03
[download]  74.9% of ~902.40KiB at  3.43MiB/s ETA 00:03
[download]  75.8% of ~902.40KiB at 70.28KiB/s ETA 00:03
[download]  77.6% of ~902.40KiB at 117.29KiB/s ETA 00:03
[download]  81.1% of ~902.40KiB at 145.21KiB/s ETA 00:02
[download]  83.3% of ~902.40KiB at 189.12KiB/s ETA 00:02
[download]  83.3% of ~902.40KiB at 189.12KiB/s ETA 00:02
[download]  82.9% of ~908.29KiB at 189.12KiB/s ETA 00:02
[download]  83.1% of ~908.29KiB at  1.47MiB/s ETA 00:02
[download]  83.6% of ~908.29KiB at  2.28MiB/s ETA 00:02
[download]  84.4% of ~908.29KiB at 85.68KiB/s ETA 00:02
[download]  86.2% of ~908.29KiB at 174.09KiB/s ETA 00:01
[download]  89.7% of ~908.29KiB at 178.67KiB/s ETA 00:01
[download]  91.7% of ~908.29KiB at 225.39KiB/s ETA 00:01
[download]  91.7% of ~908.29KiB at 225.39KiB/s ETA 00:01
[download]  91.9% of ~906.95KiB at 225.39KiB/s ETA 00:01
[download]  92.1% of ~906.95KiB at 225.39KiB/s ETA 00:01
[download]  92.6% of ~906.95KiB at  2.28MiB/s ETA 00:01
[download]  93.5% of ~906.95KiB at 85.96KiB/s ETA 00:00
[download]  95.2% of ~906.95KiB at 174.65KiB/s ETA 00:00
[download]  98.7% of ~906.95KiB at 181.01KiB/s ETA 00:00
[download] 100.0% of ~906.95KiB at 211.82KiB/s ETA 00:00
[download] 100.0% of ~906.95KiB at 211.82KiB/s ETA 00:00
ERROR: unable to rename file: [WinError 183] Det går inte att skapa en fil som redan finns: 'D:\\tempdrive\\generic\\NA\\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4.part' -> 'D:\\tempdrive\\generic\\NA\\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4'
Traceback (most recent call last):
  File "c:\users\***\appdata\local\programs\python\python37\lib\site-packages\youtube_dl\downloader\common.py", line 200, in try_rename
    os.rename(encodeFilename(old_filename), encodeFilename(new_filename))
FileExistsError: [WinError 183] Det går inte att skapa en fil som redan finns: 'D:\\tempdrive\\generic\\NA\\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4.part' -> 'D:\\tempdrive\\generic\\NA\\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4'

[download] 100% of 1.01MiB in 00:13
[debug] ffmpeg command line: ffprobe -show_streams "file:D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4"
[ffmpeg] Fixing malformed AAC bitstream in "D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.mp4" -c copy -f mp4 "-bsf:a" aac_adtstoasc "file:D:\tempdrive\generic\NA\playlist-playlistsession_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4_playlist_token=13bf824970d303817086a6d6a90dac8477d17452aed589560e3d7125cc86c789.temp.mp4"

If you take a look at this, thanks a lot for your time. And if not, I understand, thanks anyway, youtube-dl is awesome.

m4gnet98 commented 5 years ago

ffmpeg works just fine with just the playlist-link as input, continuously records from there on.

johnthecracker commented 5 years ago

yes, but when I want to script it 24-7 it is pain getting playlist. It would be straightforward job with youtube-dl working with main url.

youtube-dl must work with "politically correct" url like https://live.fc2.com/39828463/

Here is one link to my old post that was labeled invalid because url was not acceptable. It may be for legal reasons. I see posts and users removed from github because of censorship. this is probably why url for youtube-dl can't be other than main url, https://live.fc2.com/39828463/

https://github.com/ytdl-org/youtube-dl/issues/14684

I posted main url in this issue, https://live.fc2.com/67780606/, and here isn't much interest from developers side.

Your post is appreciated by me at the least, looks like we are on our own with content of this website.

m4gnet98 commented 5 years ago

@johnthecracker I made a batch file for some "minor" automation... (I'm bad lol)

First though you'll have to go to the stream page, (i.e https://live.fc2.com/67780606/ ), open the Developer tools (clicking Ctrl+Shift+I?), click the Network tab, click the "_playlist?sessiontoken=xxx", and copy the link under General and Request URL.

The correct link will look something like this:

https://hls.live.fc2.com/hls/69985628/32/playlist?session_token=2a42f3f6c14de84e286b5491a800c3f0039c030381040799e4ef18115bb55cc4&playlist_token=fa4ef35b413969302ddeeb0d886015ec595b05aafef06f8ca5b09516157aacbf

Open the .bat file, paste the playlist url, Enter, enter the name you want, Enter, and it'll download it for you..

You'll have to edit the batch a bit to fit your download location etc. Mine are saved in "e:\pyri\fc2\"

@echo off
set /p "address=Video Address:"
set /p "filename=File name:"

FOR /F "tokens=1,2 delims=:." %%A IN ("%Time%") DO (
    SET Hours=%%A
    SET Minutes=%%B
)
SET /A Hours = 100%Hours% %% 100
ECHO.%Minutes% | FIND /I "P" >NUL && SET /A Hours += 12
SET Minutes=%Minutes:~0,2%
SET /A Minutes = 100%Minutes% %% 100

md e:\pyri\fc2\%filename%

ffmpeg -i "%address%" -c copy "e:\pyri\fc2\%filename%\%filename%_%Date%_%Hours%-%Minutes%.mp4"

Copy this code, make a new txt file, paste it in there and "save as all files" as whatever.bat

Good luck following my crappy instructions lol

johnthecracker commented 5 years ago

Thanks,

My aim is to get playlist without opening web browser like I do it for world news web sites, and who knows who where else. I can write scripts and batch files. Here is a program that works just fine for https://live.fc2.com/adult/:

https://mastodon.cloud/@ctbrec

and my advice is to run this program in linux, not in windows. The reason for that is that in windows cpu gets very hot, and in linux remains ice cold. It is beyond me how this program is written. It works perfectly fine for bunch of webcam sites, all without web browser.

https://live.fc2.com/adult/ should ideally work with youtube-dl as https://www.trtworld.com/live does. Sites are very different, but I must talk about main url only for youtube-dl support.

ytdl-org / youtube-dl

live.fc2.com support #21096

Description