search

BurntSushi / nflvid

An experimental library to map play meta data to footage of that play.
The Unlicense
91 stars 17 forks source link

Broadcast Footage -> Always Fails #5

Closed manicNFL closed 11 years ago

manicNFL commented 11 years ago

While I understand the "Broadcast Footage" feature is still experimental, personally I'd love to see it successfully implemented.

I was messing around with it today, and found that the download ALWAYS fails when I specify the broadcast footage.

Example of a command I attempted:

$ nflvid-footage --season 2012 --season-type POST --weeks 4 --broadcast --quality 4500 --  /home/nflvid/full
These are the games that match your search criteria:
(Season: 2013, Week: 4, BAL (34) at SF (31))
Are you sure you want to start downloading? [y/n] y
BAD URLs for game (Season: 2013, Week: 4, BAL (34) at SF (31)): http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/02/03/55835/2_55835_bal_sf_2012_h_whole_1_4500.mp4.m3u8, http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/02/03/55835/2_55835_bal_sf_2012_h_whole_2_4500.mp4.m3u8, http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/02/03/55835/2_55835_bal_sf_2012_h_whole_3_4500.mp4.m3u8
FAILED to download game (Season: 2013, Week: 4, BAL (34) at SF (31))

Coaches footage of the same game above seems to download fine.

BurntSushi commented 11 years ago

I'm running short on time and I'll be away most of the weekend, so I apologize in advance for the terse response. But in any case, thank you! You are my first person to test this stuff. :-)

Broadcast footage is incredibly finicky. In fact, I've been unable to get pre or post season games to work. When I get a chance, I'll show you the URL to last year's super bowl, which is subtly different from regular season game urls. And even with that, I couldn't manage to play it in VLC. Not sure what's going on.

Please try downloading a regular season game.

Additionally, downloading broadcast footage seems to be much more restrictive. If I do too much at a time, the downloads seem to cut out. One at a time seems to work reasonably well, but there are still drops.

This is completely unlike coach footage, where I can download multiple games simultaneously. Occasionally there is a failure, but it's on the order of several in each season. (For me anyway...) Note that broadcast footage uses Apple's HTTP Live Streaming while coach footage uses an rtmp stream.

although the file downloaded doesn't seem to want to play in anything at the moment (could just be me).

Are you referring to coach footage? Could you please state which video players you've tried and any relevant error messages? I pretty much exclusively use vlc, and it has worked fine. mplayer might also work.

BurntSushi commented 11 years ago

Here is the URL to the superbowl:

http://nlds82.cdnak.neulion.com/nlds_vod/nfl/vod/2013/02/03/55835/3_55835_bal_sf_2012_h_whole_2_4500.mp4.m3u8
manicNFL commented 11 years ago

I believe my issue with superbowl coaches footage download not playing is due to a broken download (there was no error from nflvid though, didn't even say it was incomplete). the command I'm using to get the file:

$ nflvid-footage --season 2012 --season-type POST --weeks 4 -- /home/nflvid//full
These are the games that match your search criteria:
(Season: 2013, Week: 4, BAL (34) at SF (31))
Are you sure you want to start downloading? [y/n] y
Downloading game 2013020300 (Season: 2013, Week: 4, BAL (34) at SF (31))
DONE with game 2013020300 (Season: 2013, Week: 4, BAL (34) at SF (31))

the resulting file can not be played by ffplay, vlc, or mplayer.

Just tried this again, and the file was bigger the the previous time, but still won't play.

Here's the log when I try to play it in mplayer:

$ mplayer 2013020300.mp4 
MPlayer svn r34540 (Ubuntu), built with gcc-4.6 (C) 2000-2012 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.
Playing 2013020300.mp4.
libavformat version 53.21.1 (external)
Mismatching header version 53.19.0
libavformat file format detected.
[flv @ 0xb6bce3a0]Unsupported video codec (0)
[flv @ 0xb6bce3a0]negative cts, previous timestamps might be wrong
[h264 @ 0xb64879e0]AVC: nal size 1014137051
[h264 @ 0xb64879e0]no frame!
[flv @ 0xb6bce3a0]Estimating duration from bitrate, this may be inaccurate
[lavf] stream 0: video (h264), -vid 0
VIDEO:  [H264]  640x480  0bpp  29.970 fps    0.0 kbps ( 0.0 kbyte/s)
Clip info:
 moovPosition: 32
 avcprofile: 77
 avclevel: 30
 videoframerate: 30
Load subtitles in ./
Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: No such file or directory
[vdpau] Error when calling vdp_device_create_x11: 1
[VO_XV] It seems there is no Xvideo support for your video card available.
[VO_XV] Run 'xvinfo' to verify its Xv support and read
[VO_XV] DOCS/HTML/en/video.html#xv!
[VO_XV] See 'mplayer -vo help' for other (non-xv) video out drivers.
[VO_XV] Try -vo x11.
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 53.35.0 (external)
Mismatching header version 53.32.2
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
Audio: no sound
Starting playback...
Unsupported PixelFormat 61
Unsupported PixelFormat 53
Unsupported PixelFormat 81
[h264 @ 0xb64879e0]AVC: nal size 1014137051
[h264 @ 0xb64879e0]no frame!
Error while decoding frame!
V:   0.0   0/  0 ??% ??% ??,?% 0 0 
Exiting... (End of file)

Log, from when I try to play it with ffplay:

$ ffplay 2013020300.mp4 
ffplay version git-2013-09-06-bcd1c20 Copyright (c) 2003-2013 the FFmpeg developers
  built on Sep  6 2013 03:05:08 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: --prefix=/home/nflvid/ffmpeg_build --extra-cflags=-I/home/nflvid/ffmpeg_build/include --extra-ldflags=-L/home/nflvid/ffmpeg_build/lib --bindir=/home/nflvid/bin --extra-libs=-ldl --enable-gpl --enable-nonfree --enable-x11grab
  libavutil      52. 43.100 / 52. 43.100
  libavcodec     55. 31.101 / 55. 31.101
  libavformat    55. 16.101 / 55. 16.101
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 83.104 /  3. 83.104
  libswscale      2.  5.100 /  2.  5.100
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
Input #0, flv, from '2013020300.mp4':B vq=    0KB sq=    0B f=0/0   
  Metadata:
    moovPosition    : 32
    avcprofile      : 77
    avclevel        : 30
    videoframerate  : 30
  Duration: 01:24:43.12, start: 0.033000, bitrate: 1599 kb/s
    Stream #0:0: Video: h264 (Main), yuv420p, 640x480, 29.97 tbr, 1k tbn, 59.94 tbc
Segmentation fault (core dumped)   0KB vq=    0KB sq=    0B f=0/0

here's the log when I try and play it with VLC:

$ cvlc 2013020300.mp4 
VLC media player 2.0.8 Twoflower (revision 2.0.8a-0-g68cf50b)
[0x8931520] dummy interface: using the dummy interface module...
[flv @ 0x8928180] Unsupported video codec (0)
[flv @ 0x8928180] negative cts, previous timestamps might be wrong
[h264 @ 0x8928b80] AVC: nal size 1014137051
[h264 @ 0x8928b80] no frame!
[flv @ 0x8928180] Estimating duration from bitrate, this may be inaccurate
[flv @ 0xb5111440] Unsupported video codec (0)
[flv @ 0xb5111440] negative cts, previous timestamps might be wrong
[h264 @ 0xb5113200] AVC: nal size 1014137051
[h264 @ 0xb5113200] no frame!
[flv @ 0xb5111440] Estimating duration from bitrate, this may be inaccurate
[0xb5181dd0] xcb_xv vout display error: no available XVideo adaptor
[h264 @ 0x88dee20] AVC: nal size 1014137051
[h264 @ 0x88dee20] no frame!

Personally, I have no issues downloading one game at a time. perhaps when the "--broadcast" switch is used, "--threads" is defaulted to '1' ?

Thanks for the SB link, I've got it processing with ffmpeg (only thing that would take it) I'll let it run and then report back when it's done.

I will try a regular season game to test and see if I can get the broadcast footage, once the SB finishes processing.

One thing I did notice about one of the coaches videos I did manage to download successfully... command used:

$ nflvid-footage --season 2012 --season-type POST --weeks 3 --teams NE -- /home/nflvid/full
These are the games that match your search criteria:
(Season: 2013, Week: 3, BAL (28) at NE (13))
Are you sure you want to start downloading? [y/n] y
Downloading game 2013012000 (Season: 2013, Week: 3, BAL (28) at NE (13))
DONE with game 2013012000 (Season: 2013, Week: 3, BAL (28) at NE (13))

... I noticed that (like may mp4 files) all players (ffplay, vlc, mplayer) seemed to struggle with parsing the whole file for runtime and seeking information.

MKV is a much better container, and typically more light-weight than mp4.

I ran the following command, and the resulting MKV file played and seeked like butter in all player I tested.

$ ffmpeg -i 2013012000.mp4 -c copy 2013012000.mkv

Perhaps the default file format could be changed to MKV?

BurntSushi commented 11 years ago

Doesn't changing the format require transcoding the video? If so, there's no way I could make it the default operation. I wouldn't be averse to adding a flag to nflvid-footage, though.

The error messages suck pretty hard right now, apparently. Namely, they appear to be nonexistent!

Personally, I have no issues downloading one game at a time. perhaps when the "--broadcast" switch is used, "--threads" is defaulted to '1' ?

That is a good idea. I'll make the change as soon as I can.

Thanks for the SB link, I've got it processing with ffmpeg (only thing that would take it) I'll let it run and then report back when it's done.

Please let me know how that turns out!

One thing I did notice about one of the coaches videos I did manage to download successfully...

Interesting. I didn't know you were able to download some successfully. Note that I haven't experienced the same sort of problems as you. Seeking and what not work great. Here's my vlc output (I opened a video and seeked halfway through the game):

[andrew@Serval full] vlc 2012123007.mp4 
VLC media player 2.0.8 Twoflower (revision 2.0.8a-0-g68cf50b)
[0x1ffb138] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Fontconfig warning: FcPattern object size does not accept value "0"
Fontconfig warning: FcPattern object size does not accept value "0"
[andrew@Serval full]

One other thing I can think of is to check your rtmpdump, which is what is used to download coach footage. I should have mentioned this in my first comment, but I forgot (my excuse is that I had just woken up). Namely, every version of rtmpdump has spectacularly and mysteriously failed for me except for one: rtmpdump with KSV's patch. You can get the full code with the patch applied at BurntSushi/rtmpdump-ksv. I think cloning and a make should do the trick (although I had to staticly compile with make SHARED=). (Errm, if you're on Archlinux, just use the rtmpdump-ksv package in the AUR.)

If things are still failing, then the next step is to start issuing the rtmpdump commands directly and see what's going on. I'll point you to the coach footage download code here. You can get the data you need for a particular game with something like:

>>> import nflvid
>>> import nflgame
>>> g = nflgame.game.Game('2012090500')
>>> print nflvid.coach_url(g)
('rtmp://neulionms.fcod.llnwd.net', 'a5306/e1', u'mp4:u/nfl/nfl/coachtapes/2012/55504_all_1600')

The first value is the server, the second is the app and the last is the playpath. So this should work:

rtmpdump --rtmp 'rtmp://neulionms.fcod.llnwd.net' --app 'a5306/e1' --playpath 'mp4:u/nfl/nfl/coachtapes/2012/55504_all_1600' -o 2012090500.mp4

Note that I've never been able to get ffmpeg to download an rtmp stream successfully. I imagine this is because ffmpeg is using a different version of rtmpdump? (And like I said, every version except for the latest + KSV's patch has failed spectacularly.)

Thank you for hanging in there! I really appreciate it. FYI, I may not be able to respond until tomorrow morning...

BurntSushi commented 11 years ago

Interesting! I'm on a different machine with a run-of-the-mill rtmpdump. I ran the rtmpdump command in my previous comment. It downloaded fine. But when I opened it with VLC...

[andrew@Serval ~] vlc test.mp4 
VLC media player 2.0.8 Twoflower (revision 2.0.8a-0-g68cf50b)
[0xf0d138] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[flv @ 0x7fd1fcc1f220] Stream discovered after head already parsed
[flv @ 0x7fd1fcc1f220] Unsupported audio codec (d)
Truncating packet of size 16736019 to 140942
[flv @ 0x7fd1fcc1f220] decoding for stream 0 failed
[flv @ 0x7fd1fcc1f220] decoding for stream 1 failed
[flv @ 0x7fd1fcc1f220] Could not find codec parameters (Audio: none ([13][0][0][0] / 0x000D), 5512 Hz, 2 channels)
[flv @ 0x7fd1fcc228e0] Stream discovered after head already parsed
[flv @ 0x7fd1fcc228e0] Unsupported audio codec (d)
Truncating packet of size 16736019 to 140942
[flv @ 0x7fd1fcc228e0] decoding for stream 0 failed
[flv @ 0x7fd1fcc228e0] decoding for stream 1 failed
[flv @ 0x7fd1fcc228e0] Could not find codec parameters (Audio: none ([13][0][0][0] / 0x000D), 5512 Hz, 2 channels)
[0x7fd1fc002558] main decoder error: no suitable decoder module for fourcc `undf'. VLC probably does not support this sound or video format.
Fontconfig warning: FcPattern object size does not accept value "0"
Fontconfig warning: FcPattern object size does not accept value "0"
[andrew@Serval ~]

Kaboom. I'm going to install ksv's patch and try again. Note that if you're on Archlinux, there is a rtmpdump-ksv package in the AUR.

BurntSushi commented 11 years ago

OK. Just tried out the redownload with ksv's patch, and all works well. So hopefully that's the problem.

manicNFL commented 11 years ago

alright, installed rtmpdump-ksv package and now broadcast games seem to be working. but now I'm trying to test that one coaches footage that was problematic before using this command:

$ nflvid-footage --season 2012 --season-type POST --weeks 4 -- /home/nflvid/full

But I get the following error returned:

No games matched your search criteria.

odd, to say the least, since the game was at least found earlier.

SB game came down brilliantly, using ffmpeg, going directly into an MKV

when I say changing the file format, what I really mean is changing the container. there's no encoding involved, you're just copying the streams into the new container. the following is the command I used to go from a problematic mp4 to a perfectly fine MKV:

ffmpeg -i NFLGAME.mp4 -c copy NFLGAME.mkv

I also attempted to see if last nights opening game was available yet (which it might not be), here's the command and the output:

$ nflvid-footage --season 2013 --weeks 1 -- /home/nflvid/full
Traceback (most recent call last):
  File "/usr/local/bin/nflvid-footage", line 101, in 
    and nflvid.footage_full(args.footage_dir, g.eid) is not None:
AttributeError: 'NoneType' object has no attribute 'eid'

didn't know if it's a bug in your system, or if the game just isn't available yet. so I figure I'd mention it anyways, just to be safe.

Thanks again - it's working better than when I started with it already!

manicNFL commented 11 years ago

ok, actually it's not working, well, coaches footage now anyways...

$ nflvid-footage --season 2012 --season-type POST --weeks 4 -- /home/nflvid/full/
These are the games that match your search criteria:
(Season: 2013, Week: 4, BAL (34) at SF (31))
Are you sure you want to start downloading? [y/n] y
Downloading game 2013020300 (Season: 2013, Week: 4, BAL (34) at SF (31))
Could not run 'rtmpdump --rtmp rtmp://neulionms.fcod.llnwd.net --app a5306/e1 --playpath mp4:u/nfl/nfl/coachtapes/2012/55835_all_1600 --timeout 60 -o /home/nflvid/full/2013020300.mp4' (errno: 2): No such file or directory
FAILED to download game 2013020300 (Season: 2013, Week: 4, BAL (34) at SF (31))

Not quite sure it's having issues, the directory is the same as when I do the broadcast footage (which works fine). it seems to have issues creating the file? I'm a little confused.

EDIT - definitely not working for any coaches footage:

$ nflvid-footage --season 2012 --teams NE --weeks 5 -- /home/nflvid/full/
These are the games that match your search criteria:
(Season: 2012, Week: 5, DEN (21) at NE (31))
Are you sure you want to start downloading? [y/n] y
Downloading game 2012100709 (Season: 2012, Week: 5, DEN (21) at NE (31))
Could not run 'rtmpdump --rtmp rtmp://neulionms.fcod.llnwd.net --app a5306/e1 --playpath mp4:u/nfl/nfl/coachtapes/2012/55577_all_1600 --timeout 60 -o /home/nflvid/full/2012100709.mp4' (errno: 2): No such file or directory
FAILED to download game 2012100709 (Season: 2012, Week: 5, DEN (21) at NE (31))

I've found that if I use the "--show-url" switch, and then use the URL directly with ffmpeg, I can download the footage without issue.

BurntSushi commented 11 years ago

I am now back and hopefully will be able to respond more quickly.

Not quite sure it's having issues, the directory is the same as when I do the broadcast footage (which works fine). it seems to have issues creating the file?

One of the "features" of nflvid-footage (and nflvid-slice) is that they are idempotent. This means that if you download game footage to a directory coach, then it will create a file called coach/eid.mp4. If you run the exact command again, it will notice the file coach/eid.mp4 and ignore it. This is what "Could not find any games matching criteria" means. I think it should say, "Could not find any games matching the criteria that haven't already been downloaded."

The problem with this feature is that if there was a problem downloading the footage, sometimes it will still create a (possibly empty) file. This means you must manually remove that file before attempting another download.

Finally, coach and broadcast video use the same naming convention. Therefore, you must keep use different directories to store them. Using the same directory with and without the --broadcast flag is never correct.

I've found that if I use the "--show-url" switch, and then use the URL directly with ffmpeg, I can download the footage without issue.

Those URLs are for broadcast video. There are no URLs for coach footage since they are rtmp streams. (I guess technically there are "rtmp URLs", but I've never been able to get them to reliably work with rtmpdump.)

I actually think this is documented in the output of nflvid-footage --help.

nflvid-footage --season 2012 --teams NE --weeks 5 -- /home/nflvid/full/

Are you sure you're specifying the right directory here? /home/nflvid looks a little weird. It suggests you have a user account named nflvid on your system. I could obviously be wrong, but it looked a little fishy and thought it'd be worth checking before diving deeper.

BurntSushi commented 11 years ago

alright, installed rtmpdump-ksv package and now broadcast games seem to be working.

To be clear, rtmpdump is not used at all when downloading broadcast games. Broadcast games are streamed via HTTP Live Streaming, which is downloaded using straight-up ffmpeg. Only the coach footage is an rtmp stream.

SB game came down brilliantly, using ffmpeg, going directly into an MKV

Really? Hmm. I'll have to investigate again. I only tried playing the URL via vlc.

not much was different in the URL compared with what nflvid gave me, curious whether it's the same basic changes for all the post-season games?

I haven't confirmed it, but I imagine they are. I'll check a few more postseason games, and if they're the same format as the SB, then I'll add that into the logic for generating broadcast URLs.

what about pre-season?

I have not been able to find preseason games freely available on Neulion. Truthfully, I suspect they are there. But I don't know how to get to them. (I've gotten as far as I have because I have a subscription to NFL Game Rewind, which doesn't include preseason games.)

manicNFL commented 11 years ago

All that make sense.

To be quite honest, I'm most interested in broadcast footage, as I'm unable to always watch the games live, nor do I get all the channels. That being said, I'm perfectly happy with being able to grab the broadcasts from your script. But I'm willing to keep testing, so that any bugs can get worked out of your script.

with the broadcasts being downloaded with ffmpeg already, I again recommend/request a switch to have ffmpeg output an MKV file.

to acheive this, the ffmpeg script would look like this:

ffmpeg -i NFLGAME-BROADCAST-URL.m3u8 -c copy NFLGAME.mkv

again, this does not transcode anything, it simply copies the video and audio streams into an MKV file.

As I mentioned before, there seems to be an issue with the 2013 games (error listed above).

Furthermore, it would be GREAT if you could add a switch for the condensed games as well. I was able to try a friends GamePass service, and came across the XML files containing the condensed games playlist urls.

Example 1:

<?xml version="1.0" encoding="UTF-8"?>
<channel version="3.5" currentTime="0" defaultStreamIndex="5">
    <streamDatas>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_4500.mp4" blockDuration="2000" bitrate="4608000" duration="2380581">
            <video width="1280" height="720" fps="29.970030" bitrate="4269056" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_3000.mp4" blockDuration="2000" bitrate="3072000" duration="2380581">
            <video width="1280" height="720" fps="29.970030" bitrate="2778112" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_2400.mp4" blockDuration="2000" bitrate="2457600" duration="2380581">
            <video width="960" height="540" fps="29.970030" bitrate="2187264" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="93184" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_1600.mp4" blockDuration="2000" bitrate="1638400" duration="2380581">
            <video width="960" height="540" fps="29.970030" bitrate="1419264" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="76800" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_1200.mp4" blockDuration="2000" bitrate="1228800" duration="2380581">
            <video width="640" height="360" fps="29.970030" bitrate="1103872" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_800.mp4" blockDuration="2000" bitrate="819200" duration="2380581">
            <video width="640" height="360" fps="29.970030" bitrate="727040" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_400.mp4" blockDuration="2000" bitrate="409600" duration="2380581">
            <video width="400" height="224" fps="29.970030" bitrate="344064" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds131.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds131.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
    </streamDatas>
</channel>

Example 2:

<?xml version="1.0" encoding="UTF-8"?>
<channel version="3.5" currentTime="0" defaultStreamIndex="5">
    <streamDatas>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_4500.mp4" blockDuration="2000" bitrate="4608000" duration="1899098">
            <video width="1280" height="720" fps="29.970030" bitrate="4320256" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_3000.mp4" blockDuration="2000" bitrate="3072000" duration="1899098">
            <video width="1280" height="720" fps="29.970030" bitrate="2823168" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_2400.mp4" blockDuration="2000" bitrate="2457600" duration="1899098">
            <video width="960" height="540" fps="29.970030" bitrate="2225152" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="93184" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_1600.mp4" blockDuration="2000" bitrate="1638400" duration="1899098">
            <video width="960" height="540" fps="29.970030" bitrate="1452032" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="76800" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_1200.mp4" blockDuration="2000" bitrate="1228800" duration="1899098">
            <video width="640" height="360" fps="29.970030" bitrate="1113088" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_800.mp4" blockDuration="2000" bitrate="819200" duration="1899098">
            <video width="640" height="360" fps="29.970030" bitrate="738304" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55844/2_55844_kc_jac_2013_h_snap2w_1_400.mp4" blockDuration="2000" bitrate="409600" duration="1899098">
            <video width="400" height="224" fps="29.970030" bitrate="345088" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds130.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds130.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
    </streamDatas>
</channel>

Example 3:

<?xml version="1.0" encoding="UTF-8"?>
<channel version="3.5" currentTime="0" defaultStreamIndex="5">
    <streamDatas>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_4500.mp4" blockDuration="2000" bitrate="4608000" duration="1847448">
            <video width="1280" height="720" fps="29.970030" bitrate="4552704" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_3000.mp4" blockDuration="2000" bitrate="3072000" duration="1847448">
            <video width="1280" height="720" fps="29.970030" bitrate="2968576" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="124928" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_2400.mp4" blockDuration="2000" bitrate="2457600" duration="1847448">
            <video width="960" height="540" fps="29.970030" bitrate="2328576" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="93184" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_1600.mp4" blockDuration="2000" bitrate="1638400" duration="1847448">
            <video width="960" height="540" fps="29.970030" bitrate="1526784" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="76800" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_1200.mp4" blockDuration="2000" bitrate="1228800" duration="1847448">
            <video width="640" height="360" fps="29.970030" bitrate="1162240" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_800.mp4" blockDuration="2000" bitrate="819200" duration="1847448">
            <video width="640" height="360" fps="29.970030" bitrate="773120" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
        <streamData url="/nlds_vod/nfl/vod/2013/09/08/55839/2_55839_sea_car_2013_h_snap2w_1_400.mp4" blockDuration="2000" bitrate="409600" duration="1847448">
            <video width="400" height="224" fps="29.970030" bitrate="359424" codec="avc1"/>
            <audio channelCount="2" samplesRate="44100" sampleBitSize="16" bitrate="56320" codec="mp4a"/>
            <httpservers>
                <httpserver name="nlds133.cdnak.neulion.com" port="80"/>
                <httpserver name="nlds133.cdnl3nl.neulion.com" port="80"/>
            </httpservers>
        </streamData>
    </streamDatas>
</channel>
BurntSushi commented 11 years ago

but I'm confused as to why the broadcast footage download only started working after installing rtmpdump-ksv

I don't know. It's likely to do some other change that we're not thinking of. But I can assure you that rtmpdump has nothing to do with fetching broadcast video. :-)

To be quite honest, I'm most interested in broadcast footage, as I'm unable to always watch the games live, nor do I get all the channels. That being said, I'm perfectly happy with being able to grab the broadcasts from your script. But I'm willing to keep testing, so that any bugs can get worked out of your script.

Thank you :-)

with the broadcasts being downloaded with ffmpeg already, I again recommend/request a switch to have ffmpeg output an MKV file.

I'm honestly not seeing the point at this juncture. You originally requested it because of a misdiagnosis. For whatever reason, the coach stream being downloaded with a busted rtmpdump program was corrupt. But using ksv's patch, this seems to go away and the mp4 container is working just fine. And I don't see any substantial difference in file size either:

[andrew@Liger full] ffmpeg -i full/2013090800.mp4 -c copy 2013090800.mkv
[andrew@Liger full] ls -l
total 6112556
-rw-r--r-- 1 andrew users 3127319162 Sep  9 01:01 2013090800.mkv
-rw-r--r-- 1 andrew users 3131926105 Sep  9 01:00 2013090800.mp4

I would rather just stick to whatever format is being used by Neulion.

As I mentioned before, there seems to be an issue with the 2013 games (error listed above).

Ah yeah, I missed that. Sorry. I fixed that in the master branch a few days ago but hadn't pushed a new release. I just pushed a new release that should include the change. An upgrade with pip should fix that. Tony Hoare's billion dollar mistake strikes again...

Furthermore, it would be GREAT if you could add a switch for the condensed games as well.

Ah that would be a cool addition. I'm not sure when I'll get around to it, though. :-(

manicNFL commented 11 years ago

alright, I'll just continue to going to MKVs on my own.

I've updated nflvid with pip and 2013 games work now.

Upon some testing on my own, adding support for the condensed games seems like it would be incredibly easy...

given the whole broadcast stream address of: http://nlds82.cdnak.neulion.com/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_whole_1_4500.mp4.m3u8 the condensed broadcast stream address is simply: http://nlds82.cdnak.neulion.com/nlds_vod/nfl/vod/2013/09/08/55838/2_55838_ne_buf_2013_h_snap2w_1_4500.mp4.m3u8

The only thing that seems to change in the URLs, is the word "whole", which changes to "snap2w"

I'm not really a talented programmer, but it seems like a simple thing to add? give it a new switch --condensed

Thanks again.

BurntSushi commented 11 years ago

I agree that it's probably easy. But there are a billion easy things to do on my plate. :-)

If you like, you could take a crack at it and submit a pull request. I'd expect a function added to nflvid/__init__.py called condensed_urls that closely resembles the structure of broadcast_urls. (There are multiple URLs because we have to guess which is correct. I don't know how to know which one ahead of time.) You may reuse the broadcast_url_status and first_valid_broadcast_url functions. Finally, the download_broadcast function should get an extra optional parameter condensed=False, that when True, calls condensed_urls instead of broadcast_urls.

manicNFL commented 11 years ago

Well, like I said, I'm not a talented programmer... and I don't have ANY experience with python...

I might take a stab at trying to figure it out though, but who knows.

Thanks.

manicNFL commented 11 years ago

I haven't made any progress on the programming... but I've made some progress in helping your script work better!

Firstly, regarding post-season games...

So currently, your script currently returns post-season games a URL like this: http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/02/03/55835/2_55835_bal_sf_2012_h_whole_1_4500.mp4.m3u8

The correct URL in this case is: http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/02/03/55835/3_55835_bal_sf_2012_h_whole_1_4500.mp4.m3u8

Here's some more examples (first line = nflvid URL, second line = correct URL):

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/01/20/55834/2_55834_bal_ne_2012_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/01/20/55834/3_55834_bal_ne_2012_h_whole_1_4500.mp4.m3u8

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/01/20/55833/2_55833_sf_atl_2012_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/01/20/55833/3_55833_sf_atl_2012_h_whole_1_4500.mp4.m3u8

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2012/01/13/55832/2_55832_hou_ne_2012_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/01/13/55832/3_55832_hou_ne_2012_h_whole_1_4500.mp4.m3u8

As you can probably work out, there are two subtle, but important differences. Firstly, the year. With the URL from your script, even though those games are part of the 2012 season, they took place in 2013. Therefore, your script needs to ensure the correct year of the game. For post-season games, it will always be the season year +1 Secondly, the leading number of the filename. With the URL from our script, the leading number is 2. I assume this is because that's what is used for the regular season games. However, for post-season games, that leading number needs to be 3

Making those two adjustments, will allow all post-season games to be processed by your script.

-obviously the number before the bitrate, 1 in the above examples, will vary, but your script currently takes that into account, by returning three URLs, with 1, 2, and 3 as the number before the bitrate.

Secondly, regarding pre-season games...

So currently, your script currently returns post-season games a URL like this: http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/09/56102/2_56102_ne_phi_2013_h_whole_1_4500.mp4.m3u8

The correct URL in this case is: http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/09/56102/1_56102_ne_phi_2013_h_whole_1_4500.mp4.m3u8

Here's some more examples (first line = nflvid URL, second line = correct URL):

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56095/2_56095_bal_tb_2013_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56095/1_56095_bal_tb_2013_h_whole_1_4500.mp4.m3u8

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56097/2_56097_was_ten_2013_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56097/1_56097_was_ten_2013_h_whole_1_4500.mp4.m3u8

http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56096/2_56096_stl_cle_2013_h_whole_1_4500.mp4.m3u8
http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/08/08/56096/1_56096_stl_cle_2013_h_whole_1_4500.mp4.m3u8

As you can probably work out, there is one subtle, but important difference. Much like the second adjustment for the post-season games, the problem is the leading number of the filename. With the URL from our script, the leading number is 2. I assume this is because that's what is used for the regular season games. However, for post-season games, that leading number needs to be 1

-obviously the number before the bitrate, 1 in the above examples, will vary, but your script currently takes that into account, by returning three URLs, with 1, 2, and 3 as the number before the bitrate.

And as for condensed pre/post-season games, it works out the just the same as regular season games.

The only thing that seems to change in the URLs, is the word "whole", which changes to "snap2w"

The Pro Bowl games are also available, example: http://nlds82.cdnl3nl.neulion.com/nlds_vod/nfl/vod/2013/01/27/55836/3_55836_afc_nfc_2012_h_whole_1_4500.mp4.m3u8

BurntSushi commented 11 years ago

Wow! Nice find! So the first digit is either 1, 2 or 3 depending on whether it's pre, reg or post. I didn't catch that.

I'll take all of this info and incorporate it into nflvid as soon as I can. Thanks so much!

(The season year being off was also a nice find. Whoops!)

manicNFL commented 11 years ago

Awesome!

Tested, and working!

Thanks for doing this!

BurntSushi commented 11 years ago

:+1: