search

soraxas / echo360

Commandline tool for automated downloads of echo360 videos hosted by university
https://cs.tinyiu.com/echo360
MIT License
261 stars 51 forks source link

Failed to find audio/video m3u8 #37

Closed bwhurd closed 3 years ago

bwhurd commented 3 years ago

Hi, I haven't had any issues until I recently started a new class and I'm getting the following error "ERROR: Failed to find audio/video m3u8... skipping this one"

Any thoughts on how to fix this? Thank you!

Here is the debug result: https://pastebin.com/q2i04Ekm

soraxas commented 3 years ago

Hi, your pastebin is private.

On 5 May 2021 08:05:32 bwhurd @.***> wrote:

Hi, I haven't had any issues until I recently started a new class and I'm getting the following error "ERROR: Failed to find audio/video m3u8... skipping this one"

Any thoughts on how to fix this? Thank you!

Here is the debug result: https://pastebin.com/q2i04Ekm

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

bwhurd commented 3 years ago

Sorry. Should be public now.

soraxas commented 3 years ago

Does this link lead to the video content?

https://content.echo360.org/0000.18ff41c7-5a40-4929-bb90-3b3ee5e64ffb/633b2de6-6841-4135-a2c5-b525ea7c0992/1/s1_av.m3u8
bwhurd commented 3 years ago

I don't think so. If I load that url, chrome downloads the m3u8 file if I am logged in to echo360. In a text editor, the m3u8 file looks like this:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="q0",NAME="Default",DEFAULT=YES,AUTOSELECT=YES,URI="s0q0.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="q1",NAME="Default",DEFAULT=YES,AUTOSELECT=YES,URI="s0q1.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=55528,RESOLUTION=640x360,PROGRAM-ID=1,AUDIO="q0",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0
s1q0.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=220997,RESOLUTION=1920x1080,PROGRAM-ID=1,AUDIO="q1",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0
s1q1.m3u8

If I use the --interactive flag and try to download just one lecture, same result as the debug:

continue
>> I'm gonna assume you are responsible enough to had finished logged in by now ;)
>> Retrieving echo360 Course Info...  1/1 videos Done!
============================================================
    Course: [[UNTITLED]]
      Total videos to download: 1 out of 1
============================================================

------------------------------------------------------------
Downloading "2021-05-06 - Lecture 1.3"
ERROR: Failed to find audio/video m3u8... skipping this one

=================================================================
    Course: [[UNTITLED]]
=================================================================
    Successfully downloaded:
=================================================================

C:\echo>
bwhurd commented 3 years ago

So, the one thing that is different with the courses that are not working versus the course that is working is the non-working courses have a subfolder as shown in this screenshot: https://i.imgur.com/IGENonx.jpg Does that help any?

getBoolean commented 3 years ago

Hi, I am having the same issue. It gives the error for any video uploaded after April 20th/21st.

soraxas commented 3 years ago

Hi @bwhurd and @getBoolean I've pushed a new commit attempting to fix it based purely on the logs. Can you see if it did the trick?

getBoolean commented 3 years ago

Hi @bwhurd and @getBoolean I've pushed a new commit attempting to fix it based purely on the logs. Can you see if it did the trick?

I would test it but I can't log in to echo360 anymore. It just says "Identity provider was not found". I think this is on echo360/ my school's side.

drdm commented 3 years ago

Hi @bwhurd and @getBoolean I've pushed a new commit attempting to fix it based purely on the logs. Can you see if it did the trick?

This definitely improved matters for me - it got as far as downloading a video, for the first time, but then errored out converting the video:

DEBUG: Found the following urls [u'https://content.echo360.org.au/0000.00c445c8-03e2-41c4-b11c-c2fa22eea211/a9978b47-7f32-4dcb-91b2-8c23e26b2a81/1/s1_av.m3u8']
>> Retrieving echo360 Course Info...  6/6 videos Done!
============================================================
    Course: [[UNTITLED]]
      Total videos to download: 6 out of 6
============================================================

------------------------------------------------------------
Downloading "2021-05-12 - Lecture 6 [Summary and review includng review of homework exercises]"
DEBUG: GET http://127.0.0.1:60555/session/985aee85d987ecf9441b5e3e2c07147e/cookie {"sessionId": "985aee85d987ecf9441b5e3e2c07147e"}
DEBUG: http://127.0.0.1:60555 "GET /session/985aee85d987ecf9441b5e3e2c07147e/cookie HTTP/1.1" 200 2357
DEBUG: Finished Request
- Downloading video feed 1...
DEBUG: Starting new HTTPS connection (1): content.echo360.org.au:443
DEBUG: https://content.echo360.org.au:443 "GET /0000.00c445c8-03e2-41c4-b11c-c2fa22eea211/a9978b47-7f32-4dcb-91b2-8c23e26b2a81/1/s1_av.m3u8 HTTP/1.1" 200 518
DEBUG: Searching for m3u8 with content [u'#EXTM3U', u'#EXT-X-VERSION:7', u'#EXT-X-INDEPENDENT-SEGMENTS', u'', u'#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="q0",NAME="Default",DEFAULT=YES,AUTOSELECT=YES,URI="s0q0.m3u8"', u'#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="q1",NAME="Default",DEFAULT=YES,AUTOSELECT=YES,URI="s0q1.m3u8"', u'', u'#EXT-X-STREAM-INF:BANDWIDTH=55000,RESOLUTION=640x360,PROGRAM-ID=1,AUDIO="q0",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0', u's1q0.m3u8', u'#EXT-X-STREAM-INF:BANDWIDTH=220055,RESOLUTION=1920x1080,PROGRAM-ID=1,AUDIO="q1",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0', u's1q1.m3u8', u'']
DEBUG: Found video with line s1q0.m3u8 with prev line #EXT-X-STREAM-INF:BANDWIDTH=55000,RESOLUTION=640x360,PROGRAM-ID=1,AUDIO="q0",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0
DEBUG: Found video with line s1q1.m3u8 with prev line #EXT-X-STREAM-INF:BANDWIDTH=220055,RESOLUTION=1920x1080,PROGRAM-ID=1,AUDIO="q1",CODECS="avc1.640029,mp4a.40.2",FRAME-RATE=25.0
  > Downloading video:
DEBUG: GET http://127.0.0.1:60555/session/985aee85d987ecf9441b5e3e2c07147e/cookie {"sessionId": "985aee85d987ecf9441b5e3e2c07147e"}
DEBUG: http://127.0.0.1:60555 "GET /session/985aee85d987ecf9441b5e3e2c07147e/cookie HTTP/1.1" 200 2357
DEBUG: Finished Request
DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG: Starting new HTTPS connection (1): content.echo360.org.au:443
DEBUG: https://content.echo360.org.au:443 "GET /0000.00c445c8-03e2-41c4-b11c-c2fa22eea211/a9978b47-7f32-4dcb-91b2-8c23e26b2a81/1/s1q1.m3u8 HTTP/1.1" 200 None
  > Progress: [>                   ] 0.00%  0/1DEBUG: https://content.echo360.org.au:443 "GET /0000.00c445c8-03e2-41c4-b11c-c2fa22eea211/a9978b47-7f32-4dcb-91b2-8c23e26b2a81/1/s1q1.m4s HTTP/1.1" 200 172744399
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 173M/173M [00:34<00:00, 4.98MiB/s]
  > Converting to mp4... Traceback (most recent call last):
  File "echo360.py", line 7, in <module>
    sys.exit(main())
  File "/Users/damian/Downloads/echo360-master/echo360/main.py", line 356, in main
    downloader.download_all()
  File "/Users/damian/Downloads/echo360-master/echo360/downloader.py", line 235, in download_all
    if video.download(self._output_dir, filename):
  File "/Users/damian/Downloads/echo360-master/echo360/videos.py", line 268, in download
    result = self.download_single(session, single_url, output_dir, new_filename, pool_size)
  File "/Users/damian/Downloads/echo360-master/echo360/videos.py", line 331, in download_single
    os.rename(video_file, final_file)
NameError: global name 'final_file' is not defined

I should add that I simply downloaded the current master and ran it unmodified. This is my command line:

python echo360.py https://echo360.org.au/section/1d351654-a234-4dba-9bb7-31b606f2d4af/home --output Lectures --alternative_feeds --debug

I've tried with and without alternative feeds, and also using the run.sh script instead of python echo360.py. All with the same error at the end.

bwhurd commented 3 years ago

it downloaded only the first video, then failed to convert it and terminated. Here is the debug. Really appreciate it!

EDIT: actually, this debug is cleaner. the first one had some files leftover in the output folder causing conflicts.

soraxas commented 3 years ago

Thanks @drdm and @bwhurd.

Give the latest commit another go? Also let me know if the audio/video track of the resulting file seems fine.

drdm commented 3 years ago

Huge step forward - with the latest commit, successfully downloaded and converted all 6 videos to playable mp4 videos. However, none of them have any audio. I've tried with and without --alternative_feeds.

Debug log is here: https://gist.github.com/drdm/eedd12dc05fff4121eaf8cc116f4f490

BTW, the echo360 utility is awesome, and your time in updating it for whatever change in Echo360 broke it is much appreciated. Thank you!

soraxas commented 3 years ago

Thanks @drdm for your kind words. I actually don't have the needs to use this tools for like 3 years now (nor the access to), so mostly I've just been coping to fix any breaking upstream updates.

I think the posted debug log are not full? I need the log of the downloading process (as it seems like it fails to discover audio stream). If you are concern with privacy issue feel free to email me the log instead.

drdm commented 3 years ago

@soraxas is this the log you need? https://gist.github.com/drdm/6ed07b91055925c5f86e059d0c0939cd

soraxas commented 3 years ago

Feel free to test the latest push.

drdm commented 3 years ago

@soraxas you're a genius :)

That's pretty quick going for a new (modified) parser. Works perfectly.

Thanks!

soraxas commented 3 years ago

Great that it works!

bwhurd commented 3 years ago

Hi there soraxas-- so, now the downloader works if I download all of the videos, so thank you!

However, the --interactive flag broke with the latest commits. Yesterday, when the videos would not download, the --interactive flag was working even though the videos would not download. (meaning, the list of lectures would appear, and I could select them)

Here's the debug for the latest commit when it terminates after parsing with the interactive flag before downloading begins.

bwhurd commented 3 years ago

Here is a debug of a successful download of all of the files using the latest commit if that helps.

soraxas commented 3 years ago

Hi @bwhurd I cannot reproduce what you had mentioned as the --interactive flag was not affected from my end. What do you mean by it's broke? I also couldn't find any anything wrong with the log. Did it hangs (the log stopped somewhere there)? Was it because of your netowrk issue?

The commit shouldn't have touched anything that is related to the interactive flag functionality.

bwhurd commented 3 years ago

I think I was wrong and --interactive is working for me. Issue closed. Thank you!