"TypeError" in get_angle_playlists

[dharmrajsharma16]

Context

• OS: Windows 10
• Install method: EXE
• Your location: Outside Campus
• Course URL: https://a.impartus.com/ilc/#/course/160200/706
• Lecture range: 4:9 Additional context NA

Describe the bug Encountered with "TypeError" exception while trying to download the lectures.

Screenshots If applicable, add screenshots to help explain your problem.

Program output

Click to see output

``` Saving to "C:\ImpartusLacture\Python Fundamentals for Data Science (S2-19_DSECLPFDS) 2019-20 Sem 2" Downloading lecture numbers: 4 5 6 7 8 Traceback (most recent call last): File "ilc_scrape.py", line 345, in File "ilc_scrape.py", line 335, in main File "multiprocessing\pool.py", line 276, in starmap File "multiprocessing\pool.py", line 657, in get File "multiprocessing\pool.py", line 121, in worker File "multiprocessing\pool.py", line 47, in starmapstar File "downloader.py", line 167, in download_stream File "downloader.py", line 104, in get_angle_playlists TypeError: list indices must be integers or slices, not NoneType Processing 04. Orientation-M.Tech In Dse Online Cluster 2020-03-28.mkv Processing 06. Python-2 2020-04-05.mkv Processing 08. Python Fundamentals For Data Science Part 4 2020-04-12.mkv list indices must be integers or slices, not NoneType ```

[dharmrajsharma16]

Course Page

[iamkroot]

As I said in #10, this seems to a Pilani specific issue, due to how cameras have been set up there. I don't have access to the lectures, and so I can't fix this. If someone else wants to do so, they're welcome to make a proper PR with a fix.

[shaksham95]

@iamkroot Can you help me in figuring this out? I have been trying to poke around your code and the error which I mentioned in #10 is present below with more information:

As you can see, I am only passing the option to capture " left" camera angle and what is happening in get_angle_playlists is that a None value is getting returned when you execute the following line last_key = find_startswith(angle1, "#EXT-X-KEY", rev=True).

For your reference, in the output, you can see the values of the following variables: angle1_end, angle, and last_key.

Do let me know if you can figure out now what might be going wrong? Also, in the output, I don't see the download link for the video? Where can I find that?

[sundrb]

@iamkroot , @shaksham95 Solving this issue should help us lot. Please do consider what should be done to resolve the issue. Please let me know what support I can give you.

[iamkroot]

To debug, I'll need to see the m3u8 playlist file for the lecture. If you have setup using the manual method, open downloader.py and add

with open("playlist.m3u8", "w") as f:
    f.write(variant_pls)

after line 165, before the invocation of get_angle_playlists function. Then save it and run it again. This ought to store the file on your computer. Attach it here.

[iamkroot]

PS: @shaksham95 your password is visible in the screenshot.

[sundrb]

I think I understood and did correctly. Added the code u said. But, No action on sceen

[shaksham95]

@iamkroot I have the file as you requested, but I can't upload here since there's a restriction on the type of file that can be uploaded.

The file has 6 hours of footage with the video running in chunks, it seems.

[iamkroot]

@shaksham95 upload it to pastebin then. I need to see the structure of the file.

[shaksham95]

@iamkroot I am not sure how to upload a video file on pastebin. Any ideas?

[iamkroot]

What video file? I was talking about the m3u8 file that would've been saved. It's a text file, not video.

On Mon, Apr 27, 2020, 10:26 AM Shaksham Kapoor notifications@github.com wrote:

@iamkroot https://github.com/iamkroot I am not sure how to upload a video file on pastebin. Any ideas?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iamkroot/ilc-scraper/issues/19#issuecomment-619717785, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHBGJ6P3YCE3ZYEHWRGXLFTROUF7PANCNFSM4MH7N44Q .

[shaksham95]

@iamkroot Okay, the code which you shared the output for it is coming to a video (m3u8) file. It's a file running videos in chunks. I suppose you are asking for the text input which I have attached here, let me know if this is what you need.

playlist.txt

[sundrb]

Thanks @iamkroot & @shaksham95 for working on it :)

[iamkroot]

Ok I had a look through that playlist file and it seems that these are very different from the pre-recorded streams we get. The video stream is unencrypted, and can be downloaded even without logging into impartus.

[iamkroot]

@shaksham95 I've pushed a possible fix in the pilani-live branch. Clone that and try to run the scraper again (using cmd line). See if its working (wait for the entire vid to be downloaded).

[shaksham95]

Okay, I tried it out and now getting the following error;

[iamkroot]

@shaksham95 updated, try now

[shaksham95]

@iamkroot Okay, I tried that out and the videos are now downloading, thanks a lot for your time.

However, the lectures are of 2-2.30 hrs but the video footage is usually of 5-6 hrs, the same video is repeated twice, any idea what might be causing that?

[sundrb]

Dear @iamkroot & @shaksham95, thanks a lot for your time and efforts...

@iamkroot, Can we try through release because, i am using office laptop and cannot define environment variables for ffmpeg for executing python.

[iamkroot]

any idea what might be causing that?

No clue, impartus shenanigans.

I can't add any more specific stuff, because it will have to be hardcoded and it will break the normal use case.

Can we try through release because

I was waiting for confirmation that it works. A new version will be released soon.

[shaksham95]

@iamkroot Okay, no problem. Everything works fine; please release a new version at your earliest convenience. Thanks again for your help :)

[sundrb]

any idea what might be causing that?

No clue, impartus shenanigans.

I can't add any more specific stuff, because it will have to be hardcoded and it will break the normal use case.

Can we try through release because

I was waiting for confirmation that it works. A new version will be released soon.

Any idea when it will be released? because otherwise Ill try to raise a ticket with my IT team as it require elevated permission to make changes to environment variables.

[sundrb]

New Release working :)

Also, it will be great if u can support us in downloading all quiz questions in one page. That is much required.