iamkroot / ilc-scraper

A scraper for Impartus Lecture Capture videos for BITS Hyderabad
68 stars 30 forks source link

"TypeError" in get_angle_playlists #19

Closed dharmrajsharma16 closed 4 years ago

dharmrajsharma16 commented 4 years ago

Context

Describe the bug Encountered with "TypeError" exception while trying to download the lectures.

Screenshots If applicable, add screenshots to help explain your problem. image image image image

Program output

Click to see output

``` Saving to "C:\ImpartusLacture\Python Fundamentals for Data Science (S2-19_DSECLPFDS) 2019-20 Sem 2" Downloading lecture numbers: 4 5 6 7 8 Traceback (most recent call last): File "ilc_scrape.py", line 345, in File "ilc_scrape.py", line 335, in main File "multiprocessing\pool.py", line 276, in starmap File "multiprocessing\pool.py", line 657, in get File "multiprocessing\pool.py", line 121, in worker File "multiprocessing\pool.py", line 47, in starmapstar File "downloader.py", line 167, in download_stream File "downloader.py", line 104, in get_angle_playlists TypeError: list indices must be integers or slices, not NoneType Processing 04. Orientation-M.Tech In Dse Online Cluster 2020-03-28.mkv Processing 06. Python-2 2020-04-05.mkv Processing 08. Python Fundamentals For Data Science Part 4 2020-04-12.mkv list indices must be integers or slices, not NoneType ```

dharmrajsharma16 commented 4 years ago

Course Page image

iamkroot commented 4 years ago

As I said in #10, this seems to a Pilani specific issue, due to how cameras have been set up there. I don't have access to the lectures, and so I can't fix this. If someone else wants to do so, they're welcome to make a proper PR with a fix.

shaksham95 commented 4 years ago

@iamkroot Can you help me in figuring this out? I have been trying to poke around your code and the error which I mentioned in #10 is present below with more information:

image

As you can see, I am only passing the option to capture " left" camera angle and what is happening in get_angle_playlists is that a None value is getting returned when you execute the following line last_key = find_startswith(angle1, "#EXT-X-KEY", rev=True).

For your reference, in the output, you can see the values of the following variables: angle1_end, angle, and last_key.

Do let me know if you can figure out now what might be going wrong? Also, in the output, I don't see the download link for the video? Where can I find that?

sundrb commented 4 years ago

@iamkroot , @shaksham95 Solving this issue should help us lot. Please do consider what should be done to resolve the issue. Please let me know what support I can give you.

iamkroot commented 4 years ago

To debug, I'll need to see the m3u8 playlist file for the lecture. If you have setup using the manual method, open downloader.py and add

with open("playlist.m3u8", "w") as f:
    f.write(variant_pls)

after line 165, before the invocation of get_angle_playlists function. Then save it and run it again. This ought to store the file on your computer. Attach it here.

iamkroot commented 4 years ago

PS: @shaksham95 your password is visible in the screenshot.

sundrb commented 4 years ago

I think I understood and did correctly. Added the code u said. image But, No action on sceen image

shaksham95 commented 4 years ago

@iamkroot I have the file as you requested, but I can't upload here since there's a restriction on the type of file that can be uploaded.

The file has 6 hours of footage with the video running in chunks, it seems.

iamkroot commented 4 years ago

@shaksham95 upload it to pastebin then. I need to see the structure of the file.

shaksham95 commented 4 years ago

@iamkroot I am not sure how to upload a video file on pastebin. Any ideas?

iamkroot commented 4 years ago

What video file? I was talking about the m3u8 file that would've been saved. It's a text file, not video.

On Mon, Apr 27, 2020, 10:26 AM Shaksham Kapoor notifications@github.com wrote:

@iamkroot https://github.com/iamkroot I am not sure how to upload a video file on pastebin. Any ideas?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iamkroot/ilc-scraper/issues/19#issuecomment-619717785, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHBGJ6P3YCE3ZYEHWRGXLFTROUF7PANCNFSM4MH7N44Q .

shaksham95 commented 4 years ago

@iamkroot Okay, the code which you shared the output for it is coming to a video (m3u8) file. It's a file running videos in chunks. I suppose you are asking for the text input which I have attached here, let me know if this is what you need.

playlist.txt

sundrb commented 4 years ago

Thanks @iamkroot & @shaksham95 for working on it :)

iamkroot commented 4 years ago

Ok I had a look through that playlist file and it seems that these are very different from the pre-recorded streams we get. The video stream is unencrypted, and can be downloaded even without logging into impartus.

iamkroot commented 4 years ago

@shaksham95 I've pushed a possible fix in the pilani-live branch. Clone that and try to run the scraper again (using cmd line). See if its working (wait for the entire vid to be downloaded).

shaksham95 commented 4 years ago

Okay, I tried it out and now getting the following error;

image

iamkroot commented 4 years ago

@shaksham95 updated, try now

shaksham95 commented 4 years ago

@iamkroot Okay, I tried that out and the videos are now downloading, thanks a lot for your time.

However, the lectures are of 2-2.30 hrs but the video footage is usually of 5-6 hrs, the same video is repeated twice, any idea what might be causing that?

sundrb commented 4 years ago

Dear @iamkroot & @shaksham95, thanks a lot for your time and efforts...

@iamkroot, Can we try through release because, i am using office laptop and cannot define environment variables for ffmpeg for executing python.

iamkroot commented 4 years ago

any idea what might be causing that?

No clue, impartus shenanigans.

I can't add any more specific stuff, because it will have to be hardcoded and it will break the normal use case.

Can we try through release because

I was waiting for confirmation that it works. A new version will be released soon.

shaksham95 commented 4 years ago

@iamkroot Okay, no problem. Everything works fine; please release a new version at your earliest convenience. Thanks again for your help :)

sundrb commented 4 years ago

any idea what might be causing that?

No clue, impartus shenanigans.

I can't add any more specific stuff, because it will have to be hardcoded and it will break the normal use case.

Can we try through release because

I was waiting for confirmation that it works. A new version will be released soon.

Any idea when it will be released? because otherwise Ill try to raise a ticket with my IT team as it require elevated permission to make changes to environment variables.

sundrb commented 4 years ago

New Release working :)

Also, it will be great if u can support us in downloading all quiz questions in one page. That is much required.