inconsistent bad page exceptions

[madmagic007]

parsing videos/playlists is inconsistent. one time it says a couple times in a row trying to parse a playlist, that it "Could not parse web page", and sometimes it works fine (just running the code again, changing nothing). sometimes when it managed to parse the playlist, it errors on one of the videos, and sometimes it doesnt.

failed to parse video (id was correct, i double check it every time): com.github.kiulian.downloader.YoutubeException$BadPageException: Could not parse web page at com.github.kiulian.downloader.extractor.DefaultExtractor.extractYtPlayerConfig(DefaultExtractor.java:78) at com.github.kiulian.downloader.parser.DefaultParser.getPlayerConfig(DefaultParser.java:79) at com.github.kiulian.downloader.YoutubeDownloader.getVideo(YoutubeDownloader.java:69)

failed parsing playlist (id is correct aswell): com.github.kiulian.downloader.YoutubeException$BadPageException: Could not parse web page at com.github.kiulian.downloader.extractor.DefaultExtractor.extractYtInitialData(DefaultExtractor.java:89) at com.github.kiulian.downloader.parser.DefaultParser.getInitialData(DefaultParser.java:224) at com.github.kiulian.downloader.YoutubeDownloader.getPlaylist(YoutubeDownloader.java:86)

[sealedtx]

@madmagic007 if code works fine sometimes for same video, it probably means that the problem with youtube is present, it may return you bad page. Please, provide videoId so I can check if it is bug in this lib or youtube problem

[Shabinder]

Could not parse web page, I am also Encountering this error on various Videos Like: for example on this Video ID: 4UIHLepr0Uc or https://www.youtube.com/watch?v=4UIHLepr0Uc NOTE After Upgrading to 2.4.3 , this Issue is fixed for me.

[madmagic007]

https://www.youtube.com/playlist?list=PL7MxrNF8eKO5RkZw9_LktHETFhqgytrhf this is the playlist i was having it with. i havent had time to test if its on other playlists yet. its an unlisted playlist, idk if it has to do with it. The problem happened sometimes on parsing the playlist itself, and sometimes on any video that was in it. however most of the time it was the first video.

edit: i tried with other playlists and using 2.4.3, this still happens, but less often. However this video: https://youtube.com/watch?v=EWJR2KKBk2g seems to fail every single time for me.

[kangsLee]

@madmagic007 I tested video EWJR2KKBk2g you provided, and it was always successful. Check again if the version of the project is 2.4.3.

[Dolanpls]

With the version 2.4.3 all its working perfect for me. Its true that in the last version I had some random exceptions with any video, but since 2.4.3 all works great 👍

[madmagic007]

Double checked that i am indeed using 2.4.3, but EWJR2KKBk2g still fails evey time for me. Is there a posibility that it has to do with bad/slow ineternet? because mine is really bad. I can test with better internet in a couple days.

[sealedtx]

@madmagic007 yes, please, test with another internet connection, so we know the reason of these fails. Thank you

[Dolanpls]

@madmagic007 srry you were right. Your video is a +18 video and the method 'getVideo(videoId)' return BadPageException -> "Could not parse web page"

• But for public and non +18 videos the method 'getVideo(videoId)' works perfect

UPDATE I dont know if the video is +18, but the problem is that require login 'Login to confirm your age. This video may be inappropriate for some users.'

The video

[sealedtx]

Probably connected with #40

[Shabinder]

Yeah , Upon Usage I am Facing This Same Issue again and It appears Inconsistently , Sometimes It does and Sometimes Doesn't....

for Instance Fetching this Playlist Causes this Issue, while on Multiple retries it do loads Correctly sometimes:

2020-11-08 12:41:38.965 20348-20348/com.shabinder.spotiflyer I/Intent Received: https://www.youtube.com/playlist?list=PL9bw4S5ePsEHwKpj0RkGOr2Is2uxvP0OD 2020-11-08 12:41:39.150 20348-20540/com.shabinder.spotiflyer I/YT Playlist ID : PL9bw4S5ePsEHwKpj0RkGOr2Is2uxvP0OD 2020-11-08 12:41:40.134 20348-20540/com.shabinder.spotiflyer E/AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1Process: com.shabinder.spotiflyer, PID: 20348com.github.kiulian.downloader.YoutubeException$BadPageException: Could not parse web pageat com.github.kiulian.downloader.extractor.DefaultExtractor.extractYtInitialData(DefaultExtractor.java:71)at com.github.kiulian.downloader.parser.DefaultParser.getInitialData(DefaultParser.java:210)at com.github.kiulian.downloader.YoutubeDownloader.getPlaylist(YoutubeDownloader.java:66)at com.shabinder.spotiflyer.ui.youtube.YoutubeViewModel$getYTPlaylist$1.invokeSuspend(YoutubeViewModel.kt:63)at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

[sealedtx]

@Shabinder your problem is not connected with author's. There was an issue with extracting YtInitialData for playlists, Youtube from time to time sends pages with different structure of content. But I just fixed it, check latest version 2.4.4

[madmagic007]

just tested it on better internet and using 2.4.4 and after 20+ tries, it works evey single time. looks like having slow(er) internet caused the occasional badpageexceptions. is there any possible way to support those "18+" videos or does the fact that you have to log in, makes this impossible?

[sealedtx]

@madmagic007 It is possible, but not implemented, look at #40

[Shabinder]

@Shabinder your problem is not connected with author's. There was an issue with extracting YtInitialData for playlists, Youtube from time to time sends pages with different structure of content. But I just fixed it, check latest version 2.4.4

Great , will update and check it out... Thanks For Active Development!

[Shabinder]

@sealedtx this issue still occurs but few times. I will share more info when i get some free time.