[YouTube] Slow initial loading of videos

[goyalyashpal]

Checklist

• [X] I am able to reproduce the bug with the latest version.
• [X] I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• [X] I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• [X] This issue contains only one bug.
• [X] I have read and understood the contribution guidelines.

Affected version

0.23.0

Steps to reproduce the bug

1. Open Newpipe for the first time in the session that is, say, after a very long time or like after it is frozen by the system or slept or whatever
2. Search for any youtube video
3. Click on any video to open its details page and wait for details to load (note the timings if you want to)
4. Click play button on the video viewerport
5. Swipe down and close the player
6. Redo step 3 and 4 with a different video (to remove "cache" interfering with readings)

Expected behavior

No response

Actual behavior

• The time taken for the first run is huge ( 19 + 4 = ) 23 ± 2s
• Compare that to 5s total for the subsequent run which is comparable to what it takes when youtube is run on the mobile browser.

Screenshots/Screen recordings

No response

Logs

No response

Affected Android/Custom ROM version

Android 11 / realme UI v2.0

Affected device model

No response

Additional information

No response

[goyalyashpal]

Closed issue : https://github.com/TeamNewPipe/NewPipe/issues/7630

I am opening it as new issue, as the existing issue was for version pre v0.23, and was closed as "(will be) fixed in this version".

[tsiflimagas]

This should be fixed in 0.23.1.

[goyalyashpal]

yes, it seems to be. but i would like to wait and test for a week more before closing this as fixed.

as initially v0.23.0 seemed to be fixed too, but after some days, the slowness started to manifest

[goyalyashpal]

no, v0.23.1 is also as slow as earlier. i can provide a screenrecord if that would help

On Saturday, July 9, 2022, Yash @.***> wrote:

yes, it seems to be. but i would like to wait and test for a week more before closing this as fixed.

as initially v0.23.0 seemed to be fixed too, but after some days, the slowness started to manifest

[opusforlife2]

@yashpalgoyal1304 Yes please.

[goyalyashpal]

time took	newpipe	youtube
to load the video details	11 s (0:11 to 0:22)	?? (0:45 to --)
to play the video	2 s (0:24 to 0:26)	?? (-- to 0:47)
total:	13 s	2s

• Newpipe version: 0.23.1
• Browser used: Vivaldi 5.3.2683.43

https://user-images.githubusercontent.com/19423063/178547546-06179e93-6f6c-41b1-be8b-6bf1e65c0e23.mp4

Positive Control experiment with player load time in newpipe and youtube

Comments:

• i couldnt make the video like those side by side comparison videos with timers shown in inset, but still, the differences are verrry easily noticable even without exact measure
• the recording is continuous, no pause or break in between
• the issue i filled is exactly captured in this video, that's, recording is not a one time occurence – it always happens like this

Record environment vars:

Variable to change:	the method of playing (newpipe vs youtube)
Variable to measure:	time took in 1. opening the player ui 2. playing the video (after clicking play or after player is loaded)
Variables to keep same:	cache * the sample yt video was played for first time on both
	internet connection the connection was not changed the duration of experiment was kept very small
	sample youtube video * same youtube video was used
	procedure performed & means to measure it exact same steps were performed in both the applications both were unlogged instances * time in consideration is time between clicking it, to the point it just starts to play
Variables not controlled/measured:	default video quality youtube web doesnt provide a way to set the video quality i forgot to record what quality it played * however, my newpipe setting is default resolution: 240p @ default format: MPEG-4
	exact internet speed * the speed on the status bar updates fairly slowly , so, is NOT realtime however, it can be used to have pretty good spaced estimates of the speed

[goyalyashpal]

self hiding as resolved

i am trying again and again, and am unable to upload the video, meanwhile, have pasted the input... will keep on trying to upload the video

> Try again with a * GIF, JPEG, JPG, MOV, MP4, PNG, SVG, WEBM, CSV, DOCX, FODG, FODP, FODS, FODT, GZ, LOG, MD, ODF, ODG, ODP, ODS, ODT, PDF, PPTX, TGZ, TXT, XLS, XLSX or ZIP. * of size lesser than 10 MiB aaaah, that sucks

[AudricV]

7630 reported slow initial playback, the issue you reported is completely different (slow initial loading). Let me explain it.

To load a YouTube video, we have to do several requests:

• if the latest client version and API key of YouTube's desktop website client of the InnerTube internal API is not known, we need to get it:
  • we fetch the service worker JavaScript file which contained these information before but that's not the case anymore;
  • as a fallback, we fetch the HTML page of results of a YouTube search (between 300 and 400KB, transfer size between 30 and 40KB).
• we need to get the full YouTube JavaScript base player file, to perform decryption of the n parameter of videoplayback URLs and deobfuscate signatureCiphers from YouTube's desktop website. To do so, we fetch YouTube's iFrame API (pretty tiny) to get the player hash but there is an issue which seems to cancel this call and fetch the fallback (a video embed HTML page, which is not really big, but not tiny too. Then we load the JavaScript player which is huge (around 2MB, which is not the transfer size at all (around 600KB));
• once the JavaScript player has been loaded, we request the player endpoint of the InnerTube internal API of the desktop website client;
• if the content is considered as unplayable (including age-restricted):
  • we fetch the player endpoint of the InnerTube internal API of the client used to bypass age-restrictions;
  • if the content is still unplayable, no more requests are made and an exception is thrown.
• if the content is a live content not age-restricted:
  • we fetch the player endpoint of the InnerTube internal API of the iOS client;
• if the content is a video not age-restricted:
  • we fetch the player endpoint of the InnerTube internal API of the Android client;
• finally we fetch the next endpoint of the InnerTube internal API of the desktop website client.

All the InnerTube requests are POST requests sending a small JSON body which get JSON responses from YouTube servers (pretty small as most of transfer sizes are between 15 and 50KB, even if I didn't checked this completely).

Note that we fetch data from different clients in order to get different formats and resolutions as much as possible in a reasonable way, and also to avoid throttling on streaming URLs of the desktop website client (you heard of the n parameter issues, if its decryption in streaming URLs by the extractor fails, downloads (and so playback) are throttled (this would also happen with the client used to bypass age-restrictions, because this parameter is also present in streaming URLs of it, like signatureCiphers)).

This behavior, of course, does not happen on YouTube's websites, as the JavaScript player is fetched when you load the website and only two requests to the InnerTube internal API, using the website's client, are made to load a video: one for the player endpoint and one for the next endpoint.

The player endpoint of the InnerTube internal API of YouTube is used mainly by us to get streaming data and some metadata and the next endpoint mainly to get metadata such as related items or likes.

In these requests, we (us and libraries we use to parse HTML and JSON responses for instance) have to parse responses. When they are big, it can take several time to process parsing operations and do the necessary calculations.

This behavior is also depend of the device used and the current network conditions: when you have a good connection and a good device, you will get better results than on a "potato" one with a bad connection (and even with a good one), even if the difference should be not really big when network conditions are the same or close to each other.

I'm afraid that for the reasons I explained above, YouTube will be always faster to load a video than us.

Closing as not feasible/won't fix.

[goyalyashpal]

@audricv okay, thanks for the detailed breakdown. one question: why does the subsequent videos play faster in newpipe too? i mean what steps are not performed in them?

[AudricV]

@yashpalgoyal1304 Getting client version and API key of the WEB client of the InnerTube API + getting YouTube JavaScript base player (these contents are cached in memory).

[goyalyashpal]

so, can't it be that those things are automatically fetched with some random video in the background - when say the person is doing some other work in newpipe - like entering stuff on search bar, and browsing through search results etc....

obviously this cant be done all the time, but say, when the newpipe is in youtube mode, or like some logic ...

umh, i dont know 😅 , just was thinking if something can make this work.... sorry

[goyalyashpal]

can it be that these videos are stored in the newpipe's internal queue - so that at least user can continue enquing effortlessly without having to wait for the player and stuff to load?

[SameenAhnaf]

@goyalyashpal Already requested. https://github.com/TeamNewPipe/NewPipe/issues/1928

[goyalyashpal]

Already requested. #1928

umh, that seems to be different. i am saying for storing the "current playing queue" while the "player" loads - that issue says to store the "actual video/audio data" of next stream while current stream is already playing.