Error unexpected EOF and it stop audio record immediately but stream is not end

[Nekofoxmiu]

2023/03/17 04:36:51 INFO: Loaded cookie file mypath
2023/03/17 04:36:52 Channel: なつめえりちゃんねる / Natsume Eri ch.
2023/03/17 04:36:52 Video Title: 【雑談 】おさぎょうしよ！【なつめえり】
2023/03/17 04:36:52 DEBUG: Retrieving URLs from web DASH manifest
2023/03/17 04:36:54 TRACE: Setting itag 136 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 137 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 298 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 299 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 139 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 140 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 133 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 135 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 134 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 160 from web adaptive formats
2023/03/17 04:36:54 DEBUG: Retrieving URLs from web adaptive formats
2023/03/17 04:36:54 Selected quality: 1080p60 (h264)  https://www.youtube.com/watch?v=-qE4e0H4kss
2023/03/17 04:36:54 INFO: Starting download to H:\bot\youtube video\-qE4e0H4kss__1567626896\【雑談 】おさぎょうしよ！【なつめえり】--qE4e0H4kss.f140.ts
2023/03/17 04:36:54 INFO: Starting download to H:\bot\youtube video\-qE4e0H4kss__1567626896\【雑談 】おさぎょうしよ！【なつめえり】--qE4e0H4kss.f299.ts
2023/03/17 05:33:21 DEBUG: audio1: Error with fragment 21706: unexpected EOF
2023/03/17 05:36:52 ERROR: Error retrieving player response: unable to retrieve player response object from watch page
2023/03/17 05:36:53 DEBUG: audio1: Stream is finished and highest sequence reached
2023/03/17 05:36:53 DEBUG: audio1: exiting
2023/03/17 05:36:53 DEBUG: audio-download thread closing
2023/03/17 07:19:52 DEBUG: video1: Stream is finished and highest sequence reached
2023/03/17 07:19:52 DEBUG: video1: exiting
2023/03/17 07:19:52 DEBUG: video-download thread closing
2023/03/17 07:19:52 Download Finished
2023/03/17 07:19:52 WARNING: Mismatched number of video and audio fragments.
2023/03/17 07:19:52 WARNING: The files should still be mergable but data might be missing.

It often happen after 1 hour record. Also I am not sure it will affect or not, I am running ytarchive in node.js childprocess. Here is my code.

 let ytdl = child_process.exec(`ytarchive.exe --trace --merge --add-metadata --no-frag-files --thumbnail --retry-frags 100 --cookies "${rootFloder}\\setting\\yt_cookies.txt" -w -o "${rootFloder}\\youtube video\\%(title)s-%(id)s" ${video_ID} best `, {
        cwd: `${rootFloder}\\exe_tool`,
        maxBuffer: 1024 * 1024 * 1024
    })

[Kethsar]

Hmm, I should add an option to dump the HTML when it cannot find the data we're looking for. The EOF is not actually part of the issue, that's benign. It's the automatic 1-hour refresh encountering an issue. This causes yta to assume the stream is unavailable and just stop there. I assume video stops before the stream has actually ended as well, but takes longer to catch up since you're only running one thread.

The main thing I can think of that would cause this is it serving you some kind of verification page on that refresh attempt.

[Nekofoxmiu]

Hmmmm...OK...so should I closed this issue or change the title?

[Kethsar]

You can leave it as is for now. I might add the option later to dump the HTML on failing to extract a player response from it, and then upload a binary here for you to use. That way if it's something fixable I'll have something to go on, but if it's some kind of bot-check then we'll know you're out of luck for a while. I have an idea of something I can do instead of hard-stopping on failure too though.

[Narratur]

After some admittedly unscientific testing, I think this has to do with using a cookies file. I’ve been running into this near 100% of the time around 1 hour in to streams recently, so I was messing around with it and tried several things including changing my IP to no effect, before noticing that dropping --cookies seemed to stop the issue. So I recorded two more streams with the exact same options except one loading the cookies file:

ytarchive 0.3.2
2023/03/21 22:33:12 INFO: Loaded cookie file E:\YouTube\DL\cookies.txt
2023/03/21 22:33:13 Channel: Subaru Ch. 大空スバル
2023/03/21 22:33:13 Video Title: 【＃生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ /  biohazard　HD REMASTER【ホロライブ/大空スバル】

2023/03/21 22:33:13 Stream starts at 2023-03-22T12:00:00+00:00 in 30407 seconds.
2023/03/21 22:33:13 Waiting for this time to elapse...
2023/03/22 06:59:59 DEBUG: Woke up 1 seconds early. Continuing sleep...
Stream is 15 seconds late...
Stream is 30 seconds late...
Stream is 45 seconds late...
Stream is 60 seconds late...
Stream is 75 seconds late...
Stream is 90 seconds late...
Stream is 105 seconds late...
Stream is 120 seconds late...
Stream is 135 seconds late...
Stream is 150 seconds late...
Stream is 165 seconds late...
Stream is 180 seconds late...
Stream is 195 seconds late...
Stream is 210 seconds late...
Stream is 225 seconds late...
Stream is 240 seconds late...
Stream is 255 seconds late...
Stream is 270 seconds late...
Stream is 285 seconds late...
Stream is 300 seconds late...
Stream is 315 seconds late...
Stream is 330 seconds late...
Stream is 345 seconds late...
Stream is 360 seconds late...
Stream is 375 seconds late...
Stream is 390 seconds late...
Stream is 405 seconds late...
Stream is 420 seconds late...
Stream is 435 seconds late...
Stream is 450 seconds late...
Stream is 465 seconds late...
Stream is 480 seconds late...
Stream is 495 seconds late...
Stream is 510 seconds late...
Stream is 525 seconds late...
Stream is 540 seconds late...
Stream is 555 seconds late...

2023/03/22 07:09:32 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 07:09:33 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 07:09:33 Selected quality: 1080p60 (h264)
2023/03/22 07:09:33 WARNING: Formatted filename is too long. Truncating the title to try and fix.
2023/03/22 07:09:33 INFO: Starting download to E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f140.ts
2023/03/22 07:09:33 INFO: Starting download to E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f299.ts
2023/03/22 07:33:53 DEBUG: video3: Error with fragment 1466: unexpected EOF
2023/03/22 07:34:14 DEBUG: audio1: Error with fragment 1487: unexpected EOF
2023/03/22 08:00:15 DEBUG: audio4: Error with fragment 3048: unexpected EOF
Video Fragments: 3608; Audio Fragments: 3607; Max Fragments: 3606; Max Sequence: 3606; Total Downloaded: 1.20GiB
2023/03/22 08:09:33 ERROR: Error retrieving player response: unable to retrieve player response object from watch page
2023/03/22 08:09:33 DEBUG: audio4: Stream is finished and highest sequence reached
2023/03/22 08:09:33 DEBUG: audio4: exiting
2023/03/22 08:09:33 DEBUG: audio2: exiting
2023/03/22 08:09:33 DEBUG: audio1: exiting
2023/03/22 08:09:33 DEBUG: audio3: exiting
2023/03/22 08:09:34 DEBUG: audio-download thread closing
2023/03/22 08:09:34 DEBUG: video4: Stream is finished and highest sequence reached
2023/03/22 08:09:34 DEBUG: video4: exiting
2023/03/22 08:09:34 DEBUG: video3: exiting
2023/03/22 08:09:34 DEBUG: video2: exiting
2023/03/22 08:09:34 DEBUG: video1: exiting
2023/03/22 08:09:34 DEBUG: video-download thread closing
2023/03/22 08:09:34 9; Audio Fragments: 3608; Max Fragments: 3607; Max Sequence: 3607; Total Downloaded: 1.20GiB
Download Finished
2023/03/22 08:09:34 WARNING: Mismatched number of video and audio fragments.
2023/03/22 08:09:34 WARNING: The files should still be mergable but data might be missing.
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f140.ts to E:\YouTube\DL\【＃生スバル】バイ オハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f140.ts
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f299.ts to E:\YouTube\DL\【＃生スバル】バイ オハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.f299.ts
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.jpg to E:\YouTube\DL\【＃生スバル】バイオハ ザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.jpg
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【＃生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.description to E:\YouTube\DL\【＃生スバル】 バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ！！！！！！！ _  biohazard-puNX_86hAM0.description
2023/03/22 08:09:34 Muxing final file...

and one without it:

ytarchive 0.3.2
2023/03/21 22:31:42 Channel: Matsuri Channel 夏色まつり
2023/03/21 22:31:42 Video Title: 【 APEX 】V最協顔合わせ！【ホロライブ/夏色まつり】

2023/03/21 22:31:42 Stream starts at 2023-03-22T09:00:00+00:00 in 19698 seconds.
2023/03/21 22:31:42 Waiting for this time to elapse...
2023/03/22 03:59:59 DEBUG: Woke up 1 seconds early. Continuing sleep...
Stream is 15 seconds late...
Stream is 30 seconds late...
Stream is 45 seconds late...

2023/03/22 04:00:47 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 04:00:47 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 04:00:47 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 04:00:47 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 04:00:47 Selected quality: 1080p (h264)
2023/03/22 04:00:47 INFO: Starting download to E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f140.ts
2023/03/22 04:00:47 INFO: Starting download to E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f137.ts
2023/03/22 04:14:17 DEBUG: audio4: Error with fragment 831: unexpected EOF
2023/03/22 04:47:50 DEBUG: audio4: Error with fragment 2844: unexpected EOF
2023/03/22 05:00:46 DEBUG: video2: Error with fragment 3617: unexpected EOF
2023/03/22 05:00:48 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 05:00:48 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 05:00:48 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 05:00:48 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 05:26:41 DEBUG: audio3: Error with fragment 5173: unexpected EOF
2023/03/22 05:37:05 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:05 DEBUG: video4: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:12 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:14 DEBUG: video4: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:20 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 06:00:49 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 06:00:49 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 06:00:49 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 06:00:49 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 06:08:02 DEBUG: video1: Error with fragment 7632: unexpected EOF
2023/03/22 06:09:10 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:10 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:17 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:18 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:24 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:27 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:32 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:35 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:40 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:44 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:47 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:52 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:55 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:01 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:03 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:05 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:06 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:08 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:09 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:09 DEBUG: audio1: Fragment 7694: 10/10 retries
2023/03/22 06:10:10 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 06:10:11 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:12 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 06:10:12 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 06:10:12 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 06:10:12 DEBUG: audio2: exiting
2023/03/22 06:10:12 DEBUG: audio3: exiting
2023/03/22 06:10:12 DEBUG: audio4: exiting
2023/03/22 06:10:12 DEBUG: audio1: exiting
2023/03/22 06:10:12 DEBUG: video4: Stream has ended and fragment number is within two of the known max, probably not actually created
2023/03/22 06:10:12 DEBUG: audio-download thread closing
2023/03/22 06:10:12 DEBUG: video3: exiting
2023/03/22 06:10:12 DEBUG: video2: exiting
2023/03/22 06:10:12 DEBUG: video1: exiting
2023/03/22 06:10:12 DEBUG: video4: exiting
2023/03/22 06:10:12 DEBUG: video-download thread closing
2023/03/22 06:10:12 4; Audio Fragments: 7694; Max Fragments: 7693; Max Sequence: 7693; Total Downloaded: 3.77GiB
Download Finished
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.f140.ts to E:\YouTube\DL\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f140.ts
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライ
ブ_夏色まつり】-XeRxEkENvzQ.f137.ts to E:\YouTube\DL\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f137.ts
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.jpg to E:\YouTube\DL\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.jpg
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.description to E:\YouTube\DL\【 APEX 】V最協顔合わせ！_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.description
2023/03/22 06:10:12 Muxing final file...

The former died at an hour despite the stream continuing while the latter was fine. Neither of these streams are members streams or actually need cookies, but I tend to leave the option in for convenience and only update the cookies file on the rare occasion I archive a members stream, so they are usually being loaded while out of date. Up until now this hasn’t been a problem outside of members streams themselves, but now I wonder if whatever refresh is happening at 1 hour is causing youtube to look at the outdated cookies for some reason.

[Kethsar]

Someone else I have contact with ran into this too recently. I had him use a quick custom build that dumped the HTML on error. The image of the page is below. Unfortunately, I have no idea what to do about this. Apparently it's not even consistent.

[CatsCodes]

Removing cookies so far does seem avoid the error. I found this error again here: https://arca.live/b/vtubers/51239046 and they talk about environment variable 1024. I hope this helps.

[NeloBlivion]

Also been getting this issue with cookies, but it's not the most consistent... some streams break at 1 hour (and can break later after starting another download), while others run perfectly fine for hours. And sometimes when a stream ends and the archive's finished downloading, it will show the player response error even though nothing's broken.

now I wonder if whatever refresh is happening at 1 hour is causing youtube to look at the outdated cookies for some reason.

I noticed that yt-dlp actively updates the cookies.txt file whenever you download through it; I haven't paid attention to what it changes, but perhaps this needs to be implemented here?

[NeloBlivion]

I think that's exactly what we need to implement, i've tried exporting a new cookies.txt before the hour mark and even though the error still popped up, it continued to download after (presumably by reading the file again)

[Kethsar]

That is interesting, but also separate from the issue at hand. That one does appear to be cookies no longer working mid-download, but it still retrieved the actual watch page, as opposed to some random page about cookie settings. Also it does not re-read the cookies file in the middle of downloading, only once at the start of loading yta (or at the start of each loop when monitoring a channel).

According to the Golang docs: The [Cookie]Jar is used to insert relevant cookies into every outbound Request and is updated with the cookie values of every inbound Response. This means that any updated cookies are already in use in the running instance, so writing them to file won't actually help. If the cookies in the file itself were stale, it would immediately cause issues for a membership stream. I'm not sure what Youtube has been doing to break cookies on occasion like this.

I could potentially write updated cookies to the file for the hell of it, but if you are running multiple instances of yta at once, that could be messy. I'm not sure how useful that actually is either since they are already desynced from your browser cookies the moment you export and load another Youtube page.

[lbmaian]

Related issue for yt-dlp has a user workaround: https://github.com/yt-dlp/yt-dlp/issues/8227#issuecomment-1793513579

A simple workaround would be to generate the cookies file in an incognito window, where the cookies are never used again in browser and therefore will never be rotated.