Sullyscharger
commented
11 months ago Yeah, I've been getting this issue as well, also another where it freezes whilst recording so even when the live is over the "recording" doesn't stop.
Closed CyberbobGR closed 9 months ago
Sullyscharger
commented
11 months ago Yeah, I've been getting this issue as well, also another where it freezes whilst recording so even when the live is over the "recording" doesn't stop.
Marksmanship256
commented
11 months ago I've encountered the same issue, but I began using proxies and initially suspected it might be due to them.
Liam942
commented
11 months ago I have the same issue with long recordings ending up much shorter, started noticing it yesterday on several users. no proxy used.
roju
commented
11 months ago I have not seen it happen in my testing. Anyone have an idea what could be causing this? Maybe it has something to do with ffmpeg freezing up without returning any errors?
When the recording is shorter than expected, which portion gets recorded? Does it record only the beginning, or are there random segments missing? Does the end get recorded?
CyberbobGR
commented
11 months ago Here is another example that happened a few hours ago.
2023-09-23 21:18:54,643 - INFO - username is live 2023-09-23 21:18:54,643 - INFO - Getting live url for room ID 7282086754151041824 2023-09-23 21:18:55,088 - INFO - Live URL: https://pull-f5-tt03.fcdn.eu.tiktokcdn.com/stage/stream-3284316747631690621_or4.flv 2023-09-23 21:18:55,089 - INFO - Output directory: C:\tiktok-live-recorder-main\ 2023-09-23 21:18:57,025 - INFO - Started recording Press 'q' to re-start recording, CTRL + C to stop frame=43785 fps= 25 q=-1.0 Lsize= 92488kB time=00:29:17.14 bitrate= 431.2kbits/s speed= 1x 2023-09-23 21:48:10,489 - ERROR - FFmpeg error: 2023-09-23 21:48:10,490 - ERROR - [tls @ 0000020111f71580] Decryption has failed. [tls @ 0000020111f71580] The specified session has been invalidated for some reason. [NULL @ 00000201120f8a40] Invalid NAL unit size (5159 > 3048). [NULL @ 00000201120f8a40] missing picture in access unit with size 3052 [tls @ 0000020111f71580] The specified session has been invalidated for some reason. https://pull-f5-tt03.fcdn.eu.tiktokcdn.com/stage/stream-3284316747631690621_or4.flv: I/O error 2023-09-23 21:48:17,634 - INFO - Started recording Press 'q' to re-start recording, CTRL + C to stop frame=125069 fps= 25 q=-1.0 Lsize= 252080kB time=01:23:36.53 bitrate= 411.6kbits/s speed= 1x 2023-09-23 23:11:52,285 - INFO - username is offline 2023-09-23 23:11:52,286 - INFO - Concatenating 2 video files 2023-09-23 23:11:53,141 - INFO - Recording finished: C:\tiktok-live-recorder-main\TK_username_2023.09.23_23-11-52_concat.mp4
total duration of the final video: 00:29:17
I think only the part before the error was saved. Maybe the rest of the live was also recorded in a seperate file but for some reason it wasn't merged at the combination? I'm not sure.
Liam942
commented
11 months ago total duration of the final video: 00:29:17
I think only the part before the error was saved. Maybe the rest of the live was also recorded in a seperate file but for some reason it wasn't merged at the combination? I'm not sure.
In my case I checked for extra files after it combined, I could not find any. The end result when this error occurs for me is always 1 combined file but shorter and smaller in size than what is supposed to be.
CyberbobGR
commented
11 months ago total duration of the final video: 00:29:17 I think only the part before the error was saved. Maybe the rest of the live was also recorded in a seperate file but for some reason it wasn't merged at the combination? I'm not sure.
In my case I checked for extra files after it combined, I could not find any. The end result when this error occurs for me is always 1 combined file but shorter and smaller in size than what is supposed to be.
After the files are combined the old files are automatically deleted.
Liam942
commented
11 months ago After the files are combined the old files are automatically deleted.
Yes I understand that the combined file is supposed to replace the other files, but it seems like in this error all the old files get removed once the live has ended like it should, but only some of them end up in the combined file making the finished file shorter and smaller in size.
MoosMas
commented
11 months ago I've also had this issue with a live from last night. This is the first time I noticed. The logs are below. As you can see, the live lasted for about 26 minutes, but the video is only 48 seconds long. I hope it can be fixed soon!
Command I used: py ttlr.py -mode auto -out_dir downloads -ffmpeg -combine -user username
2023-09-24 22:58:40,881 - INFO - username is live
2023-09-24 22:58:40,881 - INFO - Getting live url for room ID 7282499238*********
2023-09-24 22:58:41,873 - INFO - Live URL: https://pull-f5-tt03.fcdn.eu.tiktokcdn.com/stage/stream-3284323195851637629_or4.flv
2023-09-24 22:58:41,875 - INFO - Output directory: downloads\
2023-09-24 22:58:44,182 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 1179 fps= 21 q=-1.0 Lsize= 13135kB time=00:00:48.88 bitrate=2201.0kbits/s speed=0.891x
2023-09-24 22:59:38,532 - ERROR - FFmpeg error:
2023-09-24 22:59:38,532 - ERROR - [tls @ 0000010d672c3700] Decryption has failed.
[tls @ 0000010d672c3700] The specified session has been invalidated for some reason.
[NULL @ 0000010d6786a400] Invalid NAL unit size (6392 > 3825).
[NULL @ 0000010d6786a400] missing picture in access unit with size 3829
[tls @ 0000010d672c3700] The specified session has been invalidated for some reason.
Error demuxing input file 0: I/O error
https://pull-f5-tt03.fcdn.eu.tiktokcdn.com/stage/stream-3284323195851637629_or4.flv: I/O error
2023-09-24 22:59:49,202 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 220 fps= 19 q=-1.0 Lsize= 2393kB time=00:00:08.81 bitrate=2224.3kbits/s speed=0.75x
2023-09-24 23:00:07,607 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame=29102 fps= 25 q=-1.0 Lsize= 321459kB time=00:19:27.25 bitrate=2256.1kbits/s speed=1.01x
2023-09-24 23:19:35,344 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 7643 fps= 25 q=-1.0 Lsize= 40453kB time=00:05:06.35 bitrate=1081.7kbits/s speed=1.02x
2023-09-24 23:24:41,506 - INFO - username is offline
2023-09-24 23:24:41,506 - INFO - Concatenating 4 video files
2023-09-24 23:24:41,607 - INFO - Recording finished: downloads\TK_username_2023.09.24_23-24-41_concat.mp4```I was able to reproduce it one time. Doing some more testing and debugging now.
MoosMas
commented
11 months ago I don't know how I can help with debugging except for providing more logs, since I don't have much experience with Python. I hope the logs can be of use.
Command: py ttlr.py -mode auto -out_dir downloads -ffmpeg -combine -user username
2023-10-02 02:11:54,508 - INFO - username is live
2023-10-02 02:11:54,508 - INFO - Getting live url for room ID 72851466107********
2023-10-02 02:11:55,111 - INFO - Live URL: https://pull-f5-tt01.fcdn.eu.tiktokcdn.com/stage/stream-3860825310833148172_or4.flv
2023-10-02 02:11:55,112 - INFO - Output directory: downloads\
2023-10-02 02:11:57,938 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 2409 fps= 25 q=-1.0 Lsize= 34439kB time=00:01:42.58 bitrate=2750.1kbits/s speed=1.07x
2023-10-02 02:13:33,227 - ERROR - FFmpeg error:
2023-10-02 02:13:33,227 - ERROR - [tls @ 0000022e65023700] Decryption has failed.
[tls @ 0000022e65023700] The specified session has been invalidated for some reason.
[NULL @ 0000022e6502ed40] Invalid NAL unit size (6405 > 3805).
[NULL @ 0000022e6502ed40] missing picture in access unit with size 3809
[tls @ 0000022e65023700] The specified session has been invalidated for some reason.
Error demuxing input file 0: I/O error
https://pull-f5-tt01.fcdn.eu.tiktokcdn.com/stage/stream-3860825310833148172_or4.flv: I/O error
2023-10-02 02:13:40,367 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 1861 fps= 27 q=-1.0 Lsize= 21889kB time=00:01:17.08 bitrate=2326.2kbits/s speed=1.11x
2023-10-02 02:14:56,609 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 4882 fps= 26 q=-1.0 Lsize= 68419kB time=00:03:15.51 bitrate=2866.7kbits/s speed=1.04x
2023-10-02 02:18:11,417 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 3276 fps= 27 q=-1.0 Lsize= 46441kB time=00:02:10.98 bitrate=2904.5kbits/s speed=1.06x
2023-10-02 02:20:21,390 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 8786 fps= 25 q=-1.0 Lsize= 116183kB time=00:06:03.41 bitrate=2618.9kbits/s speed=1.02x
2023-10-02 02:26:23,445 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame=11422 fps= 25 q=-1.0 Lsize= 148752kB time=00:07:40.53 bitrate=2646.0kbits/s speed=1.02x
2023-10-02 02:34:05,652 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 9725 fps= 25 q=-1.0 Lsize= 116752kB time=00:06:29.78 bitrate=2453.7kbits/s speed=1.01x
2023-10-02 02:40:32,476 - ERROR - FFmpeg error:
2023-10-02 02:40:32,477 - ERROR - [tls @ 00000286835a3700] Decryption has failed.
[tls @ 00000286835a3700] The specified session has been invalidated for some reason.
Last message repeated 1 times
Error demuxing input file 0: I/O error
https://pull-f5-tt01.fcdn.eu.tiktokcdn.com/stage/stream-3860825310833148172_or4.flv: I/O error
2023-10-02 02:40:39,547 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 4001 fps= 26 q=-1.0 Lsize= 53232kB time=00:02:40.01 bitrate=2725.3kbits/s speed=1.05x
2023-10-02 02:43:17,718 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 5706 fps= 26 q=-1.0 Lsize= 81367kB time=00:03:48.45 bitrate=2917.7kbits/s speed=1.04x
2023-10-02 02:47:04,186 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 5265 fps= 26 q=-1.0 Lsize= 71013kB time=00:03:31.58 bitrate=2749.5kbits/s speed=1.04x
2023-10-02 02:50:33,392 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 3776 fps= 26 q=-1.0 Lsize= 51847kB time=00:02:30.98 bitrate=2813.1kbits/s speed=1.06x
2023-10-02 02:53:02,640 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame=11662 fps= 25 q=-1.0 Lsize= 162163kB time=00:07:46.45 bitrate=2848.0kbits/s speed=1.02x
2023-10-02 03:00:47,648 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 6809 fps= 26 q=-1.0 Lsize= 93991kB time=00:04:32.71 bitrate=2823.3kbits/s speed=1.03x
2023-10-02 03:05:18,002 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 5807 fps= 25 q=-1.0 Lsize= 81661kB time=00:03:52.97 bitrate=2871.4kbits/s speed=1.02x
2023-10-02 03:09:06,114 - ERROR - FFmpeg error:
2023-10-02 03:09:06,114 - ERROR - [tls @ 000001816d023700] Decryption has failed.
[tls @ 000001816d023700] The specified session has been invalidated for some reason.
Last message repeated 1 times
Error demuxing input file 0: I/O error
https://pull-f5-tt01.fcdn.eu.tiktokcdn.com/stage/stream-3860825310833148172_or4.flv: I/O error
2023-10-02 03:09:12,945 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 4048 fps= 26 q=-1.0 Lsize= 56283kB time=00:02:41.88 bitrate=2848.2kbits/s speed=1.05x
2023-10-02 03:11:53,898 - INFO - Started recording
Press 'q' to re-start recording, CTRL + C to stop
frame= 3727 fps= 26 q=-1.0 Lsize= 51856kB time=00:02:31.79 bitrate=2798.6kbits/s speed=1.06x
2023-10-02 03:14:22,257 - INFO - username is offline
2023-10-02 03:14:22,257 - INFO - Concatenating 16 video files
2023-10-02 03:14:22,479 - INFO - Recording finished: downloads\TK_username_2023.10.02_03-14-22_concat.mp4The live lasted for about an hour according to the logs, but the resulting video is only 1 minute and 42 seconds long.
Marksmanship256
commented
11 months ago Hello, @roju !
Can we create a new branch in the repository where we eliminate the delete of file segments? Instead, these files would be saved in a folder named as the username. The concatenated files would still be stored in a shared folder as usual. This way, we can manually review the recording video pieces and delete them if needed.
roju
commented
11 months ago Hey sorry everyone, I haven’t had much time to work on this recently.
For now, to prevent the automatic deletion of recorded files, you can either remove the -combine option, or remove line 196 in tiktok_bot.py
Also, it would be helpful if you could let me know if there is still data loss even if the recorded video files are not automatically deleted after concatenation. Are the recorded segments lost during live recording, or during concatenation?
Liam942
commented
11 months ago from what I can tell, it seems like the files exist before the combination at the end which makes some disapear. Does seem to affect towards the mid-end of the recording, not the begining.
Sullyscharger
commented
10 months ago Is anyone else getting the issue where the recording just freezes when the stream lags hence not recording the rest of the stream and doesn't end until you manually end the recording?
Liam942
commented
10 months ago Is anyone else getting the issue where the recording just freezes when the stream lags hence not recording the rest of the stream and doesn't end until you manually end the recording?
For me I think I stopped having that issue of having to force end it when I disabled quick edit mode in cmd
Sullyscharger
commented
10 months ago Is anyone else getting the issue where the recording just freezes when the stream lags hence not recording the rest of the stream and doesn't end until you manually end the recording?
For me I think I stopped having that issue of having to force end it when I disabled quick edit mode in cmd
That hasn't worked for me, sometimes it will run perfectly fine but other times it would just linger on the recording with the timer stopped until I press ctrl c to manually end it
roju
commented
10 months ago I think I've found the cause. I have a proof-of-concept which seems to fix the issue, but it's a bit hacky and complicated. Would prefer a more simple/elegant solution if possible. I may need to move the fix to a separate dev branch for testing, not sure yet.
MoosMas
commented
10 months ago Curious to know what the cause and fix is. Meanwhile, I don't know if you've noticed before, but if part of the livestream is lost, it often seems like the concatenated video is the same length as the first video/part of the recorded livestream. I have removed line 196 in tiktok_bot.py, so all parts are saved along with the concatenated video.
In the screenshot below, you can see the first video of the livestream is 1:53 long, and the concatenated video has the same length.
Don't know if this is relevant to the bug, but I did some testing and found something that might be worth checking out. I have used ffprobe to count the frames in both the first video and the concatenated one, and the concatenated one has 1 frame less. Also, the first video throws this error when running the command:
[NULL @ 0000018cfee09080] Invalid NAL unit size (7039 > 3837).
[NULL @ 0000018cfee09080] missing picture in access unit with size 3841The ffprobe command I used to count the frames:
ffprobe -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 {filename}
Full output of the ffprobe commands:
The number of frames in each video is in the last line of both outputs. As you can see, the concatenated video has 1 frame less. I have also tested this with other videos where the concatenated video is the same length as the first video, and they all have 1 frame less. The first video also throws the same error as the one above (just with different numbers).
I have also tested this with videos where the concatenation was successful, so the length of the concatenated video is equal to the sum of all parts. Here, the concatenated video's framecount is also equal to the sum of the framecount of all parts (i.e.: it's not missing any frames), and the first video doesn't throw any errors.
I don't know if this is what's causing it, but I hope all the info is useful during debugging.
roju
commented
10 months ago @MoosMas Thanks for the info. You are correct. I have been doing some extensive testing and I noticed the same.
Basically, when recording, sometimes the video clip gets corrupted with the "Invalid NAL unit size" error. The corrupted file causes the concat operation to fail at that point. However, only the very end of the video clip contains corrupted frames (or maybe only one corrupted frame?). So the fix is to remove some frames from the very end of the video. After that, the concatenation works without errors. I'm working on automating this which includes improved error handling.
I could not figure out which options I can add to ffmpeg to tell it to simply drop the corrupted frames while recording. Maybe it isn't possible. So for now I have to do it this way. Also it would be nice to prevent the NAL unit size error in the first place so none of this would be necessary but I couldn't figure that out either.
roju
commented
9 months ago Made some changes in v4.2. Now moves the recorded segments to a new folder after the concat instead of deleting them. Also catches and logs any ffmpeg concat errors.
roju
commented
9 months ago In my testing, it seems that the "invalid NAL unit" error no longer occurs and the concat operation no longer fails in v4.2. I believe the additional ffmpeg options added on lines 153-155 may have fixed it.
If this issue is still occurring for anyone on v4.2, please let me know. Otherwise I will close this issue.
roju
commented
9 months ago Although the related issue of occasional corrupted segments may or may not be resolved, the original issue of data loss during concatenation was fixed in v4.2 and 6001510c3d1578933631994893667f9b9c2ec992. So I am closing this issue.
There is some problem when merging multiple videos together. For example this from a 3-hour live, the final video saved is only 33 minutes.
I used this fromat:
ttlr.py -mode auto -ffmpeg -combine -user username
I had no problems in the past, but today this is the second time a live is not saved properly.