Leave Some Disk Space

[Odogwu3000]

Hello, I really appreciate this great work. Would it be possible to set a disk space to be left to avoid filling the disk to 100%.

[DerBunteBall]

Hi,

I think this actually would only be solveable with external tools.

You could try to solve it by setting up user quotas for the disk. So create a user for streaMonitor and run it under this user. For the user set the wanted disk quota (check Debian/Ubuntu Package quota - other distros it would be named similar I think).

I think it's not totally clear how the app would behave if the quota is reached. Normally youtube-dl and ffmpeg should throw an error like "disk full". I think in best case just the worker threads would switch to Not Running. In a bad case the Manager thread also dies. I think there is no explicit error handling for such an situation actually.

Best Regards

[lossless1024]

I put this on my To-Do list. The config file needs to be restructured a bit, but then you will be able to set a limit and if that is reached, the manager will pause checking online shows. Another problem is that currently it is not possible to stop the recordings, so when the quota level is reached, there is a chance that the running shows will still fill the disk. I started to restructure the manager from to use processes instead of threads, but not everything works properly. (TLDR it is problematic.) The big issue is that I used youtube-dl for the downloading process, but it doesn't have an interface to stop itself. This means I would have to kill the process to stop it and that way the file will still be an mp4.part so... I think I'll implement a hls downloader myself.

[Odogwu3000]

I put this on my To-Do list. The config file needs to be restructured a bit, but then you will be able to set a limit and if that is reached, the manager will pause checking online shows. Another problem is that currently it is not possible to stop the recordings, so when the quota level is reached, there is a chance that the running shows will still fill the disk. I started to restructure the manager from to use processes instead of threads, but not everything works properly. (TLDR it is problematic.) The big issue is that I used youtube-dl for the downloading process, but it doesn't have an interface to stop itself. This means I would have to kill the process to stop it and that way the file will still be an mp4.part so... I think I'll implement a hls downloader myself.

Thanks for the response. I think you are spot-on with the suggested approach. I figured stopping Youtube-dl downloading processes as you mentioned with be a challenge. How about using ffmpeg for the downloading processes? I see you have it in the bot code. May be capturing using ffmepg as .ts instead of mp4 and having a post-processing after the download process is finished can help.

[lossless1024]

I think this will be the solution. I started to migrate to ffmpeg. Also ffmpeg can save directly to mp4, so there are no drawbacks. Even better, because ffmpy stops ffmpeg properly, so there will be no more remaining .part files. I'll push the updated code soon.

[sminato27]

There's any way to add a feature to compress file without losing quality/resolution when a cam end the recording? It would help with disk space.

[DerBunteBall]

I think a feature to move completed downloads to another directory would be a first helpful thing.

So it's easy to cleanup a recording system or simply have multiple disks.

[lossless1024]

@sminato27 Everything is possible, but my time is limited. I'll think about it.

@DerBunteBall Your only request is the ability to set a different download directory or something else?

[DerBunteBall]

@lossless1024: An ability to set two direcotries.

For example the incmoing (when a download runs) its stored in /data/incoming and then one for completed downlaods (data will be moved to after downlaod finished) e.g. /data/completed/

So because of the fact that you can mount like you want it could also be something like /disk1 and /disk2.

It in fact can be every directory but it would be the best to have a way to set these directories in config.json or so. Something like this:

{
   "incoming_dir": "/data/incoming",
   "completed_dir": "/data/completed"
}

It makes it a bit easier to e.g. rsync stuff away from a recording box to somehwere else and clenaup. Actually it's needed to check for .part extensions or so. In the above case you could get all out of /data/completed/ (be sure it's complete) and clenaup this dir without having the risk to delete stuff away from running processes by accident.

[lossless1024]

Technically, is possible, I put it on my TODO list. Also the current version uses ffmpeg directly and there will be no more .part residues as this way ffmpeg can be closed cleanly.

[eldepor]

Just a question, with the new code there're no .part files anymore. Now when you manually stop a recording it leaves an mp4 file, but that file can't be played, like it's corrupted or something. Is it normal? Thanks

[sminato27]

Just a question, with the new code there're no .part files anymore. Now when you manually stop a recording it leaves an mp4 file, but that file can't be played, like it's corrupted or something. Is it normal? Thanks

I've faced the same thing with .part files... Sometimes when changing the .part for .mp4 it works but sometimes it gets corrupted, dunno why.

[eldepor]

Just a question, with the new code there're no .part files anymore. Now when you manually stop a recording it leaves an mp4 file, but that file can't be played, like it's corrupted or something. Is it normal? Thanks

I've faced the same thing with .part files... Sometimes when changing the .part for .mp4 it works but sometimes it gets corrupted, dunno why.

I guess because the file isn't finished so the index is broken or something. But I'm no expert. I used to fix .part files with a tool called "untrunc". Now there are no part files anymore which is nice, but the mp4 of a stopped recording still can't be played, so I was wondering if this is normal/intentional or there's any way that ffmpeg finishes the file when you stop the recording just as when the cam goes offline, so it can be played without the use of an external fix tool.

This is what the console returns when a recording is stopped manually:

[lossless1024]

Do you guys use windows? On Linux I only experience this when i quit from the program entirely and mostly those files are good too. How often does it happen to you? Everytime, almost everytime, 50-50, rarely, or occasionally? Maybe there is a slight difference on windows which makes ffmpeg not to exit cleanly.

[DerBunteBall]

They use Windows.

I'm not able to understand the stopping mechanism at all in short time.

Leaving the whole App leads to such results because e.g. giving it a Ctrl-C kills everything. It then depends in detail how the Threads and their child processes stop.

I think the problem is eventually related to line 22 of the ffmpeg Downlaoder.

FFmpy has in fact no clean way to stop FFmpeg as far as I can see. As far as I can see the process property of the FFmpeg Class and here the ff Object in fact is a Popen object.

1. Popen behaves different in detail on POSIX and Windows.
2. It has the terminate() method. The terminate method stops the process. On POSIX it's doing a SIGTERM (which is a like a soft kill for me) on Windows the Win32 API surfes this (https://docs.python.org/3/library/subprocess.html#popen-objects). I don't know which Signal this sends but it could lead to a different quitting of ffmpeg. Eventually on Windows it kills it hard with a SIGKILL or so. For me it looks like it's a harder kill (https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess; I'm not familiar with Windows process signaling).
3. Line 22 is a bit confusing because it doesn't calls terminate as a method. Instead it looks more like an attribute which is written to another variable. Due to the fact that functions are first class citizens in Python I think it's interpreted as a method call.
4. .send_signal() could be used to send a better signal on Windows.

Best Reagerds

P.S. All just thesis. p.P.S. I see now it's wanted in this way. The function is assigned to variable. So the .stopDownload() call in fact is a .terminate() call.

[lossless1024]

Thank You, @DerBunteBall. I didn't know about this windows thing. I checked the send_signal you've mentioned, however I don't use windows, so please test the fix on the latest commit.

[eldepor]

Thank You, @DerBunteBall. I didn't know about this windows thing. I checked the send_signal you've mentioned, however I don't use windows, so please test the fix on the latest commit.

Thanks for the update, but unfortunately it doesn't stop the current recording now. No error messages as well, the console just says "stopping" but nothing happens. The only way I found to stop a recording and getting a playable file is by closing the command line window clicking the X, it takes a few seconds before closing completely. So maybe there is the signal we are looking for.

Thanks

Console output while closing the window:

[DerBunteBall]

Thank You, @DerBunteBall. I didn't know about this windows thing. I checked the send_signal you've mentioned, however I don't use windows, so please test the fix on the latest commit.

Thanks for the update, but unfortunately it doesn't stop the current recording now. No error messages as well, the console just says "stopping" but nothing happens. The only way I found to stop a recording and getting a playable file is by closing the command line window clicking the X, it takes a few seconds before closing completely. So maybe there is the signal we are looking for.

Thanks

Which window do you close?

Handles windows the stuff in the way that you have the main program (Downloader.py) in a PowerShell/CMD and spawns a CMD for every ffmpeg session?

Actually the new code sends a CTRL_C_EVENT (this could be without an effect - check below). I would try to send a SIGTERM or SIGINT explicitly. If my thesis is correct .terminate() sends a SIGABRT or SIGKILL on Windows because TerminateProcess() in Win32 API does this (in detail the signals could be differently handled on POSIX and Win32). So ffmpeg has no ability to run code to handle this. When it runs in a seperate CMD eventually the CMD (windows terminal emulater) sends a SIGTERM. This is normal on POSIX. It sounds like that because it takes a few seconds to close. That an exception is thrown could be normal because ffmpeg eventually gives another exit code when it's stopped while a input source (here the stream) still gives data.

So for the Windows guys: In streamonitor/downloaders/ffmpeg.py at line 25 change CTRL_C_EVENT to SIGTERM and try again. If this doesn't help you could try to change to SIGINT. Give a feedback so that @lossless1024 can bring the needed signal mainline.

Also check docs CTRL_C_EVENT can only be used with os.kill(). So it seems it has no effect at .send_signal(). https://docs.python.org/3/library/signal.html

Best Regards

[eldepor]

Thank You, @DerBunteBall. I didn't know about this windows thing. I checked the send_signal you've mentioned, however I don't use windows, so please test the fix on the latest commit.

Thanks for the update, but unfortunately it doesn't stop the current recording now. No error messages as well, the console just says "stopping" but nothing happens. The only way I found to stop a recording and getting a playable file is by closing the command line window clicking the X, it takes a few seconds before closing completely. So maybe there is the signal we are looking for. Thanks

Which window do you close?

Handles windows the stuff in the way that you have the main program (Downloader.py) in a PowerShell/CMD and spawns a CMD for every ffmpeg session?

Actually the new code sends a CTRL_C_EVENT (this could be without an effect - check below). I would try to send a SIGTERM or SIGINT explicitly. If my thesis is correct .terminate() sends a SIGABRT or SIGKILL on Windows because TerminateProcess() in Win32 API does this (in detail the signals could be differently handled on POSIX and Win32). So ffmpeg has no ability to run code to handle this. When it runs in a seperate CMD eventually the CMD (windows terminal emulater) sends a SIGTERM. This is normal on POSIX. It sounds like that because it takes a few seconds to close. That an exception is thrown could be normal because ffmpeg eventually gives another exit code when it's stopped while a input source (here the stream) still gives data.

So for the Windows guys: In streamonitor/downloaders/ffmpeg.py at line 25 change CTRL_C_EVENT to SIGTERM and try again. If this doesn't help you could try to change to SIGINT. Give a feedback so that @lossless1024 can bring the needed signal mainline.

Also check docs CTRL_C_EVENT can only be used with os.kill(). So it seems it has no effect at .send_signal(). https://docs.python.org/3/library/signal.html

Best Regards

The window I close is cmd.exe (C:/Windows/system32/cmd.exe)

To start the Downloader I always Shift+Right Click the folder where Downloader.py is placed, open cmd there and type "py Downloader. py" (maybe I'm doing something wrong?)

I tried SIGTERM and SIGINT but no luck. With SIGINT, it returns error (ERROR - manager_zmq: Unsupported signal: 2) and with SIGTERM the download is stopped with exit status 1 and the file is still unplayable.

But if you close cmd.exe, each ffmpeg.exe stance (one per current recording) exits with a big number status and the file is playable/finished. If you compare the two screenshots I sent, you can notice the different exit status, I don't know if this has something to do but when the exit status is 1 the file is corrupted, but if the exit status is a big number the file is ok. And I can only have the second exit status by closing cmd.exe

Hope it helps. Thanks

[DerBunteBall]

I don't think that something is wrong. I actually think you have one main cmd.exe and a cmd.exe for a job. That was my question gets and ffmpeg job an own cmd.exe?

It seems to be really important for Windows. So a short research on the web gives some hints that the attaching of processes to a console leads to very different signal handling.

That the exit status is different does tell us what you already figured out: The terminal emulator stops ffmpeg in a way that it finishes cleanly.

So something is different by what the cmd.exe does and that we need to do the same way or as part of that way.

A short search on the internet says that cmd.exe sends a SIGBREAK when it's closed. SIGBREAK is also available in Python signal and is declared as only available on Windows. So a silly test would be to set SIGBREAK and try again.

The error you get with SIGINT indicates the above differences. Console attached processes seem to not get SIGINT singals. So it's possible that parts of ZeroMQ getting it because of the complex software staple behind (Python, ZeroMQ Library which is eventually not attached and so on) and tells something like I don't know what to do with that.

[eldepor]

I don't think that something is wrong. I actually think you have one main cmd.exe and a cmd.exe for a job. That was my question gets and ffmpeg job an own cmd.exe?

It seems to be really important for Windows. So a short research on the web gives some hints that the attaching of processes to a console leads to very different signal handling.

That the exit status is different does tell us what you already figured out: The terminal emulator stops ffmpeg in a way that it finishes cleanly.

So something is different by what the cmd.exe does and that we need to do the same way or as part of that way.

A short search on the internet says that cmd.exe sends a SIGBREAK when it's closed. SIGBREAK is also available in Python signal and is declared as only available on Windows. So a silly test would be to set SIGBREAK and try again.

The error you get with SIGINT indicates the above differences. Console attached processes seem to not get SIGINT singals. So it's possible that parts of ZeroMQ getting it because of the complex software staple behind (Python, ZeroMQ Library which is eventually not attached and so on) and tells something like I don't know what to do with that.

Tried SIGBREAK:

ERROR - manager _zmq: unsupported signal: 21

[DerBunteBall]

It seems it's just solvable by having a Windows box to understand what's going on in detail.

The application in general is just tested on Linux. So I think it would need a general check up for Windows.

[DerBunteBall]

If you are able you could try to restructure the function so that you send a Ctrl-C Event.

https://docs.python.org/3/library/os.html

If my above thesis is right the processes are attached so every ffmpeg session has a seperate console window. This would explain that all other types of signals goes to nowehre or killing the process hard.

lambda: os.kill(ff.process.pid, signal.CTRL_C_EVENT) if sys.platform == "win32" \
        else ff.process.terminate()

Written out of the hand could be wrong.

You need to import os at top.

import os

[eldepor]

If you are able you could try to restructure the function so that you send a Ctrl-C Event.

https://docs.python.org/3/library/os.html

If my above thesis is right the processes are attached so every ffmpeg session has a seperate console window. This would explain that all other types of signals goes to nowehre or killing the process hard.

lambda: os.kill(ff.process.pid, signal.CTRL_C_EVENT) if sys.platform == "win32" \
        else ff.process.terminate()

Written out of the hand could be wrong.

You need to import os at top.

import os

Tried it without luck. It doesn't stop the recording :(

[DerBunteBall]

Ok. If you try to close the ffmpeg cmd like you know do by clicking x does ctrl-c has an effect?

We need to keep the following in mind: FFmpy doesn't set shell to True. So normally I would expect that the process is run directly. If we have a cmd that handels ffmpeg sperately ff.process.pid contains the pid of the cmd. So we sent Ctrl-C to the cmd not to ffmpeg. So we would need to send something to cmd that leads cmd to send SIGBREAK to ffmpeg. We know that clicking x sends SIGBREAK and that keeps clean files.

[lossless1024]

Please try taskkill /PID <pid of ffmpeg> and let's see. According to this post it sends SIGINT. The subprocess API is providing the PID of the process, maybe this could be the solution.

[DerBunteBall]

It depends which PID it provides. The documentation says.

The process ID of the child process:

Note that if you set the shell argument to True, this is the process ID of the spawned shell.

I actually think we are fire around ffmpeg. FFmpy doesn't change Shell attribute but I have the feeling for some reason ffmpeg starts in a new cmd (shell attached).

So when we were able to send a signal it's send to cmd.exe not ffmpeg.exe. So the only reason why this has no effect could be that it is not focused. Windows differences between attached and detached stuff. Also it seems that it's relevant whether cmd.exe is focused. (https://learn.microsoft.com/en-us/windows/console/ctrl-c-and-ctrl-break-signals)

So it would be necessary to figure out the child of the cmd.exe or focus cmd.exe and send Ctrl-C again. SIGINT had no effect by using send_signal() and gave an zmq error (see above). But .send_signal() seems to be very limited on Windows.

But it's helpful to verify whether ffmpeg stops correctly with SIGINT. But I have no doubts that ffmpeg handels SIGINT, SIGBREAK and SIGTERM correctly when they are correctly send.

[eldepor]

Please try taskkill /PID <pid of ffmpeg> and let's see.

Where have I to try that? I don't understand sorry

It depends which PID it provides. The documentation says.

The process ID of the child process:

Note that if you set the shell argument to True, this is the process ID of the spawned shell.

I actually think we are fire around ffmpeg. FFmpy doesn't change Shell attribute but I have the feeling for some reason ffmpeg starts in a new cmd (shell attached).

So when we were able to send a signal it's send to cmd.exe not ffmpeg.exe. So the only reason why this has no effect could be that it is not focused. Windows differences between attached and detached stuff. Also it seems that it's relevant whether cmd.exe is focused. (https://learn.microsoft.com/en-us/windows/console/ctrl-c-and-ctrl-break-signals)

So it would be necessary to figure out the child of the cmd.exe or focus cmd.exe and send Ctrl-C again. SIGINT had no effect by using send_signal() and gave an zmq error (see above). But .send_signal() seems to be very limited on Windows.

But it's helpful to verify whether ffmpeg stops correctly with SIGINT. But I have no doubts that ffmpeg handels SIGINT, SIGBREAK and SIGTERM correctly when they are correctly send.

This is the processes jerarchy:

One thing I noticed is that when I close the cmd window by clicking the X at the top right corner, it takes a few seconds while all the child ffmpeg.exe processes are killed (giving a playable file) but looks like something prevents the window to fully close and I have to click X again. But meanwhile new ffmpeg processes are open and recording the webcams but without the cmd.exe and python parent processes lol, so you have to manually kill every ffmpeg process before executing Downloader.py again:

[lossless1024]

According to the picture you have to open a cmd and there you type the command with the PID number of one ffmpeg process. For example taskkill /PID 6424. Then you should see which streamer got stopped from the downloader window. (If it really did)

[eldepor]

According to the picture you have to open a cmd and there you type the command with the PID number of one ffmpeg process. For example taskkill /PID 6424. Then you should see which streamer got stopped from the downloader window. (If it really did)

Im using ProcessHacker so I can see what streamer is recording each ffmpeg.

I opened a separate cmd, chose an ffmpeg.exe, used taskkill /PID to kill it but it said I had to use /F to kill it. So I killed it with /F and got this output in the downloader, with exit code 1 and video file broken.

[eldepor]

By the way, if you start downloading a live stream with YouTube-dl (youtube-dl.exe http://www.ch*turbate.com/Streamer) it generates a .part file, and if you interrupt it with Ctrl-C the download stops and the file cleanly turns into a playable .mp4 file

The output is this:

Exiting normally, received signal 2. [ffmpeg] Interrupted by user [ffmpeg] Downloaded 6503361 bytes [download] 100% of 6.20MiB in 00:08

I don't know how it works but maybe can help you

[lossless1024]

It seems that youtube-dl simply uses process.kill()

[eldepor]

Any luck with the ffmpeg cleanly stop issue? Process kill still leaving a corrupt video file.

[DerBunteBall]

Found the solution I think:

1. There is no simple way to gracefully stop a command line process on Windows. For ffmpeg press q works as well on Linux and Winows (PowerShell) for manually started processes. On Linux .communicate(b"q") works without problems (alwo macOS). I found no way to send this on Windows actually.
2. A graceful stop makes it possible to do something like "mp4 copying". Actually a simple copy command is used. That seems to be a bad idea in general.
3. youtube-dl and yt-dlp use a two step approach for all platforms. The keyword is MPEG-TS (https://en.wikipedia.org/wiki/MPEG_transport_stream). You can recognize different behaviour on Linux and Windows shown by yt-dlp. A graceful stop leads to the fact that everything seems to be done automatically. When ffmpeg is killed like on Windows - .kill() is a alias for .terminate() - they do a fix. So they don't write by copying classically they write a MPEG-TS in comamnds:

Streamonitor uses this:

ffmpeg -i hls_link -headers blub -c:a copy -c:v copy "file:blab.mp4"

yt-dlp writes a file with plain mpegts like this:

ffmpeg -i hls_link -c copy -f mpegts "file:blab.part.mp4"

The second step is to make a working MPEG4 file:

ffmpeg -i "file:blab.part.mp4" -map 0 -dn -ignore_unknown -c copy -f mp4 "-bsf:a" aac_adtstoasc -movflags "+faststart" "file:blub.mp4"

The two step approach leads to a working file on Windows by simply killing ffmpeg. They do a check for the fix in the code but the fix seems to be not needed always on Linux and macOS because the graceful stop seems to have a handling for this. So also when this fails it should be the same result on Linux/macOS so it seems to be the right to do the two step approach on all platforms.

So the problem isn't the signaling isolated but the stuff which gets written.

I have written a dirty script which does the swo step approach with ffmpy. On Windows it leads to working Chaturbate rips.

https://gist.github.com/DerBunteBall/374eea6d12cd4d9632f5b374bea2e7d3

python.exe ffmpeg.py model_name

when model is online ffmpeg starts and runs until chanceled by Ctrl-C then the second run is donw. There should be two files in the name format of Streamonitor one with .part.mp4 and one with .mp4. The .mp4 is the end result working in VLC.

Tested on Windows 10 64 bit. pyenv-win with pyenv-win-venv in a Python 3.10.8 env with ffmpy and requests installed via pip and ffmpeg 5.1.2 installed via Chocolatey (gyan.dev build it seems).

Note: Script has no error handling and is really silly. Imports are more then needed. If someone finds a way to send a "q" key press safely to ffmpeg on Win32 this would be also helpful.

Hope this helps someone.

Best Regards

[eldepor]

Hope that solves the issue. What I noticed is that when you record a streamer with YouTube-dl, it writes a .mp4.part file, then when you Ctrl-C the recording stops and if the file is too big it takes a bit to convert that to a clean .mp4, so some process must be behind that yes.

[DerBunteBall]

Check my above posted gist: https://gist.github.com/DerBunteBall/374eea6d12cd4d9632f5b374bea2e7d3

Run python.exe ffmpeg.py model_name with a Chaturbate model that is online. Let it run for 5 minutes and press Ctrl-C you get a part file and a mp4 file the mp4 is playable. ffmpeg is hardly killed in this approach. But file works. Thanks to MPEG-TS. The part file will not be playable but you can reproduce the mp4 file manually.

[eldepor]

I downloaded it, placed it in a new folder, but nothing happens. Am I doing something wrong?

[DerBunteBall]

Make sure the mdel is online. Actually e.g. princess_moana

If the file is in C:\test

Do this after opening PowerShell:

cd C:\test\
python.exe ffmpeg.py princess_moana

Press Ctrl-C after a while then in C:\test the files hould be.

If a model is offline the script outputs nothing. It has no error handling actually.

[eldepor]

Sorry but not working :(. It just does nothing. It just puts me in a new line to write again. Maybe I have something not installed.

[DerBunteBall]

Sorry my fault.

I don't copied the main() call xD. Redownload the file it should now work.

[eldepor]

Now it works. But its funny because part.mp4 file is playable too wtf. Even without stopping the recording, but for example it lacks of thumbnail, I guess because of the mpegts format.

I commented lines 22, 31 & 32 of your script and modified line 23 changing -f mpegts to -f mp4. Recorded a streamer and stopped it with Ctrl-C, the result is the single playable mp4 (with thumbnail) we looking for.

Then I tried replicating that by adding -f mp4 in line 11 of streamonitor's ffmpeg.py:

outputs={filename: '-c:a copy -c:v copy -f mp4'})

but when I stop the recording with stop command, the file is corrupted. It only works on your script stopping with Ctrl-C.

Then I modified ffmpeg.py again but with -f mpegts instead of -f mp4

outputs={filename: '-c:a copy -c:v copy -f mpegts'})

and now yes, you can stop any recording and have a playable "mp4" file (really an mpegts right?) but as said above the file has no thumbnail like an mp4.

So I'm stuck there. Wouldn't be possible to do everything in one step? When you record a stream with YouTube-dl and stop it, the part file automatically turns into mp4. RecurbateDownloader for example writes directly on .ts and then you have to execute a "fix" script to have an mp4 file (two steps). Maybe this can be automated in streamonitor, deleting the .ts file after.

[eldepor]

Have you checked this?

https://stackoverflow.com/a/63660293

[DerBunteBall]

I know these effects.

The gist is just a proof of the fact that it is needed.

MPEG-TS is at first a protocol and at second a container format. A special way to stream MPEG with more error correction. Keep in mind that mostly we have HLS Streams. That's not to confuse with MPEG-TS. So the first step in fact writes a HLS stream to a .ts Container File (MPEG-TS Container). These files can be playable by VLC or so. In fact it's like a file written with error correction in itself. So HLS data which is gotten (pieces over HTTP) get written into a file which is like a dump of a MPEG-TS stream. In fact the pieces are already in that format. So a real convertion doesn't happen.

In result you wan't to have a clean .mp4 file. So the gracefull stop of ffmpeg seems to do this by in-place modifying the written file. So e.g. if ffmpeg is closed gracefully you should have a .mp4 file (name extensions are important). If you kill ffmpeg hard this can't be done. So you should have a .ts (MPEG-TS Container) file. The yt-dlp check is triggered by this fact. So it's triggered on Windows because ffmpeg always is stopped hardly. On Unix/Linux ffmpeg get's normally shut down gracefully so the fix isn't triggered but it could be triggered on Linux/Unix when ffmpeg is stop with SIGKILL. They just check the format the file tells it self it is. If it's a .ts (MPEG-TS container) they convert to real .mp4.

Writing HLS to MP4 directly seems to lead always to corrupted files unless no gracefull stop happens or the stream ends regulary. Writing HLS to MPEG-TS makes it unimportant how ffmpeg is stopped. Because of the concepts behind MPEG-TS.

The thumbnail thing should be related to Explorer. I think explorer renders a default thumbnail for a real .mp4 file. It wouldn't for .ts (MPEG-TS Container).

Of course that .part.mp4 is used by yt-dlp by default I think ffmpeg behaves in the gracefull stop so that it decides to make .ts to .mp4. But there are a few different ways for Extractors how they can trigger download and post processing. So this count's just where it uses ffmpeg as Downloader and Post Processor. There are other ways HLS (m3u8) can be downloaded by yt-dlp.

It's not possible to have this in one command. That's the difference between hard process kills and gracefull process kills. Gracefull process kills makes it possible to do things like fixing files in inconsistent state and so on.

It could be also possible to bring ffmpeg down gracefull in Python on Windows. But I think it's only possible by coding directly with Win32 API. So It's not possible to bring command line tools down in Windows by signaling cleanly. Windows also does know things like Message Queues. But process needs to be started with them or they need to be attached. As far as documentation states subprocess isn't able to do this. Something like process groups are possible but not message queues. But I think it could be possible to handle this with the Win32 API Bindings for Python in the hardest case with ctypes. It also needs to check how ffmpeg in fact reacts in Windows generally. So a bit C Code Reading would be needed to make this sure.

[DerBunteBall]

I will make it short.

I have working stuff on all platforms. So same stopping method for Linux/Unix/Win32 (own scripts).

I recommand stopping ffmpeg by sending it a "q".

So stopDownload should be like this.

self.stopDownload = lambda: ff.process.communicate(b"q")

ffmpy captures stdin to PIPE to support pipe protocol. That's what is needed to handle this. Ffmpeg closes cleanly by getting a "q".

Process and Thread Model on Unix/Linux and Win32 are very different. The above makes it easy to stop ffmpeg on all platforms cleanly (other ways are also available).

ffmpeg has a more or less complex cleanup function (ffmpeg_cleanup). It is triggered in a few different ways in the ffmpeg source code (signals, events and keyboard interaction). It makes the files clean and leaves the prcoess safely. ffmpeg supports Windows Console Events CTRL_C_EVENT and BREAK_EVENT. Events are a bit special. To send them you need to create a Process Group and send the Event to this Group. A CMD does this in fact creating a process group.

ffmpeg_cleanup has effects to files. It seems that ffmpeg writes formats different. MPEG-TS files are always "valid". It's totally unimportant how ffmpeg shutsdown they seem to look the same always. So checking files that are written with -f mpegts give always data on a file command and give back the same info by running mediainfo on them. That's really different with MP4 files. While file detects them as MP4 files with the same result you see a big difference with mediainfo. If ffmpeg is shutdown hard they look like they have no header. So mediainfo can't detect much information on them. While a file created by a gracefully stopped ffmpeg process looks regular. ffmpeg_cleanup sanitizes this. I don't digged into the deep details of the libav stack but it seems that some formats are written differently or handled differently by players. So it seems that the infos MPEG-TS files provide are enough to play them while this doesn't is true for MP4. I don't know whether these files are repairable by tools.

Short about this:

Have you checked this?

https://stackoverflow.com/a/63660293

This does the send ffmpeg a "q" thing also.

You can install choosenim by Chocolatey on Windows. Install nim stable and build the thing. If you have no main Python installation on Windows keep in mind that nim needs PYTHONHOME set to your virtual environment. If you try to start the wrapper without that it will fail to import stuff. It can be needed to copy the core DLL to the search paths also. The wrapper creates a trivial window (only as a drawn frame) which starts ffmpeg as a command in it (with Python subprocess). When this window get's a WM_CLOSE message (that's a way of process intercom on Windows) it sends a q to captured stdin (subprocess.PIPE). Which leads to a clean shutdown. So it does the same as mentioned above by wrapping Python into nim. Which generally is not needed because a seperate window isn't needed. Popen Objects will bring the q to the correct place (also on Windows as far as I could find out). It has the correct PID and seems to be correctly attached to stdin on all platforms.

Eventually people using StreaMonitor on Windows could test this. For me topic is completed for now.

Best Regards

[eldepor]

I recommand stopping ffmpeg by sending it a "q".

So stopDownload should be like this.

self.stopDownload = ff.process.communicate(b"q")

Tried it but got these errors right after running the Downloader.py, when it detects an online streamer and starts recording:

[lossless1024]

This way it is being run when the program gets on the line, but at the moment it hasn't been opened yet. Define it as a function: self.stopDownload = lambda: ff.process.communicate(b"q") Try it please

[eldepor]

This way it is being run when the program gets on the line, but at the moment it hasn't been opened yet. Define it as a function: self.stopDownload = lambda: ff.process.communicate(b"q") Try it please

ffmpeg.py line 25:

self.stopDownload = lambda: ff.process.communicate(b"q") if sys.platform == "win32" else ff.process.terminate

output when stopping a streamer with stop command:

ffmpeg.exe version, placed in StreaMonitor-master folder:

D:\CAM4\StreaMonitor-master>ffmpeg -version

ffmpeg version 4.4-full_build-www.gyan.dev Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.2.0 (Rev6, Built by MSYS2 project)
configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontcon
fig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-lzma --enable-libsnappy --enable-zlib --enable
-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-
sdl2 --enable-libdav1d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-
libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-libass --enable-frei0r --enable-l
ibfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --e
nable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-li
bglslang --enable-vulkan --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enab
le-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwben
c --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --en
able-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chrom
aprint
libavutil      56. 70.100 / 56. 70.100
libavcodec     58.134.100 / 58.134.100
libavformat    58. 76.100 / 58. 76.100
libavdevice    58. 13.100 / 58. 13.100
libavfilter     7.110.100 /  7.110.100
libswscale      5.  9.100 /  5.  9.100
libswresample   3.  9.100 /  3.  9.100
libpostproc    55.  9.100 / 55.  9.100

[lossless1024]

Sad :(

[DerBunteBall]

Whoops.

Forgot to make it an anonymous function.

So this tells us that stdin is closed or it thinks it is. I will investigate it.

ffmpy starts the ffmpeg process like this:

subprocess.Popen(
                self._cmd, stdin=subprocess.PIPE, stdout=stdout, stderr=stderr, env=env
            )

So it catches ffmpeg's stdin. So I think it could be related to the threaded design.

Doing the same with Popen plain works always.

In StreaMonitor the execute function runs in a thread. So I think it's possible that stdin isn't reachable or open because of that. So the question is how threads handle stdin and how stdin is handled when a thread spawns a subprocess.

Best Regards

[DerBunteBall]

I have working code that handles sending q to a subprocess of a thread on all platforms.

I think it's related to the CLIManager or ZMQ. I think it's possible they are doing something with stdin which leads to strange side effects.

Will work on a proof of concept for doing it without sending q on all platforms.

[DerBunteBall]

Have working code which brings down ffmpeg graceful by using Windows Messaging without showing something up. Should be possible to implement this for a pool of processes. And make sending q also workable with this.

I needed to code against Win32 API for this. Also it's a subprocess spawn from a Thread. So it would need seperate logic but work in general for the StreaMonitor design.

In Short:

1. Start a process with a sperate window which is hidden.
2. Get the Window Handle for this Window
3. Send it a WM_CLOSE message when it should be stopped.

Leads to working files because the Window (in fact cmd terminal emulator) sends CTRL_BREAK_EVENT in this situation. I couldn't get this working with os.kill actually. Will investigate this also. It's somehow strange.

Next step will be to find out why q doesn't work in StreaMonitor. I think it's the CLIManager or the blocking .run() method of ffmpy which calls .communicate() which could blcok stdin.

Best Regards