Open Mrw33554432 opened 5 months ago
CarlGao4
commented
5 months ago What if you click on the table header “status”? As I explained in the documentation, the progress bar shown in the table will lose its animation by default. But I manually enabled it by force repainting every 1/25 seconds. I realized that this will consume a lot of CPU time, so I then gave the user a choice to disable it by clicking on the table header
Mrw33554432
commented
5 months ago What if you click on the table header “status”? As I explained in the documentation, the progress bar shown in the table will lose its animation by default. But I manually enabled it by force repainting every 1/25 seconds. I realized that this will consume a lot of CPU time, so I then gave the user a choice to disable it by clicking on the table header
Here is what I observed after click on Status:
CarlGao4
commented
5 months ago So this means that you didn't enable the animation before I think I need to find out how Qt6 deal with large tables
CarlGao4
commented
5 months ago Can you try pausing when separating a file and see CPU usage? I’m using my phone and can’t test it myself
Mrw33554432
commented
5 months ago Can you try pausing when separating a file and see CPU usage? I’m using my phone and can’t test it myself
pause -> CPU 1%
CarlGao4
commented
5 months ago And clicking on the header when paused?
Mrw33554432
commented
5 months ago And clicking on the header when paused?
6% (no white light animation, just green bar)
It's a little tricky as enabling the animation result in a higher GPU occupacy (better). I will also test a little bit about file language issue since the GUI worked fine on another dataset in different filename language.
CarlGao4
commented
5 months ago I guess this is related to QTableWidget but I can’t prove it now It will create a widget for every table item, so if you add 10k+ files it will construct 30k+ (10k for vertical header, 10k for file name, 10k for progress bar) But fixing this requires rewriting the whole program as I’m actually passing the table item widgets as function arguments to update UI
Mrw33554432
commented
5 months ago I guess this is related to QTableWidget but I can’t prove it now It will create a widget for every table item, so if you add 10k+ files it will construct 30k+ (10k for vertical header, 10k for file name, 10k for progress bar) But fixing this requires rewriting the whole program as I’m actually passing the table item widgets as function arguments to update UI
what if we cut the files passed to the GUI into chunks of 10 - 50 files, and working on each file chunk one by one? That might be a little easier to implement since it doesn't change the UI but file handling XD.
CarlGao4
commented
5 months ago Sorry but I didn't quite understand what you mean
Mrw33554432
commented
5 months ago Sorry but I didn't quite understand what you mean
Actually It is quite simple.
Instead of 10000 files -> GUI
we do 50 files -> GUI ... 50 files -> GUI x 200
Just adding a second level queue (only display numbers, store in a python list or queue）would solve this issue. Now the GUI will only display up to 50 (n, you can determine) files that it is currently processing on, and the number of files that has been processed and waiting to be processed (basically fold them up). like
1000 files done!
file 1 - process bar
file 2 - process bar
file 3 - process bar
9000 files in queueAnd for users who pass less than 50 (n) files to GUI, there won't be any change
It is mostly a patch-like approach
Mrw33554432
commented
5 months ago New Update:
The problem can be partially solved by selecting any tab other than File queue (xxx).
For example, if you switch to the Mixer tab or Select Model tab, the low GPU usage issue will be disappear.
But the high CPU usage issue remains.
CarlGao4
commented
5 months ago That's strange. I remember that Qt renders its window CPU-only, while separation and window rendering is in different threads, so theoretically they won't affect each other...
Mrw33554432
commented
5 months ago That's strange. I remember that Qt renders its window CPU-only, while separation and window rendering is in different threads, so theoretically they won't affect each other...
Here is a quick analysis from AI, actually I partially agree with it. Instead of having the GPU thread to send signal to window thread, it might be better for the window thread to request status on GPU thread, or having another one in between that is guaranteed to be non-blocking.
I have not been working on thread programming for a while, but I remember a trick you can try
class Separator
...
self.processbar: float = 1.1 #example
and just access this param from window thread, which should be non-blocking in most cases.
(Below From AI)
Based on the provided code, it appears that the separation process and the GUI are already running on separate threads. The startSeparate method in the Separator class is decorated with @shared.thread_wrapper(daemon=True), indicating that it runs on a separate daemon thread.
However, there are a few places where the GUI thread may potentially block the GPU processing thread:
In the Separator.separate method, there are calls to setStatus, setModelProgress, and setAudioProgress. These methods emit signals that are connected to slots in the GUI thread. If the GUI thread is busy or unresponsive, it may delay the processing of these signals, potentially blocking the separation thread.
The Separator.separate method also calls the save_callback function, which is passed as an argument and is responsible for saving the separated audio files. If the save_callback function performs any blocking operations or interacts with the GUI thread, it could potentially block the separation thread.
The Separator.updateProgress method is called frequently during the separation process to update the progress information. This method emits signals to update the GUI elements. If the GUI thread is busy or unresponsive, it may delay the processing of these signals, potentially blocking the separation thread.
To minimize the chances of the GUI thread blocking the GPU processing thread, you can consider the following:
Ensure that the GUI thread remains responsive and avoids long-running or blocking operations. Optimize the rendering and updating logic in the GUI, especially in the file queue handling code, to reduce the chances of the GUI thread becoming unresponsive.
Review the save_callback function and any other callbacks passed to the separation thread. Make sure they don't perform any blocking operations or interact with the GUI thread directly. If necessary, use separate threads or asynchronous programming techniques to handle file saving and other operations.
Consider using a separate thread or a thread pool for handling the progress updates and GUI element updates. Instead of directly emitting signals from the separation thread, you can queue the updates and process them on a dedicated thread. This can help decouple the separation thread from the GUI thread and reduce the chances of blocking.
If the separation process is still being blocked by the GUI thread, you may need to investigate further and profile the application to identify specific bottlenecks or areas where the threads are interacting in a way that causes blocking.
It's important to note that the provided code snippets don't include the complete implementation of all the methods and classes involved, so there might be other places where the GUI thread and the separation thread interact that are not visible in the given code.
To fully diagnose and resolve the issue, it would be helpful to have access to the complete codebase and perform thorough testing and profiling to identify the specific points where the GUI thread may be blocking the GPU processing thread.
CarlGao4
commented
5 months ago Updating progress or UI must be done in the main thread, and I made sure that during this process, the thread called those functions must be blocked so the main thread won't be queued with a lot of callbacks... Besides, when I first wrote this program, I thought that there would only be very few people who would load thousands of files at the same time... So I even didn't create a list to store the file queue, but read the table and check the progress bar status one by one from the top of it, because this would be much easier and the effect can be really small if we only add no more than about 1000 items Currently I'll only suggest users to switch to other tabs to avoid that
Mrw33554432
commented
5 months ago Updating progress or UI must be done in the main thread, and I made sure that during this process, the thread called those functions must be blocked so the main thread won't be queued with a lot of callbacks... Besides, when I first wrote this program, I thought that there would only be very few people who would load thousands of files at the same time... So I even didn't create a list to store the file queue, but read the table and check the progress bar status one by one from the top of it, because this would be much easier and the effect can be really small if we only add no more than about 1000 items Currently I'll only suggest users to switch to other tabs to avoid that
Yeah I agree. Actually the best way to handle lot of audio files is using a custom Dataset object and manually handle batch in -> batch out -> save to audio, but the Demucs project is a little bit too complex and containing many code integrations, which make the function a little bit hard to implement. And as it will only take about 30% speed up but requires lot of time to implement, I will just stick with GUI for now XD (the Cli is even a little slower than GUI)
CarlGao4
commented
5 months ago I'll consider adding a warning in the next version when the user adds too many files to the queue (like the first time it reaches 500) and tell what to do The most basic cause of this is just I thought nobody would add thousands of files at the same time
ferropop
commented
3 months ago Lots of us out here ripping huge libraries! Any efforts to mitigate the slowdown would be ever appreciated!
happen
It happens during the whole separation process (not just loading and saving phrase) It' very likely caused by the gui I am highly suspicious on the queue management as the performance get worse when queue is larger - the degradation is almost linear
What is the platform you are using?
Windows
log