Closed andreped closed 7 months ago
Move "import fast" into func, and it works. Not sure why this error happens, but understand that threads and processes are fundamentally different. Threads share memory, processes do not.
Just tried it on my end. Works fine with having the import fast
inside the func()
.
From a TensorFlow perpective I can understand that importing within the process separately has its value, as TF defines the session globally within the process. Hence, by importing TF in separate processes, you are able to completely clear the session when killing the subprocess. However, with TF, it still works and does not produce any errors or warnings.
When importing fast
in separate processes, we do however get lots of these bad boys (see below). I assume this is written to standard output, and thus, I cannot mute this easily or?
- Powered by -
_______ __________
/ __/ _ | / __/_ __/ https://fast.eriksmistad.no
/ _// __ |_\ \ / / v4.6.0
/_/ /_/ |_/___/ /_/
- Powered by -
_______ __________
/ __/ _ | / __/_ __/ https://fast.eriksmistad.no
/ _// __ |_\ \ / / v4.6.0
/_/ /_/ |_/___/ /_/
- Powered by -
_______ __________
/ __/ _ | / __/_ __/ https://fast.eriksmistad.no
/ _// __ |_\ \ / / v4.6.0
/_/ /_/ |_/___/ /_/
- Powered by -
_______ __________
/ __/ _ | / __/_ __/ https://fast.eriksmistad.no
/ _// __ |_\ \ / / v4.6.0
/_/ /_/ |_/___/ /_/
There is no way to mute this "splash" right now. I could add an environment variable to mute it. Otherwise you can redirect stdout of your multiprocessing, so it doesn't show in the console. But then you will not see error messages either.
Describe the bug
We have started to use pyFAST for streaming image patches from WSIs. The patch generator samples random patches from a WSI.
In TensorFlow, we have used multithreading to speed up patch generation, which is available through the
tf.data.Dataset
API, where the threading is performed in C++.However, it does not seem like PyTorch 1.x supports multithreading (may change in the upcoming major release of PyTorch 2.x). In PyTorch, it is therefore common to use multiprocessing instead, which TensorFlow also supports, but for both libraries we get the same issue.
When enabling
multiprocessing
and settingnum_workers > 1
inmodel.fit()
in TensorFlow, it raises aRuntimeError
. Same applies to settingnum_workers > 0
for PyTorch's DataLoader.I believe the core reason is that pyFAST does not work with Python multiprocessing which both TensorFlow and PyTorch use.
To Reproduce
I observe the same issue, if I try to read a patch from a WSI from separate processes (does it matter if I move the fast import inside the function?):
We had issues with this before, but as we could use multithreading in TF, we did not look more into the issue. However, this needs to be resolved, as there is no other way to stream patches in parallel in PyTorch.
System
Expected behavior pyFAST should be compatible with Python multiprocessing for both TensorFlow and PyTorch.
Logs
@markusdrange on cc.