Closed malonzo47 closed 4 years ago
This might be an issue of block size again. If the job isn't huge, only one worker is assigned... or something like that. Still, seems inefficient if more processing resources are available.
I checked and setting chunksize parameters does not make much sense. If we do not specify that chunksize, the system will allocate properly according to the number of workers, won't it? After removing the chunksize parameter settings, I see the resources are being exhausted. Will close this issue and push a new release that does not have specific chunksize parameter for workers.
If I try to use 8 processors, only 1 seems to be doing anything. Only true for optimization in Step 2.