Now I'm not sure how this works and am skeptical that there might be too much overhead to get similar speedup that parallel-command-processor would (grosser/parallel did not work well for when having many jobs because of the overhead).
The problem is when you have a job like alignments - you have 102268 alignments and 28 or 48 cores, so you want a job queue of 102268 jobs that 28 or 48 cores pull off the queue one at a time and work on it. And you want that with the least overhead and least complexity code maintenance wise as possible.
Just executing a separate rake task, with the entire Rails environment loading everytime, is a lot of overhead for this (that alone can end up taking 6-12 seconds, though you might be able to speed it up with some investigation).
Ruby 3 also has ractors but that is supposedly an "experimental feature" so I did not explore that further.
See https://github.com/grosser/parallel_tests
Now I'm not sure how this works and am skeptical that there might be too much overhead to get similar speedup that parallel-command-processor would (grosser/parallel did not work well for when having many jobs because of the overhead).
The problem is when you have a job like alignments - you have 102268 alignments and 28 or 48 cores, so you want a job queue of 102268 jobs that 28 or 48 cores pull off the queue one at a time and work on it. And you want that with the least overhead and least complexity code maintenance wise as possible.
Just executing a separate rake task, with the entire Rails environment loading everytime, is a lot of overhead for this (that alone can end up taking 6-12 seconds, though you might be able to speed it up with some investigation).
Ruby 3 also has ractors but that is supposedly an "experimental feature" so I did not explore that further.