search

VEuPathDB / service-eda-compute

Apache License 2.0
0 stars 0 forks source link

Use parallel stream to look up job attributes (independent operations) #58

Closed ryanrdoherty closed 1 year ago

ryanrdoherty commented 1 year ago

The EDA compute expiration endpoint on prod (south) takes >2 mins (3.5 now but will grow) to come back, causing a 502 proxy error response rather than a 200 with some JSON telling how many jobs were expired. I'm not sure this will fix this, but will speed it up a little at least. Asking for PR because I've never used parallel streams before and wondered if you all knew of any gotchas. Also Ellie, maybe make sure the stuff I'm doing in there is threadsafe?

dmgaldi commented 1 year ago

One gotcha I've run into in the past is that the default ForkJoinPoint pool is used by default and the total number of threads is determined by the JVM. I think it's usually set to Runtime.getRuntime().availableProcessors() + 1 and I'm not sure what that will be in docker.

For I/O bound tasks the default is usually way too low, but for cpu-bound tasks it's probably fine.

ryanrdoherty commented 1 year ago

For I/O bound tasks the default is usually way too low, but for cpu-bound tasks it's probably fine.

Yeah, these are taking a long time because for each job reference we potentially hit postgres for job status, then also read the config file out of minio and check some values. There are about 650 jobs, and it took ~3.5 mins to expire them all so that's ~1/3 second each. Not too bad I guess but I feel like we could do better.

Gonna try the custom threadpool with.... 10? threads and see how it does.

ryanrdoherty commented 1 year ago

@dmgaldi See the new code for custom threadpool. I also ran a benchmark to see how much parallel could help and how much more a custom pool (10 threads) could help and got this:

Sequential: processed 600 in 3:30.115
Parallel: processed 600 in 26.613 seconds
Parallel (custom): processed 600 in 22.782 seconds

I feel like the real win with the custom pool is to not interfere with anyone else using the system pool though, so I'd like to keep it despite only marginal (by comparison) performance improvement.