Multi-thread jobs?

[squaregoldfish]

Most of the jobs we run should be easy to parallelise, at least in theory - they run on individual records that don't depend on each other. Explore this to see if it's a good way to speed up jobs.

This is probably better than having a multi-threaded job pool, since the odds of having overlapping jobs from different sources are minimal (except perhaps when processing NRT datasets on a cron schedule)

This may not be effective if the database activity is the major part of the job time. That's a whole different optimisation problem.

[squaregoldfish]

Maybe make use of the parallel streaming features of Java 8?

[squaregoldfish]

Take special care with routines that compare between multiple values.

[squaregoldfish]

Will include general performance improvements of data processing jobs in this issue.

[squaregoldfish]

ExtractDatasetJob

Database-bound

AutoQCJob

database-bound

LocateMeasurementsJob

Mostly database-bound, but there's a small improvement we can make in the code

DataReductionJob

Looks like there's some improvements to make here - mostly improving the calibration algorithm

DataReductionQCJob

Possibly some work we can do here.

[squaregoldfish]

Check the DataReductionJob for other sensors too, but after the calibration is sorted out.