Increasing resource allocation for biocache downloads service (currently only 2 CPU machine)
Investigate performance of CSV library in use - currently au.com.bytecode.opencsv.CSVWriter see code
Originally from Dave
The CPU usage on the downloads server is high when running a download.
This is the case in production as well, but there is a slight increase. Note - this isn't affecting the stability of SOLR cluster and so doesn't impact other services.
Things to investigate:
Field mapping optimizations
Use of CSV extracts in streams (as opposed to Tuples) - may reduce GC.
@djtfmartin commented on Mon Jun 14 2021
Areas of interest:
au.com.bytecode.opencsv.CSVWriter
see codeOriginally from Dave
@djtfmartin commented on Wed Jun 16 2021
Discussed with @adam-collins
Areas of interest:
au.com.bytecode.opencsv.CSVWriter
see code@adam-collins commented on Tue Jul 27 2021
Include simplification of download queue and threads.