Improve performance of download service

[djtfmartin]

@djtfmartin commented on Mon Jun 14 2021

Areas of interest:

• Threading and queuing of downloads
• Increasing resource allocation for biocache downloads service (currently only 2 CPU machine)
• Investigate performance of CSV library in use - currently au.com.bytecode.opencsv.CSVWriter see code

Originally from Dave

The CPU usage on the downloads server is high when running a download. This is the case in production as well, but there is a slight increase. Note - this isn't affecting the stability of SOLR cluster and so doesn't impact other services.

Things to investigate:

• Field mapping optimizations
• Use of CSV extracts in streams (as opposed to Tuples) - may reduce GC.

---

@djtfmartin commented on Wed Jun 16 2021

Discussed with @adam-collins

Areas of interest:

• Threading of downloads
• Increasing resource allocation for biocache downloads service (currently only 2 CPU machine)
• Investigate performance of CSV library in use - currently au.com.bytecode.opencsv.CSVWriter see code

---

@adam-collins commented on Tue Jul 27 2021

Include simplification of download queue and threads.

[adam-collins]

done