davidhwyllie
commented
2 years ago Options:
- add an option not to stop catwalk to the fn4_shutdown script. This is the easiest and best option if catwalk does not need to be shutdown (if for example there is not update to catwalk needed)
- rapidly load catwalk over http from some kind of sequence backup file, allowing faster access than using a distant database. Timings need to be worked on. Experiments done by @dvolk suggest that there's little difference in loading samples one at a time vs. en masse over a rest API; an experimental endpoint to allow bulk loading wasn't implemented because it didn't improve load speeds.
- generate fasta files suitable for catwalk bulk loading. Note, such files have to be on the machine running catwalk. In benchmarks with covid data, loading one sample took about 0.007 seconds.
[this is for TB: for covid, loading 1m samples took < 2mins] - use catwalk's own mechanisms to store data and restart. At present, such data is stored in the same directory as the catwalk client (not desirable) and there is a risk of mis-synchronisation with the contents of fn4 with this option.
Other options: build a periodic dump into catwalk cf. https://nim-lang.org/docs/marshal.html but this will block the server and is probably undesirable.
These options are not mutually exclusive.
Of these options: 1 should be implemented 4 should probably not be implemented because of the risk of loading data not in the fn4 database silently Either 2 or 3 would be satisfactory, but will still leave a multi-hour load time
- in theory option 2 might be faster, because reference compression isn't required
It would be possible to make the fn4 server restart rapidly for READING data (which doesn't need catwalk) and later for inserting data. However, this would need careful implementation.
Restarting the findneighbour4 server is slow when large numbers of samples are present. This is due to repopulation of the catwalk component from the database. We need to identify ways to reduce this.