app pipelines to speed up divergent commands

[KatherineCaley]

Known issues with this implementation:

• Fails on Windows when trying to call max on the processed seqs with default values. Error message is PermissionError: [Errno 13] Unable to synchronously open file This may or may not be related to the fact that individual methods in the DataStore class open and close the HDF5 file. Should it be rewritten so that the file is opened and closed in some other way?

Comments on usage of the dvgt CLI

in prep, the input data store that wraps the sequence data is written to a temp directory. This is because I didn't want to duplicate the input seq data. However, this feels like a bad idea... It can violate data provenance via the source attribute of DataMembers. There is probably a better way to handle this.

Usage of the apps

# data_store setup
# if fasta seqs in SINGLE FILE
convert2dstore = dvgt_seq_file_to_data_store()
in_seqs = convert2dstore("path/to/fasta/file") 

# if multiple fasta seqs in directory
in_seqs = DirectoryDataStore("path/to/directory/of/fasta/files") 

out_dstore = HDF5DataStore(source="path/to/write/prepped/seqs.dvgtseqs", limit=limit)

# prep 
prep_pipeline = dvgt_load_seqs(moltype="dna") + dvgt_write_prepped_seqs(out_dstore)
prepped_seqs = prep_pipeline.apply_to(in_seqs, show_progress=True, parallel=True)

# max 
max_app = dvgt_calc("max") 
max_app("path/to/dvgtseqs")
# table of max divergent seqs

@GavinHuttley, definitely needs much refinement and work . Happy to keep working on it or handover.

[GavinHuttley]

Fails on Windows when trying to call max ... Should it be rewritten so that the file is opened and closed in some other way?

Is the HDF5 file being opened in read-only mode? For now, mark the failing test as an xfail with the reason being "windows specific failure to be resolved". Then create an issue with the traceback.

I'll then merge into that specific branch and handover to @khiron. As @khiron has a Windows box, he can run simple experiments to see if this is HDF5 specific, perhaps it acquires a file lock even in read-only mode?

[KatherineCaley]

@GavinHuttley @khiron now passing on windows