Old implementation used pandas to load big dataframes, do QC, and append to a gzipped file
Tried multiprocessing #61 but the performance benefits weren't great
New implementation sets up a data processing pipeline with a pile of generator functions
The end result streams variants to an output file, so memory usage will be low
setting up ScoreVariant class with __slots__ and __iter__ really helped (was using a dict)
Also fixes some bugs that cropped up:
pandas introduced effect weight precision errors
some variants lost information in weird conditions
dropped --threads because it didn't help
Benchmark with first 100 files in PGS Catalog with compressed output:
combine_scorefiles -s PGS000001_hmPOS_GRCh38.txt.gz 262.74s user 1.44s system 99% cpu 4:24.99 total
combine_scorefiles -s PGS000001_hmPOS_GRCh38.txt.gz 278.85s user 16.70s system 57% cpu 8:37.21 total
ScoreVariant
class with__slots__
and__iter__
really helped (was using a dict)Also fixes some bugs that cropped up:
--threads
because it didn't helpBenchmark with first 100 files in PGS Catalog with compressed output: