Refactor combine_scorefiles - Githubissues

PGScatalog / pgscatalog_utils

(superseded by pygscatalog) Utilities for working with PGS Catalog API and scoring files

Apache License 2.0

4 stars 3 forks source link

Refactor combine_scorefiles #62

Closed nebfield closed 8 months ago

nebfield commented 9 months ago

Old implementation used pandas to load big dataframes, do QC, and append to a gzipped file
Tried multiprocessing #61 but the performance benefits weren't great
New implementation sets up a data processing pipeline with a pile of generator functions
- The end result streams variants to an output file, so memory usage will be low
setting up ScoreVariant class with __slots__ and __iter__ really helped (was using a dict)

Also fixes some bugs that cropped up:

pandas introduced effect weight precision errors
some variants lost information in weird conditions
- dropped --threads because it didn't help

Benchmark with first 100 files in PGS Catalog with compressed output:

combine_scorefiles -s PGS000001_hmPOS_GRCh38.txt.gz                            262.74s user 1.44s system 99% cpu 4:24.99 total
combine_scorefiles -s PGS000001_hmPOS_GRCh38.txt.gz                            278.85s user 16.70s system 57% cpu 8:37.21 total

nebfield commented 9 months ago

correlation of effect weights is 1 between old and new despite the floating point error in the old version