update_cBioPortal_data.py merge_mafs uses too much memory

stevekm commented 3 years ago

The script update_cBioPortal_data.py is using large amounts of memory when merging Facets maf information into the data_mutations_extended.txt file.

Some ideas;

Currently, the entire Facets maf gets stored in memory; consider only storing a subset of the Facets maf data in memory, maybe just the key fields to merge on and the value fields desired (https://github.com/mskcc/helix_filters_01/blob/master/bin/update_cBioPortal_data.py#L319)
Consider loading the data into a SQLite file on disk temporarily so it can be queried so that it doesnt need to be held in memory the whole time (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html)
try using bash built-in join command such as shown here https://unix.stackexchange.com/questions/113898/how-to-merge-two-files-based-on-the-matching-of-two-columns ; this will require pre-creating a single 'key' column, and both files will need to be sorted, also not sure how header comments might be handled

stevekm commented 3 years ago

there are some log and job files here to use for debugging this; /juno/work/ci/helix_filters_01/test_data/11089_G

stevekm commented 3 years ago

this is fixed for the moment, using the reduced Facets python dict method, if it becomes and problem again try some of these other ideas

stevekm commented 3 years ago

also note that work to test mem usage was done here /juno/work/ci/kellys5/projects/benchmarking

mskcc / pluto-cwl

update_cBioPortal_data.py merge_mafs uses too much memory #56