search

pepkit / peppy

Project metadata manager for PEPs in Python
https://pep.databio.org/peppy
BSD 2-Clause "Simplified" License
37 stars 13 forks source link

Optimize peppy code #388

Closed rafalstepien closed 2 years ago

rafalstepien commented 2 years ago

After #387 there is a possibility to optimize the code so that peppy works better.

rafalstepien commented 2 years ago

Performance on current master branch code on my new computer: image

rafalstepien commented 2 years ago

Performance after merging two iterations into one: image

khoroshevskyi commented 2 years ago

fixed

xuebingjie1990 commented 2 years ago

It still took over 10 min to load the GEO sample table, which has 49176 samples and 12 columns (I killed it at 13 min). It took 5 min to load the ENCODE hg38 sample table, which has 17116 samples and 11 columns.

nsheff commented 2 years ago

It still took over 10 min to load the GEO sample table, which has 49176 samples and 12 columns (I killed it at 13 min). It took 5 min to load the ENCODE hg38 sample table, which has 17116 samples and 11 columns.

OK, this answers the question of "is it fast enough", but another question is, "is it faster?" Can you comment on that?