Open EricR86 opened 8 years ago
Original comment by Michael Hoffman (Bitbucket: hoffman, GitHub: michaelmhoffman).
The original version of genomedata-load-data
was in pure Python and I couldn't load a dense wiggle file annotating the human genome within a few days. I believe most of the time was spent creating string objects, converting them to floats, etc. This is stuff that is very wasteful in Python given all the objects that need to be created.
Original comment by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
I think it would be possible to re-implement the loader in cython (using cdef
type declarations) and have similar performance. I'll do some benchmarking to see if this is a viable route.
Would you take a small performance hit if it improved code portability? We could ditch the gnulib dependency and I hope it would fix the distutils issue that is preventing full py3 compatibility.
Original comment by Eric Roberts (Bitbucket: ericr86, GitHub: ericr86).
I'd second a possible attempt at writing the loader in Cython especially if there isn't a huge performance hit. It would be especially nice if it could lead to expanding out the interface in the future to what the current unused interface in _load_data.py is currently (chunk-size, possible filtering, etc).
Original report (archived issue) by Jay Hesselberth (Bitbucket: jayhesselberth, GitHub: jayhesselberth).
I wonder whether it would be useful to rewrite
genomedata-load-data
in pure Python usingh5py
. I assumeh5py
was not available or immature at the time the C implementation was written. I had a stab at this and I don't think it would be too bad.Advantages
Disadvantages
Has the loading step ever been profiled? What is the slowest step of loading? If it's reading raw data, the C implementation will likely be faster. If it's writing, I would imagine
h5py
is heavily optimized and would likely have performance on par with the C interface.