Out of memory listing entries on one human chromosome on a machine with 300 GB ram and 165 GB BigBed file

deeptools / pyBigWig

A python extension for quick access to bigWig and bigBed files

MIT License

212 stars 48 forks source link

Out of memory listing entries on one human chromosome on a machine with 300 GB ram and 165 GB BigBed file #148

Open balwierz opened 4 months ago

balwierz commented 4 months ago

I thought I could use this library for querying extremely large BigWig files, but despite providing python with 300 GB of ram it crashes anyway. Is pyBigWig keeping all the data in memory at any point of operation?

dpryan79 commented 2 months ago

It'll need to keep those results in memory to output them. I wonder what the utility of such a large bigBed file is, since I suspect you'd be better served by storing some sort of summarization of the underlying data.