Closed le-ander closed 4 years ago
This is way too long. I have no clue why right now, let me think about it.
I just copied the file to the local disk of the serve I am running this on instead of continuously fetching the files from a network drive. It helped a bit, but not very much.
0%| | 11/32774 [07:51<398:27:50, 43.78s/it]
I can't really seem to find a way speeding it up right now. Any idea? 🤔
I am working on it right now. I ran the same script on my computer and the server. For some reason it finished on my computer but not on the server. So I am trying to figure out what is going wrong.
I just tried it on my laptop as well and it does not seems to be faster than on the server:
0%| | 11/32774 [24:08<1558:32:58, 171.25s/it]
My fragments.tsv file has around 140 million lines.
I think I found the problem. https://github.com/colomemaria/epiScanpy/blob/70e568e1cf1ca247168a57f0058d60627bc28b8b/episcanpy/count_matrix/_bld_atac_mtx.py#L86 This line goes through the whole barcode list for every single fragment. Avoiding this repeated iteration speeds up matrix building by approximately 10000 fold.
I have rewritten the function and will open a pull request shortly.
Okay, thanks !
Hey,
I having some problems with the extremely long runtime of the bld_mtx_fly() function on my scATAC dataset. I have a 10x dataset with 2.5k cells and am trying to build a feature matrix from the fragments.tsv file using an geneid annotation BED file with 33k features.
I have added a tqdm progress bar to get an idea of the performance.
After 10 Minutes, this is that I see:
This is telling me an expected runtime of 3 weeks, which seems to be too much, considering this statement in the function doc string "Expected running time for 10k cells X 100k features on a personal computer ~65min"
When I use 10kb windows from the make_windows() function, I get over 2 million features and an expected runtime of several years:
Any suggestion how I could still use this function and get started with my ATAC analysis?
Thanks a lot and best regards, Leander