search

DrosophilaGenomeEvolution / TrEMOLO

Transposable Elements MOvement detection using LOng reads
GNU General Public License v3.0
18 stars 5 forks source link

How to speed up TrEMOLO #14

Closed OliveiraDS-hub closed 7 months ago

OliveiraDS-hub commented 7 months ago

Dear Mourdas, first of all thank you for developing such super useful tool!

I was wondering if there is anyway to speed up TrEMOLO's processing time. Even providing 32 cores, I've realized that TrEMOLO is using only one processor with find_tsd.py script.

The analysis taking longer is this one:

[SNK]--[Wed Nov 15 10:09:14 -03 2023] [TSD:snk] : 32/93 0/2 :2/2 : CHECKING TSD TE ASSEMBLED 33/93 :: TE ASSEMBLED [SNK]--[Wed Nov 15 10:10:46 -03 2023] [TSD:snk] : 33/93 0/1010/10 : CHECKING TSD TE ASSEMBLED 34/93 :: TE ASSEMBLED [SNK]--[Wed Nov 15 10:16:11 -03 2023] [TSD:snk] : 34/93 0/1 :1/1 : CHECKING TSD TE ASSEMBLED 35/93 :: TE ASSEMBLED [SNK]--[Wed Nov 15 10:16:45 -03 2023] [TSD:snk] : 35/93

Is it a bug, or an expected behavior? In any case, could I do something to speed up the process?

Thank you!

M-D75 commented 7 months ago

Hello,

First of all, thank you for using our tool and for sharing your feedback.

Please ensure that you have the parameter MODE_PARALLELING: True in your YAML configuration file.

Be aware that the display and parallel processing of TSDs for INSIDERS are different from those for OUTSIDERS. This might give the impression that only one core is being used. Although the find_tsd.py script is designed to process one sequence at a time, it is normally called simultaneously to handle different sequences.

However, you are correct: in general, the TSD detection part is under-optimized. There are far too many system calls, which significantly reduce performance. We will carry out an update to optimize this.

For now, if you consider TSD detection not to be a priority and would like to increase the speed of TrEMOLO, you can reduce the flank size to 4 in your configuration file: SIZE_FLANK: 4.

Best regards; M-D

M-D75 commented 6 months ago

Hello,

The new update, implemented following your request, is now available. The calculation speed for TSDs has been increased. For instance, for over 6000 Transposable Elements (TE) detected with a flanking size of 50 bp (parameter SIZE_FLANK) and with 20 threads (parameter THREADS), the calculation should take less than 2 seconds.

Best regards, M-D