Time taken for demultiplexing

[bhargava-morampalli]

I have started the first step using the command below:

python3 /pathto/deeplexicon/deeplexicon_sub.py dmux -p /input/fast5/ -f multi -m /pathto/deeplexicon/models/resnet20-final.h5 > output.tsv It has been running for 2 and half days now with only 368962 reads classified at this point. I have a total of 1.6 million reads from the run. Is it supposed to run for this long?

I also just noticed that I am supposed to use -g option to use the GPU. I suspect this is just using the CPU for now. Is there a way to speed up this process? If I use GPU, how long would it take and do I need to start from the beginning again or is there a way to resume from 360k reads (at this point)? I am doing this for my pilot run with barcodes and looking at doing more if this works good. Any suggestions appreciated. Thank you very much.

[Psy-Fer]

Hello,

The current implementation uses a single CPU, and so it does the calculations one after the other. This can make it quite slow. Depending on the hardware you have available, you could speed it up by running the software on each individual fast5 in parallel, then merging the output file at the end. We do this when running the CPU mode on our HPC cluster.

The GPU implementation is a little quicker doing the inference step (barcode prediction) however the extraction and segmentation of the signal is still a bottleneck running on a single CPU.

I have been planning on releasing a multi-processor version, however my attention was diverted doing SARS-CoV-2 sequencing and analysis.

I can try releasing a faster version within the next week or so if you like, however it may not be very optimised, just faster. The results will still be the same though.

Let me know.

Cheers,

[Huanle]

Hi @bhargava-morampalli ,

you can create multiple commands of processing single multi-fast5 batch files and take advantage of your cluster or multiple CPUs on your own computer.

[690645258]

Hi,

I've been trying to parallelise this in bash but can't get it to work on individual fast5 files. It works perfectly fine with directories but individually I get the following happen python3 ~/deeplexicon/deeplexicon.py -p test.fast5 -f multi -m ~/deeplexicon/models/resnet20-final.h5 > outputs/output2.tsv Using TensorFlow backend. WARNING:tensorflow:From /home/user/Deeplexicon/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. 2020-10-29 10:43:45.843121: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-10-29 10:43:45.861849: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2245705000 Hz 2020-10-29 10:43:45.876395: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561440231260 executing computations on platform Host. Devices: 2020-10-29 10:43:45.876432: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined> WARNING:tensorflow:From /home/user/Deeplexicon/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Traceback (most recent call last): File "/home/user/deeplexicon/deeplexicon.py", line 622, in <module> main() File "/home/user/deeplexicon/deeplexicon.py", line 304, in main C = classify(model, labels, np.array(images), False, args.threshold) File "/home/user/deeplexicon/deeplexicon.py", line 611, in classify y = model.predict(x, verbose=0) File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training.py", line 1149, in predict x, _, _ = self._standardize_user_data(x) File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training.py", line 751, in _standardize_user_data exception_prefix='input') File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training_utils.py", line 128, in standardize_input_data 'with shape ' + str(data_shape)) ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (0, 1)

Is there something wrong with my command stopping me from running it on individual fast5's or?

[Psy-Fer]

Hello,

Normally when running large jobs, we are using HPC submission systems, such as SGE or PBS. In this case, we copy each multi-fast5 to the local storage on the node, then use that folders location with the -p option, as -p is meant to take a PATH, and read all fast5's in that path, rather than an individual folder.

So one trick without a submission array job, would be to split your multifast5 files into "batches", and put them in their own folder. Then just run the command on each folder separately (in parallel), then just merge the outputs.

Does that help?

I'm sorry for the delay with multi-processing support. I've been pre-occupied with COVID sequencing and analysis for our state for contact tracing.

James.

[Huanle]

Hi,

I've been trying to parallelise this in bash but can't get it to work on individual fast5 files. It works perfectly fine with directories but individually I get the following happen python3 ~/deeplexicon/deeplexicon.py -p test.fast5 -f multi -m ~/deeplexicon/models/resnet20-final.h5 > outputs/output2.tsv Using TensorFlow backend. WARNING:tensorflow:From /home/user/Deeplexicon/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. 2020-10-29 10:43:45.843121: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-10-29 10:43:45.861849: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2245705000 Hz 2020-10-29 10:43:45.876395: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561440231260 executing computations on platform Host. Devices: 2020-10-29 10:43:45.876432: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined> WARNING:tensorflow:From /home/user/Deeplexicon/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Traceback (most recent call last): File "/home/user/deeplexicon/deeplexicon.py", line 622, in <module> main() File "/home/user/deeplexicon/deeplexicon.py", line 304, in main C = classify(model, labels, np.array(images), False, args.threshold) File "/home/user/deeplexicon/deeplexicon.py", line 611, in classify y = model.predict(x, verbose=0) File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training.py", line 1149, in predict x, _, _ = self._standardize_user_data(x) File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training.py", line 751, in _standardize_user_data exception_prefix='input') File "/home/user/Deeplexicon/lib/python3.7/site-packages/keras/engine/training_utils.py", line 128, in standardize_input_data 'with shape ' + str(data_shape)) ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (0, 1)

Is there something wrong with my command stopping me from running it on individual fast5's or?

Is test.fast5 a file or path? -p should denote the top path to folder containing fast5 file(s). This means you can/should put one fast5 file in one folder.

[690645258]

Hi all,

thanks for your advice have got it working now!

For anyone else looking for a quick bash script to get it going I've come up with the following: Run this in your fast5_pass folder, change N to the number of cores you want to run it on, I've got it running on 64 at the moment and its using nearly 900 gb of ram, so memory may be rate limiting for some.

FILES=*.fast5

for f in $FILES
do
        mkdir dir_$f
        cp $f dir_$f
        echo $f
done

mkdir outputs

N=64

DIRS=dir*
echo dir*

(
for d in $DIRS
do
        ((i=i%N)); ((i++==0)) && wait
        python3 ~/deeplexicon/deeplexicon.py -p ./$d -f multi -m ~/deeplexicon/models/resnet20-final.h5 > outputs/$d.tsv &
done
)

[Psy-Fer]

To merge the outputs, you should be able to do something like

head -n1 outputs/dir_1.tsv > ./output_all.tsv
for f in dir*; do tail -n + 2 ${f}/*.tsv; done >> ./output_all.tsv

This should get the header from one file, then take all but the header for all the other files. Or you can use your favourite awk for this too.

Let us know if you have any other issues.

James.

[Huanle]

Hi all,

thanks for your advice have got it working now!

For anyone else looking for a quick bash script to get it going I've come up with the following: Run this in your fast5_pass folder, change N to the number of cores you want to run it on, I've got it running on 64 at the moment and its using nearly 900 gb of ram, so memory may be rate limiting for some.

FILES=*.fast5

for f in $FILES
do
        mkdir dir_$f
        cp $f dir_$f
        echo $f
done

mkdir outputs

N=64

DIRS=dir*
echo dir*

(
for d in $DIRS
do
        ((i=i%N)); ((i++==0)) && wait
        python3 ~/deeplexicon/deeplexicon.py -p ./$d -f multi -m ~/deeplexicon/models/resnet20-final.h5 > outputs/$d.tsv &
done
)

this is how I have been running it. but you can change cp $f dir_$f to cd dir_$f && ln -s $f . && cd -, in order to save some space.

[Psy-Fer]

Hello,

The latest update should drastically increase the speed of demultiplexing now. Happy RNA sequencing :)