Closed Kalior closed 1 year ago
Hi @Kalior
Thanks for your interest in ReadBouncer. As you correctly pointed out, MinKNOW core 5.0 and the corresponding API version include some significant changes that are not supported by our latest ReadBouncer version. @lutfia95 is currently working on that and we hope to release a new version solving this issue soon.
Cheers
Fantastic! Looking forward to it!
Are we there yet?
Hello, Would you please share with us install API for Minknow 4.5.0 ? Thanks
Hello, It seems that Minknow 5.1 is now supported by ReadBouncer, so I have compiled the last readbouncer 2h ago. However I got an error when I start readbouncer :
/usr/local/ReadBouncer/bin/ReadBouncer /home/userlocal/Bureau/Scripts.dir/ReadBouncer.dir/config_classify_test.toml
Error in command line! Error parsing the toml file: Error in [error] key "log_directory" not found in the top-level table --> | 1 | | the parsed file is empty
Here is my toml file that was working last friday :
usage = "classify" output_directory = '/home/userlocal/work/test_readbouncer_moi.dir' #all generated output files will be stored here log_directory = '/home/userlocal/work/test_readbouncer_moi.dir' #all generated log files will be stored here
[IBF]
threads = 10 #(unsigned integer; default: 3)
target_files = ['/home/userlocal/work/test_readbouncer_moi.dir/OsHV-1_microVar-A_KY242785_C.gigas.ibf']
deplete_files = ['/home/userlocal/work/test_readbouncer_moi.dir/cgigas_uk_roslin_v1.ibf']
read_files = ['/home/userlocal/Bureau/Data_test.dir/all_01_1.fastq']
exp_seq_error_rate = 0.07 #(unsigned float between 0 and 1; default: 0.1)
chunk_length = 150 #(unsigned integer; default: 250)
max_chunks = 5
Best, Germain
Hey @gchevignon,
we also updated the command line, to add the config.toml
file you have also to use the flag --config
. In your case will be:
/usr/local/ReadBouncer/bin/ReadBouncer --config /home/userlocal/Bureau/Scripts.dir/ReadBouncer.dir/config_classify_test.toml
Best, Ahmad
Hi Germain
We are currently utilizing the last tests with MinKNOW 5.1 and will Release ReadBouncer 1.2.0 today. However, you could try to call ReadBouncer using the --config parameter
. In your case that would be /usr/local/ReadBouncer/bin/ReadBouncer --config /home/userlocal/Bureau/Scripts.dir/ReadBouncer.dir/config_classify_test.toml
@lutfia95
could we please add a meaningful error message for the missing --config
parameter?
Thanks
Sorry I did not pay attention to the --config parameter .... I'm a little too excited to get readbouncer working! Thanks a lot
Me again ... I have succeeded to setup the simulated device in Minknow and readbouncer is connecting it but I got an error with guppy:
/usr/local/ReadBouncer/bin/ReadBouncer --config /home/userlocal/Bureau/Scripts.dir/ReadBouncer.dir/config_target_test.toml 59192 bins were loaded in 2.20847 seconds from the IBF 1930 bins were loaded in 0.0754784 seconds from the IBF Trying to connect to MinKNOW Host : localhost Port : 9502 Connecting to MinKNOW using the token: c17017a7-eb85-4140-b6bd-8721bc9a3f17 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : 127.0.0.1:5555 [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG [timed_out] Timeout waiting for reply to request: LOAD_CONFIG Real time : 136.666 sec CPU time : 7.94097 sec Peak RSS : 17459 MByte
Guppy seems to work properly because it produce basecalled fastq file.... Best Germain
Here is my toml file:
usage = "target" output_directory = '/home/userlocal/work/test_readbouncer_bulk.dir/' log_directory = '/home/userlocal/work/test_readbouncer_bulk.dir/'
[IBF]
kmer_size = 13
fragment_size = 100000
threads = 3
target_files = ['/home/userlocal/work/test_readbouncer_bulk.dir/chm13.v1.1_chr21_22.ibf']
deplete_files = ['/home/userlocal/work/test_readbouncer_bulk.dir/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate = 0.1
[MinKNOW]
host = "localhost"
port = "9502"
flowcell = "MS00000"
token_path = "test/tmp/minknow-auth-token.json" #path to authentication token file (if not localhost)
[Basecaller]
caller = "Guppy"
Can you give us some more information on how you use Guppy:
Thanks Jens
I have installed guppy-gpu following this protocol : https://community.nanoporetech.com/docs/prepare/library_prep_protocols/experiment-companion-minknow/v/mke_1013_v1_revch_11apr2016/installing-gpu-version-of-guppy-with-minknow-for-minion
So I have installed ont-guppy_6.1.5_linux64.tar.gz
Yes they run on the same machine
I probably run Guppy with MinKNOW but not sure ...
Thanks
Germain
Ok. Than you will probably use the default settings of Guppy, which means that the basecall_server uses IPC. You will need to add the following line to the config.toml file:
[Basecaller]
caller = "Guppy"
host = "ipc:///tmp/.guppy"
I hope this fixes the problem.
Best Jens
Hey, sorry for the late reply ... this did not fix the problem. Best Germain
Ok. Could you please check if there is a /lib/systemd/system/guppyd.service
file on your system and post the content of that file?
[Unit] Description=Service to manage the guppy basecall server. Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018
StartLimitIntervalSec=0
[Service] Type=simple Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH" ExecStart=/opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5555 --ipc_threads 3 Restart=always
RestartSec=10 User=minknow MemoryLimit=8G MemoryHigh=8G CPUQuota=200%
[Install] Alias=guppyd.service WantedBy=multi-user.target
Thanks
You would need to change the ExecStart line and add some parameters to make live-basecalling with Guppy use GPUs, which is needed for adaptive sampling. I also assume that there is a problem with the permissions on the /tmp/.guppy/5555
file. Thus you could try to change the line to
ExecStart=/opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5555 --ipc_threads 3 --gpu_runners_per_device 2 --chunks_per_runner 64 --device cuda:all --use_tcp
After changing these lines, you need to restart the guppy service with the following commands
sudo systemctl daemon-reload
sudo service guppyd restart
Now you can use the following settings in the ReadBouncer config TOML file for connection to the basecaller
[Basecaller]
caller = "Guppy"
host = "localhost"
port = "5555"
I have modified the /lib/systemd/system/guppyd.service and ReadBouncer config TOML file accordingly but I still have the same error. I have also played around with /etc/systemd/system/guppyd.service file and /opt/ont/minknow/conf/app_conf file but I am unable to get this working. I have changed permission on the /tmp/.guppy/5555 with sudo chmod a+w /tmp/.guppy/5555 (from this discussion : https://community.nanoporetech.com/posts/can-anyone-solve-it-super). Still no success ...
Thanks Germain
Hi there, I am trying to run a simulation test, but I have the same problem here...
$ /home/work/ReadBouncer-1.2.0-Linux/bin/ReadBouncer --config /home/work/ReadBouncer-1.2.0-Linux/config.toml
965 bins were loaded in 0.0401868 seconds from the IBF
29596 bins were loaded in 1.09497 seconds from the IBF
Trying to connect to MinKNOW
Host : localhost
Port : 9502
Connecting to MinKNOW using the token: 710eb963-babe-4740-bdef-854b9af4a3d1
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
Connecting to Guppy basecall server on address : localhost:5555
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG
[timed_out] Timeout waiting for reply to request: LOAD_CONFIG
Real time : 212.296 sec
CPU time : 3.87901 sec
Peak RSS : 8896 MByte
Thanks
Just for comparison, this is the Service entry of my guppyd.service file on my Nvidia Xavier
[Service]
Type=simple
Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH"
ExecStart=/usr/bin/guppy_basecall_server --log_path /media/jens/xavierSSD/guppy_logs --config dna_r9.4.1_450bps_fast.cfg --num_callers 4 --gpu_runners_per_device 3 --chunks_per_runner 64 --device cuda:all --port 5555 --ipc_threads 4 --cpu_threads_per_caller 4 --allow_non_local --use_tcp
Restart=always
# Wait ten seconds in-between restarts, to avoid spam if the service starts crashing frequently.
RestartSec=10
User=jens
MemoryLimit=28G
MemoryHigh=28G
CPUQuota=400%
and this is my ReadBouncer config.toml from a remote laptop
Basecaller]
caller = "Guppy" #(default: DeepNano)
host = "xavier" #(ip address or name of the computer hosting Guppy Basecall Server; default: 127.0.0.1)
port = 5555 #(port number on which the basecall server is running on the host; default: 5555)
config = "dna_r9.4.1_450bps_fast"
I guess the problem is that you did not provide the config file. Please also check whether the basecaller service is working by
sudo service guppyd status
I still haven't solved the problem...(I can complete testing with DeepNano) Here is my toml file:
usage = "target"
output_directory = '/home/myname/ReadBouncer-1.2.0-Linux/results/'
log_directory = '/home/myname/ReadBouncer-1.2.0-Linux/log/'
[IBF]
kmer_size = 13
fragment_size = 100000
threads = 3
target_files = ['/home/myname/ReadBouncer-1.2.0-Linux/results/chm13.v1.1_chr21_22.ibf']
deplete_files = ['/home/myname/ReadBouncer-1.2.0-Linux/results/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate = 0.1
[MinKNOW]
host = "localhost"
port = "9502"
flowcell = "MS00000"
token_path = "test/tmp/minknow-auth-token.json" #path to authentication token file (if not localhost)
[Basecaller]
caller = "Guppy"
host = "ipc:///tmp/.guppy"
config = "dna_r9.4.1_450bps_fast"
and minknow/guppyd status:
~$ sudo service minknow status
● minknow.service - MinKNOW Instrument Software for MinION (daemon)
Loaded: loaded (/etc/systemd/system/minknow.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-08-29 13:19:19 JST; 2min 3s ago
Process: 21394 ExecStartPre=/bin/sleep 15 (code=exited, status=0/SUCCESS)
Main PID: 21395 (mk_manager_svc)
Tasks: 275 (limit: 154228)
Memory: 276.5M
CGroup: /system.slice/minknow.service
├─21395 /opt/ont/minknow/bin/mk_manager_svc --simulated-integrated-devices 1
├─21397 /opt/ont/minknow/bin/crashpad_handler --database=/var/lib/minknow/data/core-dump-db --metrics-dir=/var/lib/minknow/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/6e3c4958cfd996b7000dadb02a931ff700986aa48b33903b5902dff716c95809/minidump --annotation=cmd_line=/o>
├─21448 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_addr=127.0.0.1:33667 --backend_tls_noverify --backend_tls=true --server_bind_address=0.0.0.0 --se>
├─21449 /opt/ont/minknow/bin/basecall_manager --guppy-address ipc:///tmp/.guppy/5555 --conf /opt/ont/minknow/conf --system-info /tmp/minknow_9fd0bf68-c06f-4dd7-9486-854c3ecc0eaa/system_info.pb --secure 127.0.0.1:0 --queued-command-count 2
├─21450 /opt/ont/minknow/bin/control_server --config /tmp/minknow_9fd0bf68-c06f-4dd7-9486-854c3ecc0eaa/conf-MS00000 --system-info /tmp/minknow_9fd0bf68-c06f-4dd7-9486-854c3ecc0eaa/system_info.pb --queued-command-count 2
├─21458 /opt/ont/minknow/bin/crashpad_handler --database=/var/lib/minknow/data/core-dump-db --metrics-dir=/var/lib/minknow/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/6e3c4958cfd996b7000dadb02a931ff700986aa48b33903b5902dff716c95809/minidump --annotation=cmd_line=/o>
├─21472 /opt/ont/minknow/bin/crashpad_handler --database=/var/lib/minknow/data/core-dump-db --metrics-dir=/var/lib/minknow/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/6e3c4958cfd996b7000dadb02a931ff700986aa48b33903b5902dff716c95809/minidump --annotation=cmd_line=/o>
├─21510 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_addr=127.0.0.1:39227 --backend_tls_noverify --backend_tls=true --server_bind_address=0.0.0.0 --se>
├─21621 /opt/ont/minknow/bin/analyser --config /tmp/minknow_9fd0bf68-c06f-4dd7-9486-854c3ecc0eaa/conf-MS00000 --system-info /tmp/minknow_9fd0bf68-c06f-4dd7-9486-854c3ecc0eaa/system_info.pb
├─21622 /opt/ont/minknow/bin/grpcwebproxy --server_tls_cert_file=/opt/ont/minknow/conf/rpc-certs/localhost.crt --server_tls_key_file=/opt/ont/minknow/conf/rpc-certs/localhost.key --backend_addr=127.0.0.1:8000 --backend_tls_noverify --backend_tls=true --server_bind_address=0.0.0.0 --ser>
└─21630 /opt/ont/minknow/bin/crashpad_handler --database=/var/lib/minknow/data/core-dump-db --metrics-dir=/var/lib/minknow/data/core-dump-db --url=https://submit.backtrace.io/nanoporetech/6e3c4958cfd996b7000dadb02a931ff700986aa48b33903b5902dff716c95809/minidump --annotation=cmd_line=/o>
Aug 29 13:19:04 myname-ZA9C-R39 systemd[1]: Starting MinKNOW Instrument Software for MinION (daemon)...
Aug 29 13:19:19 myname-ZA9C-R39 systemd[1]: Started MinKNOW Instrument Software for MinION (daemon).
~$ sudo service guppyd status
● guppyd.service - Service to manage the guppy basecall server.
Loaded: loaded (/etc/systemd/system/guppyd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-08-29 13:18:57 JST; 2min 52s ago
Docs: https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018
Main PID: 21306 (guppy_basecall_)
Tasks: 80 (limit: 154228)
Memory: 838.5M (high: 20.0G limit: 20.0G)
CGroup: /system.slice/guppyd.service
└─21306 /home/myname/ont-guppy/ont-guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --gpu_runners_per_device 2 --chunks_per_runner 64 --port /tmp/.guppy/5555 --ipc_threads 3 -x cuda:all
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: kernel path:
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: runners per device: 2
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA.
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: The EULA may be found in /home/myname/ont-guppy/ont-guppy/bin
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: Config loaded:
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: config file: /home/myname/ont-guppy/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: model file: /home/myname/ont-guppy/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: adapter scaler model file: None
Aug 29 13:18:57 myname-ZA9C-R39 guppy_basecall_server[21306]: Starting server on port: ipc:///tmp/.guppy/5555
Was I not able to provide the config file?
@moisann
That's strange. I used excatly the same parameter settings for the basecaller as you did and it works for me.
Can you also make sure that you have installed the Guppy-GPU version. The output from your service guppyd status
command does not tell anything about using CUDA. Mine looks like this
● guppyd.service - Service to manage the guppy basecall server.
Loaded: loaded (/lib/systemd/system/guppyd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-08-29 11:35:42 CEST; 24min ago
Docs: https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018
Main PID: 6567 (guppy_basecall_)
Tasks: 76 (limit: 4915)
CGroup: /system.slice/guppyd.service
└─6567 /usr/bin/guppy_basecall_server --log_path /media/jens/xavierSSD/guppy_logs --config dna_r9.4.1_450bps_fast.cfg --num_callers 4 --gpu_runners_per_device 3 --chunks_p
Aug 29 11:35:43 xavier guppy_basecall_server[6567]: max returned events: 50000
Aug 29 11:35:43 xavier guppy_basecall_server[6567]: gpu device: cuda:all
Aug 29 11:35:43 xavier guppy_basecall_server[6567]: kernel path:
Aug 29 11:35:43 xavier guppy_basecall_server[6567]: runners per device: 3
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: Config loaded:
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: config file: /opt/ont/ont-guppy/data/dna_r9.4.1_450bps_fast.cfg
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: model file: /opt/ont/ont-guppy/data/template_r9.4.1_450bps_fast.jsn
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: adapter scaler model file: None
Aug 29 11:35:45 xavier guppy_basecall_server[6567]: Starting server on port: ipc:///tmp/.guppy/5555
It also seems to me that you are starting Guppy with the user minknow
and when you start ReadBouncer with another user, you may have no access to the file tmp/.guppy/5555
. You could change this as well, or use TCP/IP for communication with Guppy as I recommended above.
Hello, Because I have modified so much different file I have decided to reinstall minknow and guppy ... and I have eventually reformated my computer to restart on clean base ... Unfortunatly the new install of minknow did not install guppyd.service file ... So I am stuck and I can't even install guppy-gpu so I made a ticket to Nanopore support regarding this missing guppyd.service file. Best
You can easily create the guppyd service file on your own because it is a simple text file. I also recommend you to read the repository from Miles Benton where he provides a guppyd.service file as well.
Just for comparison, this is the Service entry of my guppyd.service file on my Nvidia Xavier
[Service] Type=simple Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH" ExecStart=/usr/bin/guppy_basecall_server --log_path /media/jens/xavierSSD/guppy_logs --config dna_r9.4.1_450bps_fast.cfg --num_callers 4 --gpu_runners_per_device 3 --chunks_per_runner 64 --device cuda:all --port 5555 --ipc_threads 4 --cpu_threads_per_caller 4 --allow_non_local --use_tcp Restart=always # Wait ten seconds in-between restarts, to avoid spam if the service starts crashing frequently. RestartSec=10 User=jens MemoryLimit=28G MemoryHigh=28G CPUQuota=400%
and this is my ReadBouncer config.toml from a remote laptop
Basecaller] caller = "Guppy" #(default: DeepNano) host = "xavier" #(ip address or name of the computer hosting Guppy Basecall Server; default: 127.0.0.1) port = 5555 #(port number on which the basecall server is running on the host; default: 5555) config = "dna_r9.4.1_450bps_fast"
I guess the problem is that you did not provide the config file. Please also check whether the basecaller service is working by
sudo service guppyd status
Hi, still not working on my side ... the simulation run get stuck in "starting run" and I think this is because my /lib/systemd/system/guppyd.service is not OK ... Can you please share your two Service entry from /lib/systemd/system/guppyd.service and /etc/systemd/system/guppyd.service Thanks Germain
Hey ! Nanopore just released a patch to fix the guppyd.service problem with last version of Minknow : https://community.nanoporetech.com/posts/minknow-22-08-8-patch-rele
I hope readbouncer will work with this new release ...?
I will not be able to test it before next tuesday. Best Germain
Hello, this patch indeed fixed the problem with guppyd.service which is now guppyd-cpu.service .... So i was able to restart simulation run but readbouncer still produce the same error as before :
/usr/local/ReadBouncer/bin/ReadBouncer --config /home/userlocal/Desktop/Scripts.dir/ReadBouncer.dir/config_target_test-V2.toml 965 bins were loaded in 0.0415536 seconds from the IBF 29596 bins were loaded in 1.18629 seconds from the IBF Trying to connect to MinKNOW Host : localhost Port : 9502 Connecting to MinKNOW using the token: 5ca149dd-ba11-4d30-ab77-540ec7f59302 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : ipc:///tmp/.guppy/5555 [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG [timed_out] Timeout waiting for reply to request: LOAD_CONFIG Real time : 144.397 sec CPU time : 4.53437 sec Peak RSS : 8896 MByte
So sadly I am still stuck on this ...
@gchevignon
ONT did break the ReadBouncer-Guppy-Interface with their latest changes, as also reported in issue #54. This should be fixed in release 1.2.2
Great Thanks a lot !
Hi,
Really fantastic work on this project! Looks like a great alternative to readfish/uncalled/ONT's own adaptive sampling!
I wanted to run this on a fresh install of Minknow, but since march Minknow ships with Minkow core 5.0 (see https://community.nanoporetech.com/posts/software-release-22-03-2-f-9633). As I understand their API, this major update won't be backwards-compatible with the previous API.
Curious if you are planning on supporting the new API or if I'm stuck with the built-in adaptive sampling.