Timeout waiting for reply to request: LOAD_CONFIG

[foxchase]

Hi Jens-Uwe, I got an error when I tried to duplicate what you posted. Any advice, please. Thank you! attached is the screen-shot and the config is below. usage = "target" output_directory = '/home/jpei/Desktop/nanopore/ReadBouncer-1.1.0-Linux/output/' log_directory = '/home/jpei/Desktop/nanopore/ReadBouncer-1.1.0-Linux/log/'

[IBF]

kmer_size = 13
fragment_size = 100000
threads = 3
target_files = ['/home/jpei/Desktop/nanopore/ReadBouncer-1.1.0-Linux/chm13.v1.1_chr21_22.ibf'] deplete_files = ['/home/jpei/Desktop/nanopore/ReadBouncer-1.1.0-Linux/chm13.v1.1_chr1-20_XM.ibf'] exp_seq_error_rate = 0.1

[MinKNOW]

host = "127.0.0.1" port = "9501"
flowcell = "MS00000" channels = [1,512]

[Basecaller]

caller = "guppy"
host = "127.0.0.1" port = "5555" config = "dna_r9.4.1_450bps_fast"

[JensUweUlrich]

Hi @foxchase

Thanks for your interest in using ReadBouncer. The command line output suggests that ReadBouncer cannot connect to the Guppy basecall server. Could you please check whether the basecall server is running on your host machine and whether this basecall server is using the correct configuration file (in this case dna_r9.4.1_450bps_fast for fast Guppy basecalling). Please make also sure that you have a powerful GPU on your machine running the basecall server. Otherwise, basecalling will not be a fast enough to support adaptive sampling. I also like to mention, that we are not supporting Guppy version 6 at the moment.

Cheers Jens

[foxchase]

Thank you Jens for your reply. I have RTX A5000(16GB)/i9(8cores). I checked the guppy is OK and correct cfg file, below are the screenshots for GPU and guppy service. I tried again and got the same error. BTW, another question: do I need the depletion file if only for target enrichment? Thank you.

[JensUweUlrich]

Ok. I was not aware that Guppy 5.1.15 already introduced IPC sockets for the basecall server. I thought this was only introduced with Guppy 6. As a quick workaround you could try to change the config.toml as follows:

[Basecaller]
caller = "Guppy"
host = "ipc:///tmp/.guppy"
port = "5555"
config = "dna_r9.4.1_450bps_fast"

Please let me know if this fixes the problem. Otherwise, I will have to dive deeper into the dark matter of IPC sockets and change the code accordingly :-(

Cheers Jens

[foxchase]

Thank you for your reply. I tried with what was suggested, but no luck, attached is the screenshot.

[mahreenkn]

Hi all, just wanted to jump in to report that I'm currently seeing a similar error; running Guppy v. 5.1.15 as well. Guppy GPU is up and running, and my toml file is almost identical to above.

user@RT5698U-GGB:~$ /home/user/ReadBouncer-1.1.0-Linux/bin/ReadBouncer ReadBouncer/chr7_selection.toml 
Calculated size of the IBF : 9.43198 MBytes
IBF-build processed 1 sequences (4.63162 Mbp) in 0.160635 seconds (0.373518 Kseq/m, 1729.99 Mbp/m)
 - 1 sequences in 47 bins were written to the IBF
Trying to connect to MinKNOW
Host : 127.0.0.1
Port : 9501
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
[2022-03-10 16:27:58.860468] [0x00007fd496ffd700] [info]    Connecting to server as ''
[2022-03-10 16:27:58.861721] [0x00007fd496ffd700] [info]    Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Real time : 0.221736 sec
CPU time  : 0.200719 sec
Peak RSS  : 50 MByte

Edited to add the toml file I'm using:

  usage = "target"
  output_directory = '/home/user/ReadBouncer_out'     
  log_directory = '/home/user/ReadBouncer_out'

  [IBF]

  kmer_size = 13
  fragment_size = 100000
  target_files = ['/home/user/hg38_reference/Chr7_22q11.22-24.fa']
  deplete_files = []
  exp_seq_error_rate = 0.1
  threads = 3

  [MinKNOW]

  host = "127.0.0.1"
  port = "9501"
  flowcell = "MN28703"
  channels = [1,512]

  [Basecaller]

  caller = "Guppy"
  host = "127.0.0.1"
  port = "5555"
  config = "dna_r9.4.1_450bps_fast"

I'm not sure if this error is due to the same reason, or a different one? Thought I'd post it here anyway. Thanks for your help!

[foxchase]

Thank you Jens for your quick reply. Do I need both target files and deplete files for just enrichment? or only target files are OK for enrichment?

[JWDebler]

I have the same problem as @mahreenkn. Running Guppy 5.1.15, but in TCP mode. This is what my guppy.service line looks like, and live GPU basecalling via minknow works fine.

[Service]
Type=simple
Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH"
ExecStart=/opt/ont/guppy_5115/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5555 --ipc_threads 3 --use_tcp --device "cuda:all"

But ReadBouncer doesn't seem able to connect:

Trying to connect to MinKNOW
Host : localhost
Port : 9501
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
[2022-03-10 18:05:32.161549] [0x00007fd3a97e6700] [info]    Connecting to server as ''
[2022-03-10 18:05:32.162889] [0x00007fd3a97e6700] [info]    Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Real time : 1653 sec
CPU time  : 0.110976 sec
Peak RSS  : 71 MByte

[JensUweUlrich]

I did some investigations today. It definitely has nothing to do with the TCP or IPC usage. ONT changed something in the layout of the messages sent between basecall server and basecall client. I'm currently upgrading the client to support Guppy 6. I guess this will occupy the next days. But I will get back to you asap.

[JensUweUlrich]

Ok, this took me some time but finally, I get it done. There is a new release version 1.1.1 that supports Guppy 6 for base calling. There are some important notes when using Guppy 6:

• If your basecall server is not hosted on the same machine as ReadBouncer, you have to start the basecall server with --allow_non_local option. This implies using TCP/IP protocol for message communication with ReadBouncer.
• If both run on the same machine, you have two options. You can either start the basecall server with --use_tcp option or use IPC for message exchange with ReadBouncer, which is the default behaviour
• If you decide to use IPC, make sure that the basecall server and ReadBouncer are started under the same user privileges. By default the basecall server is started as root or minknow user.
• When using IPC, for example on the following address ipc://home/jens/guppy_ipc, ReadBouncer expects the following parameter values host = "ipc://home/jens" and port = "guppy_ipc"

I hope this resolves your issues. However, I'm not sure whether it works with Guppy 5.1.15, and I would recommend to upgrade to the latest Guppy version.

Cheers

[foxchase]

Thank you for your diligent work.

[JensUweUlrich]

Hi @mahreenkn

I removed the merging comments that were accidentally leftover on the master branch. Should compile now without errors.

Cheers

[mahreenkn]

Thank you @JensUweUlrich! I recompiled, and re-ran the command but I seem to be getting this error:

Calculated size of the IBF : 9.43198 MBytes
IBF-build processed 1 sequences (4.63162 Mbp) in 0.14161 seconds (0.423699 Kseq/m, 1962.41 Mbp/m)
 - 1 sequences in 47 bins were written to the IBF
Trying to connect to MinKNOW
Host : 127.0.0.1
Port : 9501
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
Connecting to Guppy basecall server on address : 127.0.0.1:5555
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_request] Bad request message: LOAD_CONFIG. request rejected. Server protocol version [10.0.0] is incompatible with request protocol version [10.1.0].
[bad_request] Bad request message: LOAD_CONFIG. request rejected. Server protocol version [10.0.0] is incompatible with request protocol version [10.1.0].
Real time : 186.515 sec
CPU time  : 0.268638 sec
Peak RSS  : 49 MByte

I can try with Guppy 6 instead and report back if that would be helpful (if you think that's what the Server protocol version [10.0.0] is incompatible with request protocol version [10.1.0]. error might be related to?).

Also, it may also be useful to note I'm running this on Ubuntu 20.04 - I seem to have been running into other permission errors with the Ubuntu upgrade coupled with the latest MinKNOW updates. I'm not sure how to resolve them, so I think the consensus is just waiting to see if ONT releases another patch to work around this soon? Would be interested in hearing whether you or others have had success with a similar setup though!

[JensUweUlrich]

Ok. that was something I suspected. ONT released so much different guppy protocol message versions that it's hard to keep up with all of them. I really recommend to use Guppy 6 at this point. We started an adaptive sampling run with ReadBouncer on Monday where we used Guppy 6 and did not recognize any issues.

[JensUweUlrich]

Guppy 6 support in versions 1.1.1. and 1.2.0

[merfre]

Hello,

I'm sorry to comment on this closed issue, but I am now getting this error with the new version of ReadBouncer. I have updated ReadBouncer on my machine (Ubunto 20.04) and it is running successfully (able to create ibf files and connect to minknow) except for this error occurring when I try to perform adaptive sequencing on a playback run.

Here is the output I get:

sudo /home/merfre/ReadBouncer/build/ReadBouncer-1.2.0-Linux/bin/ReadBouncer --config ./test1_config.toml

709 bins were loaded in 0.10366 seconds from the IBF 28831 bins were loaded in 3.90107 seconds from the IBF Trying to connect to MinKNOW Host : 127.0.0.1 Port : 9502 Connecting to MinKNOW using the token: 867c1cd4-2ad1-4422-8cc0-4a7c14e6c1f5 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : ipc:///home/merfre/5555 [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL Real time : 14.3886 sec CPU time : 12.0589 sec Peak RSS : 17254 MByte

Here is my config file:

usage               = "target"
output_directory    = '/home/merfre/bouncertest1/output/'
log_directory       = '/home/merfre/bouncertest1/'

[IBF]

kmer_size           = 15 #ideal for guppy
fragment_size       = 200000 #ideal for guppy
threads             = 3
target_files        = ['/home/merfre/bouncertest1/output/chm13.v1.1_chr21_22.ibf']
deplete_files       = ['/home/merfre/bouncertest1/output/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate  = 0.1

[MinKNOW]

host                = "127.0.0.1"
port                = "9502"
flowcell            = "MN17166"
token_path          = "test/tmp/minknow-auth-token.json"  #path to authentication token file (if not localhost)

[Basecaller]

caller             = "Guppy"
threads            = 3
host               = "ipc:///home/merfre"
port               = 5555
config             = "dna_r9.4.1_450bps_fast"

This is how I start my basecalling server:

sudo guppy_basecall_server --log_path bouncertest1/guppy.log --port 5555 --config dna_r9.4.1_450bps_fast.cfg --device cuda:0 --gpu_runners_per_device 8 --num_callers 4 --chunks_per_runner 1024 --chunk_size 1000

ONT Guppy basecall server software version 6.1.1+1f6bfa7f8, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: bouncertest1/guppy.log chunk size: 1000 chunks per runner: 1024 max queued reads: 2000 num basecallers: 4 num socket threads: 2 max returned events: 50000 gpu device: cuda:0 kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy/bin

Config loaded: config file: /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None Starting server on port: ipc:///home/merfre/5555

I have an RTX 3080, have tried using tcp, and tried 4 versions of guppy (6.0, 6.1, 6.2, and 6.3) - all resulted in the same error. The results above are from guppy 6.1.1 and minknow 5.2.2

[JensUweUlrich]

Hi @merfre

Just a short question: are you running ReadBouncer and Guppy on the same machine? I tested it with Guppy 6.0.6, and it worked properly, even with both tools on different machines. But maybe ONT changed the message layout again. Could you maybe try to start the basecaller with parameters --use-tcp --port 5556 and change the lines in the ReadBouncer config.toml accordingly to

host               = "localhost"
port               = 5556

I just want to be sure that it has nothing to do with read and write privileges.

[merfre]

Hi @JensUweUlrich

Thank you for the quick reply! Yes, they are all run on the same machine. I tried those changes and got the same error. I also went and changed guppy to 6.0.6 and it connected more than it has in the past (only with --use_tcp and --port 5555 because without tcp I get the same error), but now I am getting a different error:

sudo /home/merfre/ReadBouncer/build/ReadBouncer-1.2.0-Linux/bin/ReadBouncer --config ./test1_config.toml

709 bins were loaded in 0.147654 seconds from the IBF 28831 bins were loaded in 4.12127 seconds from the IBF Trying to connect to MinKNOW Host : 127.0.0.1 Port : 9502 Connecting to MinKNOW using the token: 867c1cd4-2ad1-4422-8cc0-4a7c14e6c1f5 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : localhost:5555 Start receiving live signals thread Start basecalling thread Start read classification thread Start sending unblock messages thread munmap_chunk(): invalid pointer Aborted

Here is my basecall server:

sudo guppy_basecall_server --use_tcp --port 5555 --log_path bouncertest1/guppy.log --config dna_r9.4.1_450bps_fast.cfg --device cuda:0 --gpu_runners_per_device 8 --num_callers 4 --chunks_per_runner 1024 --chunk_size 1000

ONT Guppy basecall server software version 6.0.6+8a98bbcbd, client-server API version 10.1.0 log path: bouncertest1/guppy.log chunk size: 1000 chunks per runner: 1024 max queued reads: 2000 num basecallers: 4 num socket threads: 2 max returned events: 50000 gpu device: cuda:0 kernel path:
runners per device: 8

Config loaded: config file: /opt/ont/guppy/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None Starting server on port: 5555

My config:

usage               = "target"
output_directory    = '/home/merfre/bouncertest1/output/'
log_directory       = '/home/merfre/bouncertest1/'

[IBF]

kmer_size           = 15 #ideal for guppy
fragment_size       = 200000 #ideal for guppy
threads             = 3
target_files        = ['/home/merfre/bouncertest1/output/chm13.v1.1_chr21_22.ibf']
deplete_files       = ['/home/merfre/bouncertest1/output/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate  = 0.1

[MinKNOW]

host                = "127.0.0.1"
port                = "9502"
flowcell            = "MN17166"
token_path          = "test/tmp/minknow-auth-token.json"  #path to authentication token file (if not localhost)

[Basecaller]

caller             = "Guppy"
threads            = 3
host               = "localhost"
port               = 5555
config             = "dna_r9.4.1_450bps_fast"

[JensUweUlrich]

@merfre

This error is related to issue #51. I'm currently working on solving the problem. But it will take some days since I'm currently at a conference.