INVALID_PROTOCOL responses from Guppy

[jamesemery]

I'm trying to connect ReadBouncer to a running replay experiment in MinKNOW and I have run into some trouble getting ReadBouncer to talk to the running Guppy_Server I have up. Can you confirm which version of the Guppy ReadBouncer should be compatible with? This is on the pre-compiled ReadBouncer-1.2.2-Linux. When I try to run my script I get these errors:

/targetedseqscripts/readbonucerinstall/testexperiment.small.toml
IBF-build processed 2 sequences (96.4156 Mbp) in 8.38514 seconds (0.014311 Kseq/m, 689.904 Mbp/m)
 - 2 sequences in 483 bins were written to the IBF
Trying to connect to MinKNOW
Host : 127.0.0.1
Port : 9502
Connecting to MinKNOW using the token: ---------
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
Connecting to Guppy basecall server on address : 127.0.0.1:5556
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Real time : 9.2334 sec
CPU time  : 9.12573 sec
Peak RSS  : 471 MByte

Here is my configuration:

[Basecaller]

caller             = "Guppy"
host                = "127.0.0.1"
port                = "5556"
config              = "dna_r9.4.1_450bps_fast"

My Guppy Basecaller version is being run as so:

/opt/ont/guppy/bin/guppy_basecall_server --use_tcp --port 5556 --config dna_r9.4.1_450bps_fast.cfg --log_path /home/grid/targetedseqscripts/guppy_basecaller_logs --ipc_threads 1 --max_queued_reads 2000 --chunks_per_runner 48 --num_callers 8 
ONT Guppy basecall server software version 6.2.11+e17754edc, client-server API version 11.0.0, minimap2 version 2.22-r1101

[JensUweUlrich]

For Guppy 6.2 there is a separate pre-compiled binary. Sorry for the inconvenience, I'm also annoyed by the constant changes in the API. For the next major Release we will try to get around the basecalling step with Guppy.

[jamesemery]

Thank you for the pointer to the correct version to run. It got me to the point of being able to run through simulations on the fast5 files you provided. I ran into some problems. On the packaged version you sent me here, I have hit a Segmentation Fault out of ReadBouncer and evidently crashed the MinKNOW flowcell runner (I had to re-scan and restart the control test flowcells to find it in the UI) within a few hours.

Start read classification thread
Start sending unblock messages thread
Fri Feb 17 16:06:59 2023    Fri Feb 17 16:06:59 2023
Fri Feb 17 16:29:12 2023    Fri Feb 17 16:29:12 2023
Fri Feb 17 16:33:15 2023    Fri Feb 17 16:33:16 2023
Fri Feb 17 17:42:05 2023    Fri Feb 17 17:42:06 2023
Fri Feb 17 18:20:53 2023    Fri Feb 17 18:20:54 2023
Fri Feb 17 18:37:23 2023    Fri Feb 17 18:37:24 2023
Fri Feb 17 19:25:32 2023    Fri Feb 17 19:25:33 2023
Segmentation fault (core dumped)

Interestingly this seems to have caused the MinKNOW flowcell processes to crash and report run state "stopped with error." I see this in the ReadUntilClientLog.txt:

[2023-02-17 20:57:43.746] [RUClientLog] [info] ----------------------------- Intermediate Results -------------------------------------------------------
[2023-02-17 20:57:43.772] [RUClientLog] [info] Number of successfully unblocked reads    : 202054
[2023-02-17 20:57:43.794] [RUClientLog] [info] Number of failed finished reads           : 1197
[2023-02-17 20:57:43.814] [RUClientLog] [info] Number of failed too long reads           : 0
[2023-02-17 20:57:43.819] [RUClientLog] [info] ----------------------------------------------------------------------------------------------------------
[2023-02-17 20:58:21.244] [RUClientLog] [warning] ailed sending action request number
[2023-02-17 20:58:21.645] [RUClientLog] [warning] iled sending action request number
[2023-02-17 20:58:22.045] [RUClientLog] [warning] led sending action request number
[2023-02-17 20:58:22.445] [RUClientLog] [warning] ed sending action request number
[2023-02-17 20:58:22.845] [RUClientLog] [error] Failed sending action request to MinKNOW

My MinKNOW version is: MinKNOW Installed version: 22.08.9 (focal) if that should make a difference.

[JensUweUlrich]

It's hard to say what went wrong. Seems like ReadBouncer lost the connection to MinKNOW for some reason, which caused an error. Did you use a simulation run with the provided bulk fast5 file? If not, it will be almost impossible to reproduce the error.

[jamesemery]

Yes, I'm running the simulation run posted by the Readfish Folks here: https://github.com/LooseLab/readfish (under testing). I've run it through twice and gotten segfaults at seemingly different stages into the run. I'm not entirely sure where to locate the coredump files, they don't seem to co-locate with the output directories i specified. I have been able to run ReadFish on the same (simulated) data without issue so i'm not convinced the Fast5 file is at fault. MinKNOW didn't produce the on-close experiment summary files from what I could see.

[JensUweUlrich]

I could reproduce the error after almost 2 hours in the simulation run. I will debug the code and hopefully find the issue within the next week.

[jamesemery]

Any progress on this?

[JensUweUlrich]

Sorry that it takes so long. The bug is harder to find than initially expected and my Nvidia Xavier made some troubles as well. We are still working on that. However, I can assure that we didn't have this problem in the real experiments we already performed with ReadBouncer.