I was going through the human chromosome 21 & 22 example and am encountering what looks like a communications error with guppy

[matthewlinks]

I've been able to confirm that the build of readbouncer I'm running is working based on success with the classify test and being able to build the IBFs for the human example.

Readbouncer starts up and signals its ready for the sequencing run to start. But when I start the sequencing run I see an error about the communication with guppy

$ ~/readbouncer/ReadBouncer/build/ReadBouncer-1.2.1-Linux/bin/ReadBouncer --config ./AdaptiveSamplingExampleIBFs.toml
965 bins were loaded in 0.0588028 seconds from the IBF
29596 bins were loaded in 1.66077 seconds from the IBF
Trying to connect to MinKNOW
Host : localhost
Port : 9502
Connecting to MinKNOW using the token: 11c32c86-66fe-4ba2-8d47-957c0e088420
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
Connecting to Guppy basecall server on address : ipc:///tmp/.guppy/5555
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Real time : 5.94227 sec
CPU time  : 5.91697 sec
Peak RSS  : 8896 MByte

TOML for Readbouncer is

usage               = "target"
output_directory    = '/data/readbouncer/testing/output'
log_directory       = '/data/readbouncer/testing/log'

[IBF]

kmer_size           = 13
fragment_size       = 100000
threads             = 3
target_files        = ['/data/readbouncer/testing/output/chm13.v1.1_chr21_22.ibf']
deplete_files       = ['/data/readbouncer/testing/output/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate  = 0.1

[MinKNOW]

host                = "localhost"
port                = "9502"
flowcell            = "X3"

[Basecaller]

caller             = "Guppy"
host               = "ipc:///tmp/.guppy"
port               = 5555

This is being run on a GridION (minknow-core-gridion v5.2.2, and guppy ont-guppy-for-gridion 6.2.7.

I have checked the guppy logs and the server is active and running.

ReadBouncerLog.txt reads

[2022-09-19 13:53:23.120] [ReadBouncerLog] [info] Trying to connect to MinKNOW
[2022-09-19 13:53:23.120] [ReadBouncerLog] [info] Host : localhost
[2022-09-19 13:53:23.120] [ReadBouncerLog] [info] Port : 9502
[2022-09-19 13:53:23.155] [ReadBouncerLog] [info] Connection successfully established!
[2022-09-19 13:53:23.156] [ReadBouncerLog] [info] Sequencing has begun. Starting live signal processing!
[2022-09-19 13:53:23.158] [ReadBouncerLog] [info] Set break_reads_after_seconds = 0.4
[2022-09-19 13:53:23.160] [ReadBouncerLog] [error] Failed establishing connection to Guppy basecall server!
[2022-09-19 13:53:23.160] [ReadBouncerLog] [error] Error message : [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL

ReadUntilClientLog.txt reads

[2022-09-19 13:53:23.120] [RUClientLog] [info] Connect to MinKNOW instance via secure SSL/TLS connection to localhost on port 9502
[2022-09-19 13:53:23.149] [RUClientLog] [info] RPC port for X3 resolved
[2022-09-19 13:53:23.149] [RUClientLog] [info] Trying to connect to Minknow on localhost:8004
[2022-09-19 13:53:23.155] [RUClientLog] [info] Sucessfully connected to minknow instance (version 5.2.2)
[2022-09-19 13:53:23.155] [RUClientLog] [info] Detected GridION Device with ID : X3
[2022-09-19 13:53:23.157] [RUClientLog] [info] Applications default value for break_reads_after_seconds : 0.4
[2022-09-19 13:53:23.158] [RUClientLog] [info] Set new value for break_reads_after_seconds to : 0.4

Is this an example where the API has reworked the messaging details? Any other advice on what to try?

thanks,

Matt

[JensUweUlrich]

@matthewlinks

Seems like ONT has changed the Guppy message format again with version 6.2 (sigh) I will look into it tomorrow.

[matthewlinks]

@JensUweUlrich

Thanks. Appreciate your efforts

[matthewlinks]

Please let me know if there is additional logging you'd like me to try and capture to assist with this?

[JensUweUlrich]

No thanks. I hope to release a fix for the issue next week.

[JensUweUlrich]

@matthewlinks

ReadBouncer release 1.2.2 should fix this issue.

[gchevignon]

Hello, ReadBouncer release 1.2.2 did not fix the communication problem on my side ... I still have the same error. Thanks Germain

[JensUweUlrich]

@gchevignon Which Guppy version are you using? I tested with Guppy 6.3 and everything worked fine for me.

[gchevignon]

My Guppy is Version 6.2.11 the one that come with minknow ... I'll try to install Guppy 6.3 !

[JensUweUlrich]

Seems like every minor release breaks the API (sigh)

[gchevignon]

Ok so when I install Guppy 6.3.7 this remove the /lib/systemd/system/guppyd-cpu.service ... so the simulation run crash ... I have posted a comment at Nanopore community regarding this : https://community.nanoporetech.com/posts/guppy-v6-3-1-release This is a never ending story ;)

[JensUweUlrich]

Wait...did you install ont-guppy or ont-guppy-gpu? The last one is definitely needed for adaptive sampling. I'm not sure why this should delete another service file. Sounds weird! What does the output of guppy_basecall_server --version tell you about your actually installed guppy?

[gchevignon]

Hi, (1) I have first installed Minknow for unbuntu 20 following this protocol : https://community.nanoporetech.com/docs/prepare/library_prep_protocols/experiment-companion-minknow/v/mke_1013_v1_revcl_11apr2016/installing-minknow-on-linu This come with guppy 6.2.11. (2) Then I have installed guppy-gpu for unbuntu 20 following this protocol : https://community.nanoporetech.com/docs/prepare/library_prep_protocols/experiment-companion-minknow/v/mke_1013_v1_revcl_11apr2016/installing-gpu-version-of-guppy-with-minknow-for-minion So at this point I have guppy 6.2.11 and guppy-gpu 6.2.11. (3) Then I have installed guppy 6.3.7 for ubuntu 20 following this protocol: https://community.nanoporetech.com/docs/prepare/library_prep_protocols/Guppy-protocol/v/gpb_2003_v1_revak_14dec2018/linux-guppy Here I have installed guppy-cpu with this command : sudo apt install ont-guppy-cpu (4) Then I have installed guppy-gpu 6.3.7 for ubuntu following this protocol : https://community.nanoporetech.com/docs/prepare/library_prep_protocols/experiment-companion-minknow/v/mke_1013_v1_revcl_11apr2016/installing-gpu-version-of-guppy-with-minknow-for-minion

Now I have no more /lib/systemd/system/guppyd-cpu.service And guppy_basecall_server --version print : : Guppy Basecall Service Software, (C) Oxford Nanopore Technologies plc. Version 6.3.7+532d626, client-server API version 13.0.0

Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy/bin terminate called after throwing an instance of 'boost::filesystem::filesystem_error' what(): boost::filesystem::create_directories: Permission denied [system:13]: "guppy_basecall_server-core-dump-db", "guppy_basecall_server-core-dump-db" Aborted (core dumped)

Do you think I should install guppy-gpu in (3) instead of guppy-cpu with this command: sudo apt install ont-guppy ? Then should I need to do (4) ?

Thanks again for your help !!

[gchevignon]

Hi, Would you share the method you use to install guppy 6.3 along with Minknow please ? Best

[gchevignon]

I am really interested to know how you get guppy 6.3 working in live basecalling with minknow because this should be impossible to set-up according to this post answering my question on the Nanopore community :

Hi [@Germain Chevignon],

I'm afraid it is not possible to mix versions of guppy when configuring MinKNOW for live basecalling. As MinKNOW 5.2 shipped with guppy 6.2, you must use the same version of guppy when configuring live GPU basecalling, following the instructions that you linked. If you want to call your data with a newer version of guppy than the one that MinKNOW has been built against, you can do this as an offline post process with a 6.3 guppy archive build.

Regards,

Mark

[JensUweUlrich]

Hi Mark

I did install the Guppy GPU version provided via nanopore community -> software. I did not test Guppy 6.3 for live basecalling/adaptive sampling from within MinKNOW yet because I don't need that for ReadBouncer. MinKNOW and Guppy are separate instances that run separate servers with which ReadBouncer communicates. But it should be possible to run Guppy 6.3 also from within MinKNOW by just modifying the config files (app_conf) and point MinKNOW to the executables of your new basecaller (see section "Guppy" of the config file)

Best Jens

[gchevignon]

Hi Mark

I did install the Guppy GPU version provided via nanopore community -> software. I did not test Guppy 6.3 for live basecalling/adaptive sampling from within MinKNOW yet because I don't need that for ReadBouncer. MinKNOW and Guppy are separate instances that run separate servers with which ReadBouncer communicates. But it should be possible to run Guppy 6.3 also from within MinKNOW by just modifying the config files (app_conf) and point MinKNOW to the executables of your new basecaller (see section "Guppy" of the config file)

Best Jens

Hi Jens, this was not a message from Mark for you, I have just copy/pasted his (Mark) reply to my question on the community (https://community.nanoporetech.com/posts/guppy-v6-3-1-release#comment_43679) ... Mark is software developer at Nanopore.

There is something I don't get from your last reply. You say "I did not test Guppy 6.3 for live basecalling/adaptive sampling from within MinKNOW yet because I don't need that for ReadBouncer." but on the september 27th you said "Which Guppy version are you using? I tested with Guppy 6.3 and everything worked fine for me."

Did you test it or not ?

Would you please share exactly which version of Guppy you use that works with readbouncer and how you installed it because there is several methods to install Guppy ... Which one is working for you? By working I mean without communication error between Guppy and readbouncer ?

Best Germain

[JensUweUlrich]

@gchevignon

ok there are some misunderstandings. You don't need to start Guppy from within MinKNOW unless you want to use MinKNOW's inbuilt adaptive sampling feature or want to use live-basecalling of finished reads from within MinKNOW. ReadBouncer does not require you to start live-basecalling from within MinKNOW. ReadBouncer uses the following workflow:

1. You start a sequencing run within MinKNOW
2. raw signal data are submitted from MinKNOW to ReadBouncer
3. ReadBouncer submits the raw signal data to Guppy for real-time basecalling
4. basecalled data is submitted from Guppy to ReadBouncer
5. ReadBouncer classifies the basecalled reads and sends unblock messages back to MinKNOW.

Assuming that you successfully created a connection between ReadBouncer and MinKNOW, you only need to start a guppy_basecall_server (either from the command line or configured as a linux service using the guppyd file) and tell ReadBouncer the port on which the guppy_basecall_server is running. I tested ReadBouncer 1.2.2 with Guppy 6.3.7 and that worked pretty well. For the installation of Guppy I just downloaded the tar.gz file from the nanopore community, unpacked the file, and copied the whole folder to "/opt/ont/guppy_6.3.7". Then I changed symlinks to the guppy executables in "usr/bin". That should make the system start guppy 6.3.7 basecall_server automatically after startup. Then you only need to have a look in the guppyd service file which IPC port is used by the guppy basecall server. I hope this helps a bit. Unfortunately ONT is changing so much in the guppy message setup with every minor release, which causes troubles in the interface between Guppy and ReadBouncer.

Best Jens

[matthewlinks]

I tried with the 1.2.2 build. No joy

/data/scratch/readbouncer-1.2.2/ReadBouncer-1.2.2-Linux/bin/ReadBouncer --config ./AdaptiveSamplingExampleIBFs.toml
483 bins were loaded in 0.0648476 seconds from the IBF
14802 bins were loaded in 1.81475 seconds from the IBF
Trying to connect to MinKNOW
Host : localhost
Port : 9502
Connecting to MinKNOW using the token: 62030c5c-29ce-4da3-90ec-0604bce70ad8
Connection successfully established!
Waiting for device to start sequencing!
Please start the sequencing run now!
Sequencing has begun. Starting live signal processing!
Connecting to Guppy basecall server on address : ipc:///tmp/.guppy/5555
[guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
[bad_reply] Could not interpret message from server for request: LOAD_CONFIG. Reply: INVALID_PROTOCOL
Real time : 114.748 sec
CPU time  : 6.75679 sec
Peak RSS  : 8916 MByte

[matthewlinks]

This is on a gridION with the following software versions

minknow-core-gridion:
5.2.2

ont-bream4-gridion:
7.2.8

ont-configuration-artic-gridion:
5.2.5

ont-configuration-customer-gridion:
5.2.5

ont-gridion-mooneye:
5.2.5

ont-guppy-for-gridion:
6.2.7

crudini:
0.9.3

epi2me-agent3-gridion:
3.5.6

epi2me-cli:
2021.9.14

ont-customer-base-gridion:
22.08

ont-gridion-base:
6.3.0

ont-guppyd-for-gridion:
6.2.7

ont-jwt-auth:
0.31

ont-kingfisher-ui-gridion:
5.2.13

ont-nextflow-artic-offline:
0.3.14

ont-nextflow-for-gridion:
21.10.0

ont-platform-bash-stdlib-gridion:
1.9.5

ont-platform-system-scripts-gridion:
1.18.0

ont-platform-theme-gridion:
1.4.8

ont-platform-usbmount-gridion:
1.5.1

ont-python:
3.7.4

ont-remote-support-gridion:
3.1.8

ont-system-identification-gridion:
4.2.03

ont-ubuntu-repository-gridion:
22.07.01

ont-ubuntu-tuning-gridion:
2022.05.01

ont-vbz-hdf-plugin:
1.0.6

[matthewlinks]

AdaptiveSamplingExampleIBFs.toml reads

usage               = "target"
output_directory    = '/data/readbouncer/testing/output'
log_directory       = '/data/readbouncer/testing/log'

[IBF]

kmer_size           = 15
fragment_size       = 200000
threads             = 3
target_files        = ['/data/readbouncer/testing/output/chm13.v1.1_chr21_22.ibf']
deplete_files       = ['/data/readbouncer/testing/output/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate  = 0.1

[MinKNOW]

host                = "localhost"
port                = "9502"
flowcell            = "X3"

[Basecaller]

caller             = "Guppy"
host               = "ipc:///tmp/.guppy"
port               = 5555

[matthewlinks]

There are additional updates from ONT available that I would move the system to versions

minknow-core-gridion:
5.2.4

ont-bream4-gridion:
7.2.9

ont-configuration-artic-gridion:
5.2.8

ont-configuration-customer-gridion:
5.2.8

ont-guppy-for-gridion:
6.2.11

ont-customer-base-gridion:
22.09

ont-guppyd-for-gridion:
6.2.11

ont-kingfisher-ui-gridion:
5.2.15

@JensUweUlrich Please let me know if you think I should push those updates and retry with 1.2.2? Or if there are other logs you'd like me to capture?

thanks

[JensUweUlrich]

@matthewlinks That's so annoying. The latest ReadBouncer version 1.2.2 seems to be compatible with Guppy 6.3 but not with Guppy 6.2, while ReadBouncer 1.2.1 is compatible with Guppy 6.0 but not with Guppy 6.1 and 6.2. I'm actually thinking about possibilities to get around these issues without releasing an new ReadBouncer version for every new minor release of Guppy.

You could try to install Guppy 6.3 separately on the GridION and start a separate Guppy basecall server instance to communicate with ReadBouncer, but that's not that easy if you don't have any experience with it.

As a quick fix I could try to get around the compatability issues in the message interface between ReadBouncer and Guppy and release a new minor ReadBouncer version. This would solve your issues and @gchevignon's as well.

[JensUweUlrich]

I recently uploaded precompiled binaries of ReadBouncer 1.2.2 supporting Guppy 6.2 (https://github.com/JensUweUlrich/ReadBouncer/releases/tag/v1.2.2). I successfully tested adaptive sampling with ReadBouncer and Guppy 6.2.11, which is the latest version coming with MinKNOW. This should fix your problems regarding the INVALID_PROTOCOL error message. Please provide me with feedback after testing it.

Best Jens

[gchevignon]

Thanks a lot Jens for your time ! Unfortunately I can't get it works ....

I have use ReadBouncer-1.2.2_guppy_6.2-Linux_x64.tar.gz

/home/userlocal/Downloads/ReadBouncer-1.2.2-Linux/bin/ReadBouncer --config /home/userlocal/Desktop/Scripts.dir/ReadBouncer.dir/config_target_test-V2.toml 965 bins were loaded in 0.0401222 seconds from the IBF 29596 bins were loaded in 1.09677 seconds from the IBF Trying to connect to MinKNOW Host : localhost Port : 9502 Connecting to MinKNOW using the token: 02ca633e-344d-439a-aef1-b29543055045 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : ipc:///tmp/.guppy/5555 [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG [timed_out] Timeout waiting for reply to request: LOAD_CONFIG Real time : 88.84 sec CPU time : 3.92318 sec Peak RSS : 8896 MByte

My guppy that comes with Minkow is version is 6.2.11, I have installed guppy-GPU version 6.2.11 in /opt/ont/guppy-gpu/ont-guppy-6.2.11/ The simulation start well, guppy is working well, my GPU is processing during the simulation.

Here is my toml:

usage = "target" output_directory = '/home/userlocal/Desktop/work.dir/ReadBouncer.dir/' log_directory = '/home/userlocal/Desktop/work.dir/ReadBouncer.dir/'

[IBF]

kmer_size = 13
fragment_size = 100000
threads = 3
target_files = ['/home/userlocal/Desktop/work.dir/ReadBouncer.dir/chm13.v1.1_chr21_22.ibf'] deplete_files = ['/home/userlocal/Desktop/work.dir/ReadBouncer.dir/chm13.v1.1_chr1-20_XM.ibf']
exp_seq_error_rate = 0.1

[MinKNOW]

host = "localhost" port = "9502"
flowcell = "MS00000" token_path = "test/tmp/minknow-auth-token.json" #path to authentication token file (if not localhost)

[Basecaller]

caller = "Guppy" host = "ipc:///tmp/.guppy" port = 5555

Here is my /etc/systemd/system/guppyd-cpu.service:

[Unit] Description=Service to manage the guppy basecall server. Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018

Disable start rate limiting -- the service will try and restart itself forever.

StartLimitIntervalSec=0

[Service] Type=simple Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH" ExecStart=/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5555 --ipc_threads 3 -x cuda:all Restart=always

Wait ten seconds in-between restarts, to avoid spam if the service starts crashing frequently.

RestartSec=10 User=minknow MemoryLimit=8G MemoryHigh=8G CPUQuota=200%

[Install] Alias=guppyd-cpu.service WantedBy=multi-user.target

Here is my /lib/systemd/system/guppyd-cpu.service: (if I modify something in this one the simulation crash ...)

[Unit] Description=Service to manage the guppy basecall server. Documentation=https://community.nanoporetech.com/protocols/Guppy-protocol/v/GPB_2003_v1_revQ_14Dec2018

Disable start rate limiting -- the service will try and restart itself forever.

StartLimitIntervalSec=0

[Service] Type=simple Environment="LD_LIBRARY_PATH=/usr/lib/nvidia-current-ont:$LD_LIBRARY_PATH" ExecStart=/opt/ont/guppy/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5555 --ipc_threads 3 Restart=always

Wait ten seconds in-between restarts, to avoid spam if the service starts crashing frequently.

RestartSec=10 User=minknow MemoryLimit=8G MemoryHigh=8G CPUQuota=200%

[Install] Alias=guppyd-cpu.service WantedBy=multi-user.target

Let me know if you need more info ...

Best Germain

[JensUweUlrich]

@gchevignon

you need to specify the Guppy basecalling config your basecall_server uses in the ReadBouncer config. The config should look as follows in your scenario

[Basecaller]
caller = "Guppy"  
host   = "ipc:///tmp/.guppy"
port   = "5555"
config = "dna_r9.4.1_450bps_fast"

Sorry, we are missing this parameter in the documentation. I will fix this immediately.

[gchevignon]

Same error ....

/home/userlocal/Downloads/ReadBouncer-1.2.2-Linux/bin/ReadBouncer --config /home/userlocal/Desktop/Scripts.dir/ReadBouncer.dir/config_target_test-V2.toml 965 bins were loaded in 0.0406771 seconds from the IBF 29596 bins were loaded in 1.19359 seconds from the IBF Trying to connect to MinKNOW Host : localhost Port : 9502 Connecting to MinKNOW using the token: ce0e2e52-e208-4e4f-8e7c-08916617ca96 Connection successfully established! Waiting for device to start sequencing! Please start the sequencing run now! Sequencing has begun. Starting live signal processing! Connecting to Guppy basecall server on address : ipc:///tmp/.guppy/5555 [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [timed_out] Timeout waiting for reply to request: LOAD_CONFIG [timed_out] Timeout waiting for reply to request: LOAD_CONFIG Real time : 87.1848 sec CPU time : 4.47113 sec Peak RSS : 8896 MByte

[JensUweUlrich]

Could you start the basecall_server in the shell using the following command?: /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5556 --ipc_threads 3 -x cuda:all --use_tcp

and change the following lines in the ReadBouncer config:

[Basecaller]
caller = "Guppy"  
host   = "localhost"
port   = "5556"
config = "dna_r9.4.1_450bps_fast"

Now try out adaptive sampling. If that works, please stop the basecall_server and start it again with the following command: /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5556 --ipc_threads 3 -x cuda:all

Now change the ReadBouncer config again:

[Basecaller]
caller = "Guppy"  
host   = "ipc:///tmp/.guppy"
port   = "5556"
config = "dna_r9.4.1_450bps_fast"

And try out adaptive sampling with ReadBouncer. If that works as well, the issue lies in the guppyd.service file and you may need to change port and/or user in the service file.

[gchevignon]

Here is the error I get with the first command in the terminal:

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5556 --ipc_threads 3 -x cuda:all --use_tcp terminate called after throwing an instance of 'boost::wrapexcept' what(): Failed to open file for writing: Input/output error [generic:5]: "/var/log/guppy/guppy_basecall_server_log-2022-10-07_09-36-28.log" Aborted (core dumped)

Same thing with the second one:

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5556 --ipc_threads 3 -x cuda:all terminate called after throwing an instance of 'boost::wrapexcept' what(): Failed to open file for writing: Input/output error [generic:5]: "/var/log/guppy/guppy_basecall_server_log-2022-10-07_09-39-25.log" Aborted (core dumped)

[JensUweUlrich]

Ok. So the problem is that you have no write permissions in /var/log/. Please choose another log directory path like /home/your-user-name/guppy/log and then try again to start the basecall_server.

[gchevignon]

OK I have changed writing permission to /var/log/ here is the output of the command

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5556 --ipc_threads 3 -x cuda:all --use_tcp ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

Config loaded: config file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None [guppy/error] <>::run_server: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:167: CUDA_ERROR_UNKNOWN. Error initialising basecall server using port: 5556. Aborting. The basecall server has shut down successfully.

is 5556 should be 5555 ?

[JensUweUlrich]

I tested the command on my GPU machine and it worked. Are you getting the same error when chaging the port to 5557? Since it's an error message from CUDA, which GPU model is built in your machine?

[gchevignon]

same error with second command with 5556 also

opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5556 --ipc_threads 3 -x cuda:all ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

[guppy/error] ipc_tools::ServerBase::ServerBase: Permission denied. Error binding server to port: ipc:///tmp/.guppy/5556. Aborting. [guppy/error] <>::run_server: Permission denied. Error creating basecall server. Aborting. The basecall server has shut down successfully.

[gchevignon]

NVIDIA GeForce RTX 3080 Laptop GPU

[gchevignon]

with 5557

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port 5557 --ipc_threads 3 -x cuda:all --use_tcp ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

Config loaded: config file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None [guppy/error] <>::run_server: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:167: CUDA_ERROR_UNKNOWN. Error initialising basecall server using port: 5557. Aborting. The basecall server has shut down successfully.

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5557 --ipc_threads 3 -x cuda:all ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

[guppy/error] ipc_tools::ServerBase::ServerBase: Permission denied. Error binding server to port: ipc:///tmp/.guppy/5557. Aborting. [guppy/error] <>::run_server: Permission denied. Error creating basecall server. Aborting. The basecall server has shut down successfully.

[gchevignon]

nvidia-smi Fri Oct 7 10:29:55 2022
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A | | N/A 55C P8 13W / N/A | 2589MiB / 16125MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1051 C ...bin/guppy_basecall_server 2576MiB | | 0 N/A N/A 1914 G /usr/lib/xorg/Xorg 4MiB | | 0 N/A N/A 2583 G /usr/lib/xorg/Xorg 4MiB | +-----------------------------------------------------------------------------+

[JensUweUlrich]

I don't understand why the basecall_server cannot be started using TCP, but the error when using IPC says that you have no write permission in /tmp. Maybe just change the port to --port guppy/5557 and try to start it again.

[gchevignon]

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port guppy/5557 --ipc_threads 3 -x cuda:all ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

Config loaded: config file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None [guppy/error] <>::run_server: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:167: CUDA_ERROR_UNKNOWN. Error initialising basecall server using port: ipc:///home/userlocal/guppy/5557. Aborting.

after changing permission to /tmp

/opt/ont/guppy-gpu/ont-guppy-6.2.11/bin/guppy_basecall_server --log_path /var/log/guppy --config dna_r9.4.1_450bps_fast.cfg --num_callers 1 --cpu_threads_per_caller 2 --port /tmp/.guppy/5557 --ipc_threads 3 -x cuda:all ONT Guppy basecall server software version 6.2.11+e17754e, client-server API version 11.0.0, minimap2 version 2.22-r1101 log path: /var/log/guppy chunk size: 2000 chunks per runner: 160 max queued reads: 2000 num basecallers: 1 num socket threads: 3 max returned events: 50000 gpu device: cuda:all kernel path:
runners per device: 8 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /opt/ont/guppy-gpu/ont-guppy-6.2.11/bin

Config loaded: config file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/dna_r9.4.1_450bps_fast.cfg model file: /opt/ont/guppy-gpu/ont-guppy-6.2.11/data/template_r9.4.1_450bps_fast.jsn model version id 2021-05-17_dna_r9.4.1_minion_96_29d8704b adapter scaler model file: None [guppy/error] <>::run_server: CUDA error at /builds/ofan/ont_core_cpp/ont_core/common/cuda_common.cpp:167: CUDA_ERROR_UNKNOWN. Error initialising basecall server using port: ipc:///tmp/.guppy/5557. Aborting. The basecall server has shut down successfully.

[JensUweUlrich]

Ok, last try. You may also need to stop the other guppyd.services before starting another basecall_server instance manually.

[gchevignon]

same error ...

[JensUweUlrich]

I suspected that. It seems that something in the installation of the Guppy 6.2.11 went wrong. However from your nvidia-smi output, it seems that a guppy_basecall_server was running on your GPU. Could you find out which instance that is incl. version and port?

[gchevignon]

"Could you find out which instance that is incl. version and port?" How can I do this ?

[gchevignon]

Hi, @matthewlinks did you succeed to get Readbouncer working ? Best

[JensUweUlrich]

@gchevignon

using ps -A should give you a list with all processes running on your machine. Just search for the process ID that equals to the process ID of the basecall_server listed by the nvidia-smi command. This should give you all necessary information.

[gchevignon]

Hi, here is the output of ps -A: PID TTY TIME CMD 1063 ? 00:00:06 guppybasecall