guppy error:CommandLineProcessor::create_configuration:Missing required parameter basecalling.model_file

[hanyishui0612]

Hi, thank you for your good job! When I run : (venv3) (base) ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config ../rna_r9.4.1_70bps_hac.cfg --device cuda:all -i nt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads

i met the error: trying port 5000 Using port 5000 Could not load pybasecall, trying for version earlier versions <=7.2.15 pyguppy lib

           ~  buttery-eel - SLOW5 Guppy/Dorado Server Basecalling  ~
                        version: 0.5.0

========================================================================== ARGS

args: Namespace(input='/mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5', output='DRS-WT-rep1_DRS-WT-rep1.reads', basecaller_bin=PosixPath('/home/ashui/Downloads/ont-guppy_6.4.2/bin'), config='../rna_r9.4.1_70bps_hac.cfg', call_mods=False, qscore=None, slow5_threads=4, procs=4, slow5_batchsize=4000, quiet=False, max_read_queue_size=20000, log='buttery_basecaller_logs', moves_out=False, do_read_splitting=False, min_score_read_splitting=50.0, detect_adapter=False, min_score_adapter=60.0, trim_adapters=False, detect_mid_strand_adapter=False, seq_sum=False, barcode_kits=None, enable_trim_barcodes=False, require_barcodes_both_ends=False, detect_mid_strand_barcodes=False, min_score_barcode_front=60.0, min_score_barcode_rear=60.0, min_score_barcode_mid=60.0, profile=False, duplex=False, single=False, above_7310=False, above_7412=False) ['--device', 'cuda:all', '--port', '5000', '--use_tcp']

========================================================================== Starting Guppy/Dorado Basecalling Server

[guppy/error] guppy::CommandLineProcessor::create_configuration: Missing required parameter basecalling.model_file --model_file Traceback (most recent call last): File "/home/ashui/Software/buttery-eel/venv3/bin/buttery-eel", line 8, in sys.exit(main()) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/buttery_eel.py", line 139, in main with start_guppy_server_and_client(args, other_server_args) as client_one: File "/home/ashui/miniforge3/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/basecaller.py", line 120, in start_guppy_server_and_client raise RuntimeError("Server couldn't be started") RuntimeError: Server couldn't be started

I guess it may be an error caused by the config file .My config file rna_r9.4.1_70bps_hac.cfg is null because i don't kwnow what to write in it .I just follow up the readme.md (guppy_basecaller --print_workflows and find suitable option as config file name ). I don't know how to solve this problem, can you help me solve it? Looking forward to your recovery.Thank you!!

[hasindu2008]

Hey, could you please do --config rna_r9.4.1_70bps_hac.cfg instead of --config ../rna_r9.4.1_70bps_hac.cfg? In /home/ashui/Downloads/ont-guppy_6.4.2/data/rna_r9.4.1_70bps_hac.cfg you should have this file. If you simply give as --config rna_r9.4.1_70bps_hac.cfg, buttery-eel with automatically deduce this path through path provided for -g.

[hanyishui0612]

I do --config rna_r9.4.1_70bps_hac.cfg instead of --config ../rna_r9.4.1_70bps_hac.cfg and copy the file to /home/ashui/Downloads/ont-guppy_6.4.2/data/rna_r9.4.1_70bps_hac.cfg .But I still have the same error: [guppy/error] guppy::CommandLineProcessor::create_configuration: Missing required parameter basecalling.model_file --model_file

[hasindu2008]

Can you do a

1. head /home/ashui/Downloads/ont-guppy_6.4.2/data/rna_r9.4.1_70bps_hac.cfg and paste the output here so I can have a look?
2. Also, can you do a ls /home/ashui/Downloads/ont-guppy_6.4.2/data and paste the output?

[hanyishui0612]

1,the first:The file rna_r9.4.1_70bps_hac.cfg is empty. I don’t know what should be written. can you tell me? （I got the following information through guppy_basecaller --print_workflows：FLO-MIN106 SQK-RNA002 rna_r9.4.1_70bps_hac 2020-09-07_rna_r9.4.1_minion_256_8f8fc47b ） 2，ls /home/ashui/Downloads/ont-guppy_6.4.2/data the output: [Uploading output.txt…]() Thank you for your help !!

[hasindu2008]

When you download the Guppy basecaller tar file and extract it, it should have that rna_r9.4.1_70bps_hac.cfg file. You should not be manually creating it. For example, if you extracted the downloaded Guppy tarball to /home/ashui/Downloads/ont-guppy_6.4.2/, under data/ it should have many .cfg files and one of them should be rna_r9.4.1_70bps_hac.cfg. The output seems not to have been uploaded.

[hanyishui0612]

thank you for your answer！I re-decompressed the guppy compressed package and find rna_r9.4.1_70bps_hac.cfg .I run the command is as follows:

(venv3) (base) ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac.cfg --device 'cuda:all' -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads

the output as follows:

(venv3) (base) ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac.cfg -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads trying port 5000 Using port 5000 Could not load pybasecall, trying for version earlier versions <=7.2.15 pyguppy lib

           ~  buttery-eel - SLOW5 Guppy/Dorado Server Basecalling  ~
                        version: 0.5.0

========================================================================== ARGS

args: Namespace(input='/mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5', output='DRS-WT-rep1_DRS-WT-rep1.reads', basecaller_bin=PosixPath('/home/ashui/Downloads/ont-guppy_6.4.2/bin'), config='/home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac.cfg', call_mods=False, qscore=None, slow5_threads=4, procs=4, slow5_batchsize=4000, quiet=False, max_read_queue_size=20000, log='buttery_basecaller_logs', moves_out=False, do_read_splitting=False, min_score_read_splitting=50.0, detect_adapter=False, min_score_adapter=60.0, trim_adapters=False, detect_mid_strand_adapter=False, seq_sum=False, barcode_kits=None, enable_trim_barcodes=False, require_barcodes_both_ends=False, detect_mid_strand_barcodes=False, min_score_barcode_front=60.0, min_score_barcode_rear=60.0, min_score_barcode_mid=60.0, profile=False, duplex=False, single=False, above_7310=False, above_7412=False) ['--port', '5000', '--use_tcp']

========================================================================== Starting Guppy/Dorado Basecalling Server

Setting params... Connecting... [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac [guppy/error] basecall_service::BasecallClient::worker_loop: Connection error. [basecall_config_unavailable] could not retrieve details for config from server: /home/ashui/Downloads/ont-guppy/data/rna_r9.4.1_70bps_hac Traceback (most recent call last): File "/home/ashui/Software/buttery-eel/venv3/bin/buttery-eel", line 8, in sys.exit(main()) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/buttery_eel.py", line 139, in main with start_guppy_server_and_client(args, other_server_args) as client_one: File "/home/ashui/miniforge3/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/basecaller.py", line 133, in start_guppy_server_and_client with client: File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/ont_pyguppy_client_lib-6.4.2-py3.10-linux-x86_64.egg/pyguppy_client_lib/pyclient.py", line 379, in enter self.connect() File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/ont_pyguppy_client_lib-6.4.2-py3.10-linux-x86_64.egg/pyguppy_client_lib/pyclient.py", line 184, in connect raise exception_type(exception_message) ValueError: Basecalling config unavailable: <result.basecall_config_unavailable: 6>

Do you know what should I do ?

[hasindu2008]

OK, we are in the right direction now. Could you copy paste the following command now?

 ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config rna_r9.4.1_70bps_hac.cfg --device 'cuda:all' -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads.fastq

[hanyishui0612]

I can .what should i do ? thank you for your answer

[Psy-Fer]

Hey,

When giving the config file you only have to give the name with the .cfg extension, not the full path to the file.

That seems to be the main issue here.

James

[hasindu2008]

@hanyishui0612

Did using the command ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config rna_r9.4.1_70bps_hac.cfg --device 'cuda:all' -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads.fastq solve the issue and did it work?

[hanyishui0612]

When I run: (venv3) (base) ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config rna_r9.4.1_70bps_hac.cfg -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads I met the error as follows: `trying port 5000 Using port 5000 Could not load pybasecall, trying for version earlier versions <=7.2.15 pyguppy lib

           ~  buttery-eel - SLOW5 Guppy/Dorado Server Basecalling  ~
                        version: 0.5.0

========================================================================== ARGS

args: Namespace(input='/mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5', output='DRS-WT-rep1_DRS-WT-rep1.reads', basecaller_bin=PosixPath('/home/ashui/Downloads/ont-guppy_6.4.2/bin'), config='rna_r9.4.1_70bps_hac.cfg', call_mods=False, qscore=None, slow5_threads=4, procs=4, slow5_batchsize=4000, quiet=False, max_read_queue_size=20000, log='buttery_basecaller_logs', moves_out=False, do_read_splitting=False, min_score_read_splitting=50.0, detect_adapter=False, min_score_adapter=60.0, trim_adapters=False, detect_mid_strand_adapter=False, seq_sum=False, barcode_kits=None, enable_trim_barcodes=False, require_barcodes_both_ends=False, detect_mid_strand_barcodes=False, min_score_barcode_front=60.0, min_score_barcode_rear=60.0, min_score_barcode_mid=60.0, profile=False, duplex=False, single=False, above_7310=False, above_7412=False) ['--port', '5000', '--use_tcp']

========================================================================== Starting Guppy/Dorado Basecalling Server

[guppy/error] guppy::CommandLineProcessor::create_configuration: Missing required parameter basecalling.model_file --model_file Traceback (most recent call last): File "/home/ashui/Software/buttery-eel/venv3/bin/buttery-eel", line 8, in sys.exit(main()) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/buttery_eel.py", line 139, in main with start_guppy_server_and_client(args, other_server_args) as client_one: File "/home/ashui/miniforge3/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/basecaller.py", line 120, in start_guppy_server_and_client raise RuntimeError("Server couldn't be started") RuntimeError: Server couldn't be started ` Could you tell me what to do ?

[hasindu2008]

@Psy-Fer any idea about this error?

[hanyishui0612]

Hey,

When giving the config file you only have to give the name with the .cfg extension, not the full path to the file.

That seems to be the main issue here.

James

Hi, I do as you say and change it to the name with the .cfg extension,not the full path to the file .But I still met the error as follows : `(venv3) (base) ./eel -g /home/ashui/Downloads/ont-guppy_6.4.2/bin --config rna_r9.4.1_70bps_hac.cfg -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_DRS-WT-rep1.reads trying port 5000 Using port 5000 Could not load pybasecall, trying for version earlier versions <=7.2.15 pyguppy lib

           ~  buttery-eel - SLOW5 Guppy/Dorado Server Basecalling  ~
                        version: 0.5.0

========================================================================== ARGS

args: Namespace(input='/mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5', output='DRS-WT-rep1_DRS-WT-rep1.reads', basecaller_bin=PosixPath('/home/ashui/Downloads/ont-guppy_6.4.2/bin'), config='rna_r9.4.1_70bps_hac.cfg', call_mods=False, qscore=None, slow5_threads=4, procs=4, slow5_batchsize=4000, quiet=False, max_read_queue_size=20000, log='buttery_basecaller_logs', moves_out=False, do_read_splitting=False, min_score_read_splitting=50.0, detect_adapter=False, min_score_adapter=60.0, trim_adapters=False, detect_mid_strand_adapter=False, seq_sum=False, barcode_kits=None, enable_trim_barcodes=False, require_barcodes_both_ends=False, detect_mid_strand_barcodes=False, min_score_barcode_front=60.0, min_score_barcode_rear=60.0, min_score_barcode_mid=60.0, profile=False, duplex=False, single=False, above_7310=False, above_7412=False) ['--port', '5000', '--use_tcp']

========================================================================== Starting Guppy/Dorado Basecalling Server

[guppy/error] guppy::CommandLineProcessor::create_configuration: Missing required parameter basecalling.model_file --model_file Traceback (most recent call last): File "/home/ashui/Software/buttery-eel/venv3/bin/buttery-eel", line 8, in sys.exit(main()) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/buttery_eel.py", line 139, in main with start_guppy_server_and_client(args, other_server_args) as client_one: File "/home/ashui/miniforge3/lib/python3.10/contextlib.py", line 135, in enter return next(self.gen) File "/home/ashui/Software/buttery-eel/venv3/lib/python3.10/site-packages/src/basecaller.py", line 120, in start_guppy_server_and_client raise RuntimeError("Server couldn't be started") RuntimeError: Server couldn't be started` what should i do ? Looking forward to your reply!

[hanyishui0612]

I run buttery-eel -g /home/ashui/Downloads/ont-guppy/bin/ --config rna_r9.4.1_70bps_hac.cfg -i /mnt/data/home/malab001/WorkDir/DRS-WT-rep1/signals_time.blow5 -o DRS-WT-rep1_buttery_eel.fastq--port 5566 --use_tcp --slow5_threads 10 . I does work，but I encountered another problem as follows: Skipped a read Can you tell me if it has any impact? If so, what should I do?

[hanyishui0612]

In the process of using buttery basecalling，I checked the output fastq file, the fastq file is empty, I don't know what to do, looking forward to your answer.

[Psy-Fer]

Hey,

Glad to see you could get it working.

Skipped reads can happen for a few reasons, and they are handled differently in different versions of the basecaller.

In this particular case, it looks like it's happening in the read submission phase of the basecalling, where the client is trying to send the read to the basecaller.

It will try to send the read 1000 times, with a throttle of 0.01s between each try. So about 10s.

            if tries > 1:
                time.sleep(client.throttle)
            tries += 1
            if tries >= 1000:
                if not result:
                    print("Skipped a read: {}".format(read_id))
                    skipped.append([read_id, "stage-0", "timed out trying to submit read to client"])
                    break

This looks like it's taking a long time for the server to load up to be able to communicate with the clients. What kind of system are you running this on?

If there are ever any skipped reads, at the end of the run, there will be a file called skipped_reads.txt written to your output location. In this case it will be read_id, stage-0, timed out trying to submit read to client

A solution to this can be increasing the number of tries to 2000 or 5000 to give it more time to load up the server. You can try this by modifying the number in the basecaller.py on line 190 then doing a pip install . again to rebuild the package.

If this works, then I'll make the change globally to increase the time to avoid this error in the future.

Cheers, James

[hanyishui0612]

Hey,

Glad to see you could get it working.

Skipped reads can happen for a few reasons, and they are handled differently in different versions of the basecaller.

In this particular case, it looks like it's happening in the read submission phase of the basecalling, where the client is trying to send the read to the basecaller.

It will try to send the read 1000 times, with a throttle of 0.01s between each try. So about 10s.

            if tries > 1:
                time.sleep(client.throttle)
            tries += 1
            if tries >= 1000:
                if not result:
                    print("Skipped a read: {}".format(read_id))
                    skipped.append([read_id, "stage-0", "timed out trying to submit read to client"])
                    break

This looks like it's taking a long time for the server to load up to be able to communicate with the clients. What kind of system are you running this on?

If there are ever any skipped reads, at the end of the run, there will be a file called skipped_reads.txt written to your output location. In this case it will be read_id, stage-0, timed out trying to submit read to client

A solution to this can be increasing the number of tries to 2000 or 5000 to give it more time to load up the server. You can try this by modifying the number in the basecaller.py on line 190 then doing a pip install . again to rebuild the package.

If this works, then I'll make the change globally to increase the time to avoid this error in the future.

Cheers, James

Hi,My system is manjaro.I changed the number of tries to 5000.But the same problem still occurs.And my guppy version is 6.4.2. Could you tell me what should I do ?

[Psy-Fer]

Sorry, I meant what kind of hardware are you using to run buttery-eel.

Please complete this list

OS: Manjaro CPU: RAM: GPU: Type of disk blow5 data is on (HDD/SSD/NVME/TAPE)

Thanks James

[hanyishui0612]

OS: 6.9.9-1-Manjaro CPU:8 ×Intel Core i7-4770K CPU @3.50GHz RAM:31.2GiB of RAM GPU:NVIDIA GeForce GTX TITAN X/PCIe/SSE2 Type of disk blow5 data is on (HDD/SSD/NVME/TAPE):HDD looking forward to your reply!thank you ~~~

[Psy-Fer]

Ahh, That GTX TITAN X only has a cuda compute capability of 5.2

But the basecallers require a compute capability of 6.1

https://developer.nvidia.com/cuda-gpus https://nanoporetech.com/document/requirements/minion-it-reqs

Do you have access to a system with a GPU that meets the >=6.1 compute capability?

James

[Psy-Fer]

@hasindu2008 do you know if the TITAN X can basecall with guppy?

I've never tested them, as they were pretty old when guppy was released.

[Psy-Fer]

Ahh nevermind. I confirmed with someone from ONT and TITAN X won't work with guppy or dorado (not without re-compiling it and changing some things, which isn't possible because it's closed source)

So yea, you can't basecall nanopore data with a TITAN X card.

[hanyishui0612]

thank you for your reply.I will change the hardware.

[hanyishui0612]

Ahh nevermind. I confirmed with someone from ONT and TITAN X won't work with guppy or dorado (not without re-compiling it and changing some things, which isn't possible because it's closed source)

So yea, you can't basecall nanopore data with a TITAN X card.

I try to change the hardware and solve it !Thank you!!!!All the best to you!