Playback error

[jennieli421]

Hello,

I have encountered an issue with running a simulation in MinKNOW. This is what I did:

• add the simulation = line to “sequencing_MIN106_DNA.toml”
• reload script in minknow API
• in the terminal, connect to Guppy by running guppy_basecall_server --config dna_r9.4.1_450bps_fast.cfg -p 5555 -l /tmp/guppy -x 'cuda:0' (this might not be necessary)
• connect the configuration test flow cell to the desktop.
• select flow cell type as FLO-MIN106, a random kit, and hit start.
• The flow cell temperature begin to rise, once this step is finished, the run stops and it says "run error".

Please let me know what you think might be causing this error. Thank you very much.

[Psy-Fer]

Hello,

If you look in your logs, it might have some more information. Try looking somewhere like /var/log/minknow/ and look at the latest modified files/position.

Does your simulation = line have the path to your bulk fast5 file in there correctly?

Otherwise, it could be while using the configuration test cell. I've never tried to do a simulation with that before.

[jennieli421]

Thanks for replying! I will check the logs once I get a chance. I've checked that the path is correct, and used h5dump to inspect the file looks normal. It could be the problem of using a test cell. But a tutorial I have been following says "Insert a configuration test flowcell into the sequencing device" (https://github.com/LooseLab/readfish). I am also hesitant to use a real flow cell before I know things can work. Can I use a used one for testing purposes?

[hasindu2008]

Hi

Perhaps you gave a BLOW5 file to the simulation - I thought so because this issue is opened under slow5tools? simulation = should point to a bulk-FAST5 file, not a BLOW5 file or a regular FAST5. Bulk-FAST5 is quite different to regular FAST5 files. Readfish repository has a link to an example bulk-FAST5 file you can use. If you have an old used flowcell, you can use that too, instead of a new flowcell. I remember having some trouble with the config test cell with bulk-FAST5, 2 years ago, so cannot exactly remember what the issue was.

[Psy-Fer]

Yea, and if you don't have any old flowcells, you can also simulate positions too and don't even need to plug a device in. Handy if benchmarking 100 MinION runs at once 🤣

[jennieli421]

Thanks! Good to know the difference between bulk- and regular fast5, I am using the bulk-fast5 provided in Readfish repo. I will try with an old flowcell when I can get one. Also, @Psy-Fer do you mind sharing share a tutorial on how to "simulate positions"? Couldn't find useful info online.

Again, thanks very much for helping out and being patient with a beginner.

[Psy-Fer]

Hey,

yea sure.

Just don't run this while you are running other flowcells or anything. I use this when testing real-time applications on my laptop without access to a minion.

This is also assuming running on linux

# in a separate terminal that you don't close until you are done simulating run the following

# stop the minkow service
sudo systemctl stop minknow.service

# start a simulated integrated device (like on a GridION)
sudo /opt/ont/minknow/bin/mk_manager_svc -d --print-rpc-ports --simulated-integrated-devices 1

Refresh your minknow and you should now see a new device called something like M00001

Have fun :)

[jennieli421]

Amazing! It is working very well! A follow up: For "simulate position" function, will it be simulating using the fast5 I specified? Is there anyway to make sure?

Also, How should I reset back to the normal mode? I noticed that under "connection manager" there is no "saved hosts". Don't remember editing this page when I installed minknow so not sure how to add it or find Mk1c ID.

[jennieli421]

When I try to run the simulation again, I got this issue:

$ sudo systemctl stop minknow.service $ sudo /opt/ont/minknow/bin/mk_manager_svc -d --print-rpc-ports --simulated-integrated-devices 1 Port 9502 appears to be in use, retrying for 2 minutes as it might be in TIME_WAIT state and will become available shortly. Port 9502 appears to be in use. Is there another instance of MinKNOW running?

Rebooting the desktop appears to fix this issue.

[jennieli421]

In terms of simulation using an old flowcell, it also produces "run error". I checked the most recently modified log file:

2023-05-29 17:00:12.129826    INFO: starting_up (control)
    hostname: LX310327
    system: ubuntu 22.04
            Distribution:           23.04.5 (STABLE)
            MinKNOW Core:           5.5.3
            Bream:                  7.5.9
            Protocol configuration: 5.5.13
            Guppy (build):          6.5.5+3a1e121
            Guppy (connected):      6.5.7+ca6d6af

2023-05-29 17:00:12.138173 WARNING: partially_raised_file_limits (analyser)
    actual: 1024
    target: 4096
2023-05-29 17:06:20.364178    INFO: finished (analyser)
/var/log/minknow/MN34448/analyser_log-0.txt (END)

Any new ideas on what is going on?

Also appending the head of log generated from running position simulation:

2023-05-29 16:41:32.443116    INFO: starting_up (control)
    hostname: LX310327
    system: ubuntu 22.04
            Distribution:           23.04.5 (STABLE)
            MinKNOW Core:           5.5.3
            Bream:                  7.5.9
            Protocol configuration: 5.5.13
            Guppy (build):          6.5.5+3a1e121
            Guppy (connected):      6.5.7+ca6d6af

2023-05-29 16:41:32.451353 WARNING: partially_raised_file_limits (analyser)
    actual: 1024
    target: 4096
2023-05-29 16:42:14.143543    INFO: switching_to_guppy_config (basecalling)
    new_filename: dna_r9.4.1_450bps_fast.cfg
2023-05-29 16:42:14.144469    INFO: guppy_connection_changed_state (basecalling)
    client_name: MS00000_b5b77cae-b1f8-41ef-acc4-5c275a26d6ef
    guppy_error_string: [no_error] 
    next_state: preparing_connection (1)
    previous_state: disconnected (0)
2023-05-29 16:42:14.144769    INFO: guppy_connection_changed_state (basecalling)
    client_name: MS00000_b5b77cae-b1f8-41ef-acc4-5c275a26d6ef
    guppy_error_string: [no_error] 
    next_state: loading_config (2)
    previous_state: preparing_connection (1)
2023-05-29 16:42:14.145785    INFO: guppy_connection_changed_state (basecalling)
    client_name: MS00000_b5b77cae-b1f8-41ef-acc4-5c275a26d6ef
    guppy_error_string: [no_error] 
    next_state: loading_alignment_index (3)
    previous_state: loading_config (2)

[hasindu2008]

@jennieli421 where do you see the "run error"? On Minknow, under the messages tab is there anything useful about the error?

Check the most recent of each of the all the following logs:

1. /var/log/minknow/basecall_manager_log-*.txt
2. /var/log/minknow/basecall_manager_log-*.txt
3. /var/log/minknow/MN34448/bream-*.txt
4. /var/log/minknow/MN34448/control_server_log-*.txt

One of those will have the error. Can you also double-check if you have given the path to the bulk-FAST5 correctly in the toml file - I remember due to a typo I had a similar trouble some time ago.

Also @alexomics and @mattloose mattloose might want to comment about this, but I see they are perhaps too busy https://github.com/LooseLab/readfish/issues/240

[jennieli421]

In /var/log/minknow/MN34448/control_server_log-0.txt:

023-05-29 17:06:20.125127 WARNING: hdf5_error (fast5)
    cause: unable to open file: name = '/home/readuntil/readfish_files/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0
    function: H5Fopen
    message: unable to open file
    stack_trace: 
                   #000: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5F.c line 620 in H5Fopen(): unable to open file
                     major: File accessibility
                     minor: Unable to open file

2023-05-29 17:06:20.126132   ERROR: exception_during_observer_callback (state_control)
    callback_mode: transitioning
    cause: Dynamic exception type: boost::filesystem::filesystem_error
           std::exception::what: boost::filesystem::status: Permission denied [system:13]: "/home/readuntil/readfish_files/PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5"

Then I changed the permission of the fast5 file by running chmod 777 PLSP57501_20170308_FNFAF14035_MN16458_sequencing_run_NOTT_Hum_wh1rs2_60428.fast5

Tried simulation using a minion device, still the same run error. the minknow message says "MinION Mk1B · MN19362 Stopping protocol due to internal error. All data will be saved."

In the control_server_log-0.txt of MN19362, I see a different error:

2023-06-01 14:40:26.304513 WARNING: hdf5_error (fast5)
    cause: object 'duty_time' doesn't exist
    function: H5Gopen2
    message: unable to open group
    stack_trace: 
                   #000: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5G.c line 510 in H5Gopen2(): unable to open group
                     major: Symbol table
                     minor: Can't open object
                   #001: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5VLcallback.c line 4187 in H5VL_group_open(): group open failed
                     major: Virtual Object Layer
                     minor: Can't open object
                   #002: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5VLcallback.c line 4154 in H5VL__group_open(): group open failed
                     major: Virtual Object Layer
                     minor: Can't open object
                   #003: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5VLnative_group.c line 123 in H5VL__native_group_open(): unable to open group
                     major: Symbol table
                     minor: Can't open object
                   #004: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5Gint.c line 270 in H5G__open_name(): group not found
                     major: Symbol table
                     minor: Object not found
                   #005: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5Gloc.c line 442 in H5G_loc_find(): can't find object
                     major: Symbol table
                     minor: Object not found
                   #006: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5Gtraverse.c line 837 in H5G_traverse(): internal path traversal failed
                     major: Symbol table
                     minor: Object not found
                   #007: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5Gtraverse.c line 613 in H5G__traverse_real(): traversal operator failed
                     major: Symbol table
                     minor: Callback failed
                   #008: /builds/informatics/conan-hdf5/9721449/.conan/data/hdf5/1.12.1/nanopore/stable/build/4cb52770e5d65c257e2250813532b9e31047d0eb/source_subfolder/src/H5Gloc.c line 399 in H5G__loc_find_cb(): object 'duty_time' doesn't exist
                     major: Symbol table
                     minor: Object not found
2023-06-01 14:40:26.305008 WARNING: hdf5_error (fast5)
    cause: object 'read_length_histograms' doesn't exist
    function: H5Gopen2
    message: unable to open group
    stack_trace: 

[hasindu2008]

Was this issue fixed @jennieli421?

[jennieli421]

Thanks for following up! Just fixed the playback error yesterday, as described in this thread.

Thank you again, and I'll close this issue.