search

LooseLab / readfish

CLI tool for flexible and fast adaptive sampling on ONT sequencers
https://looselab.github.io/readfish/
GNU General Public License v3.0
169 stars 33 forks source link

Handshake failed with fatal error SSL_ERROR_SSL #193

Closed syncrotron closed 2 years ago

syncrotron commented 2 years ago

I am having some issues getting readfish up and running on Ubuntu 20.04. To start, the following is the error I am getting when I run an the command readfish unblock-all --device MN26243 --port 5555 --experiment-name "Testing ReadFish Unblock All":

INFO 2022-04-25 15:08:54,850 Manager /home/shorovatin/miniconda3/envs/readfish/bin/readfish unblock-all --device MN26243 --port 5555 --experiment-name Testing ReadFish Unblock All
2022-04-25 15:08:54,850 Manager /home/shorovatin/miniconda3/envs/readfish/bin/readfish unblock-all --device MN26243 --port 5555 --experiment-name Testing ReadFish Unblock All
INFO 2022-04-25 15:08:54,851 Manager batch_size=512
2022-04-25 15:08:54,851 Manager batch_size=512
INFO 2022-04-25 15:08:54,851 Manager cache_size=512
2022-04-25 15:08:54,851 Manager cache_size=512
INFO 2022-04-25 15:08:54,851 Manager channels=[1, 512]
2022-04-25 15:08:54,851 Manager channels=[1, 512]
INFO 2022-04-25 15:08:54,851 Manager command=unblock-all
2022-04-25 15:08:54,851 Manager command=unblock-all
INFO 2022-04-25 15:08:54,851 Manager device=MN26243
2022-04-25 15:08:54,851 Manager device=MN26243
INFO 2022-04-25 15:08:54,851 Manager dry_run=False
2022-04-25 15:08:54,851 Manager dry_run=False
INFO 2022-04-25 15:08:54,851 Manager experiment_name=Testing ReadFish Unblock All
2022-04-25 15:08:54,851 Manager experiment_name=Testing ReadFish Unblock All
INFO 2022-04-25 15:08:54,851 Manager func=<function run at 0x7f4200c12d40>
2022-04-25 15:08:54,851 Manager func=<function run at 0x7f4200c12d40>
INFO 2022-04-25 15:08:54,851 Manager host=127.0.0.1
2022-04-25 15:08:54,851 Manager host=127.0.0.1
INFO 2022-04-25 15:08:54,851 Manager log_file=None
2022-04-25 15:08:54,851 Manager log_file=None
INFO 2022-04-25 15:08:54,851 Manager log_format=%(asctime)s %(name)s %(message)s
2022-04-25 15:08:54,851 Manager log_format=%(asctime)s %(name)s %(message)s
INFO 2022-04-25 15:08:54,851 Manager log_level=info
2022-04-25 15:08:54,851 Manager log_level=info
INFO 2022-04-25 15:08:54,851 Manager port=5555
2022-04-25 15:08:54,851 Manager port=5555
INFO 2022-04-25 15:08:54,851 Manager run_time=172800
2022-04-25 15:08:54,851 Manager run_time=172800
INFO 2022-04-25 15:08:54,851 Manager throttle=0.4
2022-04-25 15:08:54,851 Manager throttle=0.4
INFO 2022-04-25 15:08:54,851 Manager unblock_duration=0.1
2022-04-25 15:08:54,851 Manager unblock_duration=0.1
INFO 2022-04-25 15:08:54,851 Manager workers=1
2022-04-25 15:08:54,851 Manager workers=1
DEBUG 2022-04-25 15:08:54,851 minknow_api Reading ssl certificate
DEBUG 2022-04-25 15:08:54,851 minknow_api Getting local token
E0425 15:08:54.854633053   84112 ssl_transport_security.cc:1504] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
DEBUG 2022-04-25 15:08:54,855 root Unable to connect to manager on port '5555' to retrieve local auth token path
Traceback (most recent call last):
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager.py", line 51, in get_local_authentication_token_file
    return service.local_authentication_token_path().path
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager_service.py", line 1098, in local_authentication_token_path
    "minknow_api.manager.ManagerService")
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager_service.py", line 85, in run_with_retry
    result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1650920934.854883295","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1650920934.854882332","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"
>
DEBUG 2022-04-25 15:08:54,856 minknow_api Getting protocol token
DEBUG 2022-04-25 15:08:54,856 minknow_api No protocol token found
E0425 15:08:54.856732083   84115 ssl_transport_security.cc:1504] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
Traceback (most recent call last):
  File "/home/shorovatin/miniconda3/envs/readfish/bin/readfish", line 8, in <module>
    sys.exit(main())
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/ru/cli.py", line 43, in main
    args.func(parser, args)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/ru/unblock_all.py", line 127, in run
    position = get_device(args.device, host=args.host, port=args.port)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/ru/utils.py", line 918, in get_device
    manager = Manager(host=host, port=port)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager.py", line 155, in __init__
    version_info = self.rpc.get_version_info()
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager_service.py", line 425, in get_version_info
    "minknow_api.manager.ManagerService")
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/minknow_api/manager_service.py", line 85, in run_with_retry
    result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/shorovatin/miniconda3/envs/readfish/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1650920934.856815736","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3128,"referenced_errors":[{"created":"@1650920934.856815382","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}"

As some background to the environment I am running, after installing guppy on GPU (I have tried guppy versions 6.0.6, 6.0.7, and 6.1.2 with the appropriate associated ont-pyguppy-client-lib version without any change in error) I followed the conda setup suggested in 187 with the appropriate associated ont-pyguppy-client-lib version. Guppy is running in tcp mode on port 5555 to allow interplay between windows and Ubuntu, however all readfish work is being done on Ubuntu. My environment is as follows:

conda list
# packages in environment at /home/shorovatin/miniconda3/envs/readfish:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
attrs                     21.4.0                   pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
biopython                 1.76                     pypi_0    pypi
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.10.8                pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
google                    3.0.0                    pypi_0    pypi
grpcio                    1.44.0                   pypi_0    pypi
idna                      3.3                      pypi_0    pypi
importlib-metadata        4.11.3                   pypi_0    pypi
importlib-resources       5.7.1                    pypi_0    pypi
jsonschema                4.4.0                    pypi_0    pypi
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_16    conda-forge
libgomp                   11.2.0              h1d223b6_16    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_16    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
mappy                     2.24                     pypi_0    pypi
minknow-api               5.0.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.17.4                   pypi_0    pypi
ont-pyguppy-client-lib    6.1.2                    pypi_0    pypi
openssl                   3.0.2                h166bdaf_1    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    1.3.5                    pypi_0    pypi
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
protobuf                  3.20.1                   pypi_0    pypi
pyparsing                 3.0.8                    pypi_0    pypi
pyrfc3339                 1.1                      pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
python                    3.7.12          hf930737_100_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1                   pypi_0    pypi
read-until                3.0.0                    pypi_0    pypi
readfish                  0.0.9a3                  pypi_0    pypi
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1                   pypi_0    pypi
setuptools                62.1.0           py37h89c1867_0    conda-forge
six                       1.16.0                   pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
sqlite                    3.38.2               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2                   pypi_0    pypi
typing-extensions         4.2.0                    pypi_0    pypi
urllib3                   1.26.9                   pypi_0    pypi
watchdog                  2.1.7                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.11            h166bdaf_1014    conda-forge

My non-conda curated environment versions are: MinKNOW core = 5.0.0, Guppy = 6.1.2, Bream = 7.0.9, and Script Configuration = 5.0.8.

Finally, as a sanity check to make sure guppy can be communicated to, I ran the following python command with it returning nominally:

python -c 'from pyguppy_client_lib.pyclient import PyGuppyClient as PGC; c = PGC("127.0.0.1:5555", "dna_r9.4.1_450bps_fast.cfg"); c.connect(); print(c)'
PyGuppyClient(address='127.0.0.1:5555', config='dna_r9.4.1_450bps_fast', align_ref=None, bed_file=None, barcodes=None, status.connected, )

Any debugging advice would would be greatly appreciated.

alexomics commented 2 years ago

Hi Sam, I think that there's some confusion on the --port option for the unblock all command. If you don't provide it then we will use the default MinKNOW port (9502).

For the Guppy issue, I would expect that you would need to be using the ipc://... host format. If this guppy installation shipped with MinKNOW then the location of of the socket should be: ipc:///tmp/.guppy/5555. If you are running your own Guppy server then this will be different.

Let me know if you run into any other issues

syncrotron commented 2 years ago

Ahh, that would do it: It appears I have unblock all and the chrom21 and chrom22 experiments working now. For those of you who had the same issue I had reading this in the future: To solve the above issue, do not use tcp (flags within both the guppyd and minknow app_conf), instead opting for the ipc connection (on ubuntu/linux). Once I reverted the tcp flags, I restarted everything, unblock all worked as expected! For those following the tutorial in the readme, one additional step I took was to edit the host and port fields in the readfish selection toml to point at ipc:///tmp/.guppy at port 5555. After these edits were done, and I had a simulation run up, readfish functioned as expected.

Thank you for the help @alexomics, I will close this issue.