search

LooseLab / readfish

CLI tool for flexible and fast adaptive sampling on ONT sequencers
https://looselab.github.io/readfish/
GNU General Public License v3.0
167 stars 31 forks source link

Cannot connect to minknow remotely - Connection refused #244

Closed cmfield closed 1 year ago

cmfield commented 1 year ago

We have a Minion Mk1C running Minknow 23.04.5 (bionic) (minknow-core-minion-1c 5.5.3) and another machine running readfish bossruns 0.0.2 in a suitably configured environment.

After disabling "Remote user permission (Nanopore account required)" and double checking this was the case on the device itself with config_editor, we are still unable to connect readfish to the device for the simple Unblocking Test.

I have attached the error log. Any ideas? Is it something to do with API access tokens - do we need to set that up?

Thanks!

INFO 2023-06-30 14:52:16,328 Manager /science/fieldc/bossruns/bin/readfish unblock-all --device MC-113514 --experiment-name ReadfishUnblock --host 172.31.0.126
2023-06-30 14:52:16,328 Manager /science/fieldc/bossruns/bin/readfish unblock-all --device MC-113514 --experiment-name ReadfishUnblock --host 172.31.0.126
INFO 2023-06-30 14:52:16,328 Manager batch_size=512
2023-06-30 14:52:16,328 Manager batch_size=512
INFO 2023-06-30 14:52:16,328 Manager cache_size=512
2023-06-30 14:52:16,328 Manager cache_size=512
INFO 2023-06-30 14:52:16,328 Manager channels=[1, 512]
2023-06-30 14:52:16,328 Manager channels=[1, 512]
INFO 2023-06-30 14:52:16,328 Manager command=unblock-all
2023-06-30 14:52:16,328 Manager command=unblock-all
INFO 2023-06-30 14:52:16,328 Manager device=MC-113514
2023-06-30 14:52:16,328 Manager device=MC-113514
INFO 2023-06-30 14:52:16,329 Manager dry_run=False
2023-06-30 14:52:16,329 Manager dry_run=False
INFO 2023-06-30 14:52:16,329 Manager experiment_name=ReadfishUnblock
2023-06-30 14:52:16,329 Manager experiment_name=ReadfishUnblock
INFO 2023-06-30 14:52:16,329 Manager func=<function run at 0x7f3dbe52e8b0>
2023-06-30 14:52:16,329 Manager func=<function run at 0x7f3dbe52e8b0>
INFO 2023-06-30 14:52:16,329 Manager host=172.31.0.126
2023-06-30 14:52:16,329 Manager host=172.31.0.126
INFO 2023-06-30 14:52:16,329 Manager log_file=None
2023-06-30 14:52:16,329 Manager log_file=None
INFO 2023-06-30 14:52:16,329 Manager log_format=%(asctime)s %(name)s %(message)s
2023-06-30 14:52:16,329 Manager log_format=%(asctime)s %(name)s %(message)s
INFO 2023-06-30 14:52:16,329 Manager log_level=info
2023-06-30 14:52:16,329 Manager log_level=info
INFO 2023-06-30 14:52:16,329 Manager port=None
2023-06-30 14:52:16,329 Manager port=None
INFO 2023-06-30 14:52:16,329 Manager run_time=172800
2023-06-30 14:52:16,329 Manager run_time=172800
INFO 2023-06-30 14:52:16,329 Manager throttle=0.4
2023-06-30 14:52:16,329 Manager throttle=0.4
INFO 2023-06-30 14:52:16,329 Manager unblock_duration=0.1
2023-06-30 14:52:16,329 Manager unblock_duration=0.1
INFO 2023-06-30 14:52:16,329 Manager workers=1
2023-06-30 14:52:16,329 Manager workers=1
DEBUG 2023-06-30 14:52:16,329 minknow_api Reading ssl certificate
DEBUG 2023-06-30 14:52:16,332 minknow_api Getting protocol token
DEBUG 2023-06-30 14:52:16,332 minknow_api No protocol token found
DEBUG 2023-06-30 14:52:16,391 minknow_api Reading ssl certificate
DEBUG 2023-06-30 14:52:16,391 minknow_api Getting local token
DEBUG 2023-06-30 14:52:16,394 root Unable to connect to manager on port '9502' to retrieve local auth token path
Traceback (most recent call last):
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/manager.py", line 51, in get_local_authentication_token_file
    return service.local_authentication_token_path().path
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/manager_service.py", line 1129, in local_authentication_token_path
    return run_with_retry(self._stub.local_authentication_token_path,
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/manager_service.py", line 97, in run_with_retry
    result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Failed to connect to remote host: Connection refused"
    debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-06-30T14:52:16.394095486+02:00"}"
>
DEBUG 2023-06-30 14:52:16,395 minknow_api Retrieving local token from file: 'None'
DEBUG 2023-06-30 14:52:16,396 minknow_api Getting protocol token
DEBUG 2023-06-30 14:52:16,396 minknow_api No protocol token found
DEBUG 2023-06-30 14:52:16,398 minknow_api Calling get_version_info to test connection
INFO 2023-06-30 14:52:16,399 minknow_api Error received from rpc
2023-06-30 14:52:16,399 minknow_api Error received from rpc
WARNING 2023-06-30 14:52:16,399 minknow_api Failed to connect to minknow instance (retry 1/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
2023-06-30 14:52:16,399 minknow_api Failed to connect to minknow instance (retry 1/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
DEBUG 2023-06-30 14:52:16,901 minknow_api Calling get_version_info to test connection
INFO 2023-06-30 14:52:16,902 minknow_api Error received from rpc
2023-06-30 14:52:16,902 minknow_api Error received from rpc
WARNING 2023-06-30 14:52:16,902 minknow_api Failed to connect to minknow instance (retry 2/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
2023-06-30 14:52:16,902 minknow_api Failed to connect to minknow instance (retry 2/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
DEBUG 2023-06-30 14:52:17,404 minknow_api Calling get_version_info to test connection
INFO 2023-06-30 14:52:17,405 minknow_api Error received from rpc
2023-06-30 14:52:17,405 minknow_api Error received from rpc
WARNING 2023-06-30 14:52:17,405 minknow_api Failed to connect to minknow instance (retry 3/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
2023-06-30 14:52:17,405 minknow_api Failed to connect to minknow instance (retry 3/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
DEBUG 2023-06-30 14:52:17,907 minknow_api Calling get_version_info to test connection
INFO 2023-06-30 14:52:17,908 minknow_api Error received from rpc
2023-06-30 14:52:17,908 minknow_api Error received from rpc
WARNING 2023-06-30 14:52:17,908 minknow_api Failed to connect to minknow instance (retry 4/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
2023-06-30 14:52:17,908 minknow_api Failed to connect to minknow instance (retry 4/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
DEBUG 2023-06-30 14:52:18,410 minknow_api Calling get_version_info to test connection
INFO 2023-06-30 14:52:18,411 minknow_api Error received from rpc
2023-06-30 14:52:18,411 minknow_api Error received from rpc
WARNING 2023-06-30 14:52:18,411 minknow_api Failed to connect to minknow instance (retry 5/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
2023-06-30 14:52:18,411 minknow_api Failed to connect to minknow instance (retry 5/5): failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused
ERROR 2023-06-30 14:52:18,912 root Failed to connect to read until at 172.31.0.126: 8000
2023-06-30 14:52:18,912 root Failed to connect to read until at 172.31.0.126: 8000
Traceback (most recent call last):
  File "/science/fieldc/bossruns/bin/readfish", line 8, in <module>
    sys.exit(main())
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/ru/cli.py", line 44, in main
    args.func(parser, args)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/ru/unblock_all.py", line 129, in run
    read_until_client = RUClient(
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/ru/read_until_client.py", line 19, in __init__
    super().__init__(*args, **kwargs)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/read_until/base.py", line 137, in __init__
    self.connection = Connection(self.mk_grpc_port)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/__init__.py", line 575, in __init__
    raise error
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/__init__.py", line 547, in __init__
    self.instance.get_version_info()
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/instance_service.py", line 96, in get_version_info
    return run_with_retry(self._stub.get_version_info,
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/minknow_api/instance_service.py", line 40, in run_with_retry
    result = MessageWrapper(method(message, timeout=timeout), unwraps=unwraps)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/science/fieldc/bossruns/lib/python3.8/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused"
    debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8000: Failed to connect to remote host: Connection refused {created_time:"2023-06-30T14:52:18.41139061+02:00", grpc_status:14}"
>
Adoni5 commented 1 year ago

Hi @cmfield

Which versions of the MinKNOW API are you using? You can check this by running pip list in your readfish environment. One thing you may want to try is dropping the version of the minknow_api down to 5.0.0.1, which doesn't query for a local token.

That isn't the problem though, I don't think. I'm not super familiar with the Mk1c, but I would expect MinKNOW (on the Mk1C) to listen on port 9502? Are you certain that can access this device on the network?

Rory

cmfield commented 1 year ago

We're using minknow-api 5.5.2 - maybe I should bring it up to match the one on the device. I'll also try dropping it as you suggested.

I'm fairly sure the relevant ports are contactable after checking with nc and trying a bunch of other unrelated ports. They're definitely listed as listening and the error response is distinct from a random unopen port.

cmfield commented 1 year ago

A follow up: I've tried downgrading minknow-api but the same errors occur. There is no 5.5.3, so perhaps this is the problem - I will possibly try to downgrade minknow itself on the device to 5.5.2.

The only other idea I have is mounting a drive on the device from which I can test that readfish can contact minknow locally. We double checked the remote connection by completely disabling the firewall and it didn't help.

Adoni5 commented 1 year ago

If you could try mounting the drive locally, that would be great! I suspect that will work. We don't have a MK1C that I can test on but when I have time I will try playing with connecting remotely on a sequencing box

cmfield commented 1 year ago

I have found a big issue with running it directly on the device - there is no ont-pyguppy-client-lib for Python 3.8 for the architecture it uses (ARM). There is for 3.10.. what do you think the chances of readfish running on 3.10 are?

I'm also trying to run it on a machine connected to the device via USB in case that's enlightening.

Adoni5 commented 1 year ago

readfish is 3.10 compatible I think - certainly all our dependencies are. I actually don't know if Boss Runs is though!

cmfield commented 1 year ago

Ok an update. I compiled Python 3.10 on the device itself and could create an appropriate environment for readfish/bossruns. I had to disable the proxy server we had set up on it, but then the initial unblock test worked! On the way, I also found that the most recent release of non-bossruns readfish is not compatible with the latest version of minknow-api, as a manager function no longer accepts a particular TLS-related argument.

I was very hopeful after this that disabling the proxy would solve the problem of connecting to the device remotely, but sadly not - we still cannot connect to it from another machine. So, I will try some further tests on the device itself, but I'm a bit stumped on what else to try for remote connection. Perhaps I will write a simple script using the minknow-api to try to contact it without using readfish, and that will tell me something. Any further ideas welcome :)

cmfield commented 1 year ago

Ok I have located the issue by manual debugging:

Bossruns says to install read_until_api v3.0.0; this is the problem because that version did not forward the host IP address to the minknow-api Connection class, so it always tried localhost.

With v3.4.1 the readfish unblock-all test works remotely. We will now do the other readfish tests, then the bossruns tests (and I hope that there isn't an issue there that requires 3.0.0 because then it just won't work remotely).

Sorry for the many messages for something that was in the end, not the fault of readfish. I will make an issue with bossruns.

Adoni5 commented 1 year ago

hey @cmfield - thanks very much for keeping us updated, and well done for getting it to work! I think you should be fine to use v3.4.1 for anything in Bossruns. It does allow for some new parameters to be passed into the Read until client, but I'm pretty sure they all have sensible defaults. So just to clarify -

By using read-until-client v3.4.1, you can connect readfish remotely to a MinION Mk1C?

cmfield commented 1 year ago

That's correct. Further testing has shown that in order for readfish targets to work, I had to downgrade the mappy package to 2.20 because in the latest release it no longer has a file mappy.c.

Currently trying the boss-runs tests and unsure if my basecaller is working correctly, but eventually I'll get there :)