nanoporetech / minknow_api

Protobuf and gRPC specifications for the MinKNOW API
Other
51 stars 12 forks source link

minknow_api onremote machines implemented? #17

Closed neuropathbasel closed 3 years ago

neuropathbasel commented 3 years ago

Hello, Many thanks for forwarding me here. First step on the local linux machine worked fine and prompted:

~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host localhost --port 9501 Available sequencing positions on localhost:9501: MN32638: running secure: 8001 insecure: 8000

However, issues started with the attempt to obtain a similar readout form an identical second machine in the same subnet running under 192.18.01.18. On this machine MinKNOW GUI Installed version: 4.1.22 is running and a Minion is attached to it via USB 3. This unit is “detected” by the local MinKNOW GUI.

Running on the first linux machine ~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host 192.168.1.18 --port 9501

yields: Traceback (most recent call last): File "./python/examples/list_sequencing_positions.py", line 41, in main() File "./python/examples/list_sequencing_positions.py", line 23, in main manager = Manager(host=args.host, port=args.port, use_tls=False) File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 108, in init minknow_api.manager_service.GetVersionInfoRequest() File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1618939002.979941782","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5419,"referenced_errors":[{"created":"@1618939002.979938522","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

ping to 192.168.1.18 yields an immediate reply.

Different attempts and the content of manager.py and manager_service.py imply to me that remote hosts are (so far ?) not supported.

Could you please let me know if gRPC calls to remote machine via the minknow_api are supported? Many thanks Claus

0x55555555 commented 3 years ago

Hello @neuropathbasel,

When communicating with remote hosts, you must use tls (the examples dont support that in order to be concise).

You will also need to use port 9502 for the manager.

For example:

import argparse
from minknow_api.manager import Manager

def main():
    """Main entrypoint for list_sequencing_devices example"""
    parser = argparse.ArgumentParser(
        description="List sequencing positions connected to a host."
    )
    parser.add_argument(
        "--host", default="localhost", help="Specify which host to connect to."
    )
    parser.add_argument(
        "--port", default=None, help="Specify which porer to connect to."
    )

    args = parser.parse_args()

    # Construct a manager using the host + port provided.
    manager = Manager(host=args.host, port=args.port, use_tls=True) # Enable TLS

    # Find a list of currently available sequencing positions.
    positions = manager.flow_cell_positions()

    # Print out available positions.
    print("Available sequencing positions on %s:%s:" % (args.host, args.port))
    for pos in positions:
        print("%s: %s" % (pos.name, pos.state))

        if pos.running:
            print("  secure: %s" % pos.description.rpc_ports.secure)
            print("  insecure: %s" % pos.description.rpc_ports.insecure)

            # User could call {pos.connect()} here to connect to the running MinKNOW instance.

if __name__ == "__main__":
    main()
JensUweUlrich commented 3 years ago

Thanks for the information. I just wonder if every MinKNOW instance has its own certificate or if the certificate is the same on every remote machine?

Thanks Jens

0x55555555 commented 3 years ago

The certificate is the same for the moment, this is something we are considering changing in the future.

Thanks,

neuropathbasel commented 3 years ago

hello jorj1988, Many thanks for your prompt reply. I did specify port 9502 for the remote machine. However without any success - I am pretty sure I am missing something. On machine 1 (with IP 192.168.0.15) I ran: (gRPC) user@meqneuropat15:~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host 192.168.1.18 --port 9502

But instead of a device listing I did obtain: Traceback (most recent call last): File "./python/examples/list_sequencing_positions.py", line 41, in main() File "./python/examples/list_sequencing_positions.py", line 23, in main manager = Manager(host=args.host, port=args.port, use_tls=False) File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 108, in init minknow_api.manager_service.GetVersionInfoRequest() File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 946, in call return _end_unary_response_blocking(state, call, False, None) File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1618995241.639709258","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5419,"referenced_errors":[{"created":"@1618995241.639706702","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"

Any suggestion? best Claus

0x55555555 commented 3 years ago

Hi @neuropathbasel,

Did you modify the example to use tls - as in the snippet above?

    manager = Manager(host=args.host, port=args.port, use_tls=True) # Enable TLS
neuropathbasel commented 3 years ago

Hi George, I did modify the the example, but the error stayed the same until I started a new terminal. Now I do need SSL certificates. (gRPC) user@meqneuropat15:~/minknow_api$ python ./python/examples/list_sequencing_positions.py --host 192.168.1.18 --port 9502 Traceback (most recent call last): File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/init.py", line 190, in grpc_credentials return grpc_credentials.cached_credentials AttributeError: 'function' object has no attribute 'cached_credentials'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./python/examples/list_sequencing_positions.py", line 41, in main() File "./python/examples/list_sequencing_positions.py", line 23, in main manager = Manager(host=args.host, port=args.port, use_tls=True)#set use_TLS=True for emote hosts Wed 2021-04-21 File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 103, in init use_tls=use_tls, File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 37, in init minknow_api.grpc_credentials(), File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/init.py", line 196, in grpc_credentials "Couldn't find a valid path to MinKNOW's CA SSL certificate to initiate a secure connection" minknow_api.MissingMinknowSSlCertError: Couldn't find a valid path to MinKNOW's CA SSL certificate to initiate a secure connection

will go through this repo to find a HowTo cheers

Claus

neuropathbasel commented 3 years ago

@George I understood your first reply that I have to work on port 9502 - a secure chanel. So far I did not progress standard error is "Couldn't find a valid path to MinKNOW's CA SSL certificate to initiate a secure connection"

Currently I would prefer for a fist trial to use a insecure channel on port 9501 - at least for a first attempt. How can I get this to work? best Claus

JensUweUlrich commented 3 years ago

The certificate is the same for the moment, this is something we are considering changing in the future.

Thanks,

  • George

Thanks George!

Please also consider a way to pull the certificate from the MinKNOW instance via some API calls when using different certificates for different instances of MinKNOW. Otherwise it would be pain in the ass to manually transfer and store each certificate from the remote machines to a local one.

Cheers Jens

0x55555555 commented 3 years ago

@neuropathbasel you can change how minknow binds insecure ports in the user_conf file (in <minknow install dir>/ conf / user_conf:

the local_connection_only setting should be set to "minknow_default" currently, if you set it to all_open minknow will bind all insecure ports to the network. This is not recommended from a security POV, but I agree is simpler to test things work as expected.

neuropathbasel commented 3 years ago

Thanks for the rapid help. I introduced on both machines the changes you suggested and restarted the minknow service with sudo restart minknow service. however, so far, without any luck. for reasons unclear to me there is still the need for defining the path to a to MinKNOW's CA SSL certificate. I assumed to use an insecure connection. Entering python list_sequencing_positions_remote.py --host 192.168.1.18 --port 9501

prompts: Traceback (most recent call last): File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/init.py", line 190, in grpc_credentials return grpc_credentials.cached_credentials AttributeError: 'function' object has no attribute 'cached_credentials'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "list_sequencing_positions_remote.py", line 55, in main() File "list_sequencing_positions_remote.py", line 26, in main manager = Manager(host=args.host, port=args.port, use_tls=True) File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 103, in init use_tls=use_tls, File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/manager.py", line 37, in init minknow_api.grpc_credentials(), File "/home/user/minknow_api/gRPC/lib/python3.7/site-packages/minknow_api/init.py", line 196, in grpc_credentials "Couldn't find a valid path to MinKNOW's CA SSL certificate to initiate a secure connection" minknow_api.MissingMinknowSSlCertError: Couldn't find a valid path to MinKNOW's CA SSL certificate to initiate a secure connection.

I did find the https://github.com/nanoporetech/minknow_api/blob/2fad019ee71546c00f534bdd0ca37d885baf073a/README.md the section "How do I connect to a "secure" port?", but I am not where to implement the given code.

Any suggestion? Best Claus

0x55555555 commented 3 years ago

Hi @neuropathbasel ,

The code above implies you are using tls still:

 manager = Manager(host=args.host, port=args.port, use_tls=True)

If you are using port 9501 you will need to not use tls.

If you use tls you need to ensure you supply the certificate from the minknow sequencing device on the local machine.

Thanks,

neuropathbasel commented 3 years ago

thanks for your rapid help. I did correct the code, but did not try to start the code from another terminal. On this headless system, I am accessing via VNC I feel that something (an editor?) is blocking some functions. I have not seen this before.

Is there an example for supplying the certificate with the simple list_sequencing devices?

In my hand the link to the API description files (https://github.com/nanoporetech/minknow_api/blob/2fad019ee71546c00f534bdd0ca37d885baf073a/protos/minknow_api) is not found. many thanks Claus

0x55555555 commented 3 years ago

Please see this other issue where we discussed how to link CA certificates for minknow:

https://github.com/nanoporetech/minknow_api/issues/16