search

apache / airavata-mft

Apache Airavata Managed File Transfer Services
https://airavata.apache.org
Apache License 2.0
32 stars 39 forks source link

Storage List Command Fails #93

Closed slimandslam closed 1 year ago

slimandslam commented 1 year ago

M1 Macbook Pro with MacOS 13.2.1

$ mkdir AIR
$ cd AIR
$ python3.10 -m venv venv
$ source venv/bin/activate
(venv) $ pip3 install airavata-mft-cli
Collecting airavata-mft-cli
  Using cached airavata_mft_cli-0.1.9-py3-none-any.whl (21 kB)
Collecting typer[all]<0.8.0,>=0.7.0
  Using cached typer-0.7.0-py3-none-any.whl (38 kB)
Collecting pick==2.2.0
  Using cached pick-2.2.0-py3-none-any.whl (4.9 kB)
Collecting airavata_mft_sdk==0.0.1-alpha27
  Using cached airavata_mft_sdk-0.0.1a27-py3-none-any.whl (96 kB)
Collecting grpcio==1.47.0rc1
  Using cached grpcio-1.47.0rc1-cp310-cp310-macosx_13_0_arm64.whl
Collecting grpcio-tools==1.47.0rc1
  Using cached grpcio_tools-1.47.0rc1-cp310-cp310-macosx_13_0_arm64.whl
Collecting google-api-python-client>=2.0.0
  Downloading google_api_python_client-2.83.0-py2.py3-none-any.whl (11.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.2/11.2 MB 3.4 MB/s eta 0:00:00
Collecting six>=1.5.2
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.47.0rc1->airavata-mft-cli) (67.2.0)
Collecting protobuf<4.0dev,>=3.12.0
  Using cached protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
Collecting click<9.0.0,>=7.1.1
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting shellingham<2.0.0,>=1.3.0
  Using cached shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB)
Collecting colorama<0.5.0,>=0.4.3
  Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting rich<13.0.0,>=10.11.0
  Using cached rich-12.6.0-py3-none-any.whl (237 kB)
Collecting uritemplate<5,>=3.0.1
  Using cached uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting google-auth<3.0.0dev,>=1.19.0
  Downloading google_auth-2.17.0-py2.py3-none-any.whl (178 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 178.1/178.1 kB 13.5 MB/s eta 0:00:00
Collecting google-auth-httplib2>=0.1.0
  Using cached google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
  Using cached google_api_core-2.11.0-py3-none-any.whl (120 kB)
Collecting httplib2<1dev,>=0.15.0
  Using cached httplib2-0.22.0-py3-none-any.whl (96 kB)
Collecting pygments<3.0.0,>=2.6.0
  Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting commonmark<0.10.0,>=0.9.0
  Using cached commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting googleapis-common-protos<2.0dev,>=1.56.2
  Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
Collecting cachetools<6.0,>=2.0.0
  Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp310-cp310-macosx_11_0_arm64.whl (123 kB)
Installing collected packages: pyasn1, commonmark, urllib3, uritemplate, six, shellingham, rsa, pyparsing, pygments, pyasn1-modules, protobuf, pick, idna, colorama, click, charset-normalizer, certifi, cachetools, typer, rich, requests, httplib2, grpcio, googleapis-common-protos, google-auth, grpcio-tools, google-auth-httplib2, google-api-core, google-api-python-client, airavata_mft_sdk, airavata-mft-cli
Successfully installed airavata-mft-cli-0.1.9 airavata_mft_sdk-0.0.1a27 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 colorama-0.4.6 commonmark-0.9.1 google-api-core-2.11.0 google-api-python-client-2.83.0 google-auth-2.17.0 google-auth-httplib2-0.1.0 googleapis-common-protos-1.59.0 grpcio-1.47.0rc1 grpcio-tools-1.47.0rc1 httplib2-0.22.0 idna-3.4 pick-2.2.0 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pygments-2.14.0 pyparsing-3.0.9 requests-2.28.2 rich-12.6.0 rsa-4.9 shellingham-1.5.0.post1 six-1.16.0 typer-0.7.0 uritemplate-4.1.1 urllib3-1.26.15

(venv) $ mft init
Setting up MFT Services
Consul process id: 74795
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (62156) - No such process
Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started
(venv) $ mft storage list
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/storage/__init__.py:6 │
│ 6 in list_storage                                                                                │
│                                                                                                  │
│   63 │   │   │   │   │   │   │   │     secret_service_host = configcli.secret_service_host,      │
│   64 │   │   │   │   │   │   │   │     secret_service_port = configcli.secret_service_port)      │
│   65 │   list_req = StorageCommon_pb2.StorageListRequest()                                       │
│ ❱ 66 │   list_response = client.common_api.listStorages(list_req)                                │
│   67 │                                                                                           │
│   68 │   console = Console()                                                                     │
│   69 │   table = Table(show_header=True, header_style='bold #2070b2')                            │
│                                                                                                  │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮                     │
│ │   client = <airavata_mft_sdk.mft_client.MFTClient object at 0x10796d8a0> │                     │
│ │ list_req = <class 'rich.pretty.Node'>.__repr__ returned empty string     │                     │
│ ╰──────────────────────────────────────────────────────────────────────────╯                     │
│                                                                                                  │
│ /Users/myuser/venv/lib/python3.10/site-packages/grpc/_channel.py:946 in __call__       │
│                                                                                                  │
│    943 │   │   │   │    compression=None):                                                       │
│    944 │   │   state, call, = self._blocking(request, timeout, metadata, credentials,            │
│    945 │   │   │   │   │   │   │   │   │     wait_for_ready, compression)                        │
│ ❱  946 │   │   return _end_unary_response_blocking(state, call, False, None)                     │
│    947 │                                                                                         │
│    948 │   def with_call(self,                                                                   │
│    949 │   │   │   │     request,                                                                │
│                                                                                                  │
│ ╭──────────────────────────────────── locals ─────────────────────────────────────╮              │
│ │           call = <grpc._cython.cygrpc.SegregatedCall object at 0x1076085c0>     │              │
│ │    compression = None                                                           │              │
│ │    credentials = None                                                           │              │
│ │       metadata = None                                                           │              │
│ │        request = <class 'rich.pretty.Node'>.__repr__ returned empty string      │              │
│ │           self = <grpc._channel._UnaryUnaryMultiCallable object at 0x107adde70> │              │
│ │          state = <grpc._channel._RPCState object at 0x107adf4f0>                │              │
│ │        timeout = None                                                           │              │
│ │ wait_for_ready = None                                                           │              │
│ ╰─────────────────────────────────────────────────────────────────────────────────╯              │
│                                                                                                  │
│ /Users/myuser/venv/lib/python3.10/site-packages/grpc/_channel.py:849 in                │
│ _end_unary_response_blocking                                                                     │
│                                                                                                  │
│    846 │   │   else:                                                                             │
│    847 │   │   │   return state.response                                                         │
│    848 │   else:                                                                                 │
│ ❱  849 │   │   raise _InactiveRpcError(state)                                                    │
│    850                                                                                           │
│    851                                                                                           │
│    852 def _stream_unary_invocation_operationses(metadata, initial_metadata_flags):              │
│                                                                                                  │
│ ╭──────────────────────────────── locals ────────────────────────────────╮                       │
│ │      call = <grpc._cython.cygrpc.SegregatedCall object at 0x1076085c0> │                       │
│ │  deadline = None                                                       │                       │
│ │     state = <grpc._channel._RPCState object at 0x107adf4f0>            │                       │
│ │ with_call = False                                                      │                       │
│ ╰────────────────────────────────────────────────────────────────────────╯                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1680205081.363435000","description":"Failed to pick
subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":
[{"created":"@1680205081.363434000","description":"failed to connect to all
addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>
(venv) $
DImuthuUpe commented 1 year ago

@slimandslam Can you post the output of mft log ?

slimandslam commented 1 year ago

Here it is:

$ mft log
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/base.py:47 in         │
│ init_mft                                                                                         │
│                                                                                                  │
│   44                                                                                             │
│   45 @app.command("log")                                                                         │
│   46 def init_mft():                                                                             │
│ ❱ 47   bootstrap.print_log()                                                                     │
│                                                                                                  │
│ /Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/bootstrap.py:158 in   │
│ print_log                                                                                        │
│                                                                                                  │
│   155                                                                                            │
│   156 def print_log():                                                                           │
│   157   log_file_path = os.path.join(os.path.expanduser('~'), ".mft", "Standalone-Service-0.01   │
│ ❱ 158   log_file = open(log_file_path,"r")                                                       │
│   159   lines = follow_file(log_file)                                                            │
│   160   for line in lines:                                                                       │
│   161 │   print(line)                                                                            │
│                                                                                                  │
│ ╭────────────────────────────────────── locals ───────────────────────────────────────╮          │
│ │ log_file_path = '/Users/myuser/.mft/Standalone-Service-0.01/logs/airavata.log' │          │
│ ╰─────────────────────────────────────────────────────────────────────────────────────╯          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory:
'/Users/myuser/.mft/Standalone-Service-0.01/logs/airavata.log'
$
DImuthuUpe commented 1 year ago

@slimandslam I think the latest mft cli is having a conflict with an older mft version you had earlier. Can you try running following commands to make sure that you have a clean installation?

rm ~/mft_db.*
mft stop
rm -rf ~/.mft
mft init

Please post the output of mft init. It should download binaries from GitHub release

slimandslam commented 1 year ago

Ok.

$ rm ~/mft_db.*
rm: /Users/myuser/mft_db.*: No such file or directory
$ mft stop
Stopping MFT Services
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (83776) - No such process
Standalone Service stopped ...
MFT Stopped....
$ rm -rf ~/.mft
$ mft init
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 1372, in _path_importer_cache
KeyError: '.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myuser/venv/bin/mft", line 5, in <module>
    from airavata_mft_cli.main import app
  File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/main.py", line 21, in <module>
    import airavata_mft_cli.base
  File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/base.py", line 21, in <module>
    import airavata_mft_cli.bootstrap as bootstrap
  File "/Users/myuser/venv/lib/python3.10/site-packages/airavata_mft_cli/bootstrap.py", line 20, in <module>
    import requests
  File "/Users/myuser/venv/lib/python3.10/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/Users/myuser/venv/lib/python3.10/site-packages/urllib3/__init__.py", line 13, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 12, in <module>
    from .connection import (
  File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/connection.py", line 15, in <module>
    from .util.proxy import create_proxy_ssl_context
  File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/util/__init__.py", line 5, in <module>
    from .request import SKIP_HEADER, SKIPPABLE_HEADERS, make_headers
  File "Users/myuser/venv/lib/python3.10/site-packages/urllib3/util/request.py", line 18, in <module>
    import brotlicffi as _unused_module_brotli  # noqa: F401
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1408, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1374, in _path_importer_cache
  File "<frozen importlib._bootstrap_external>", line 1350, in _path_hooks
  File "<frozen importlib._bootstrap_external>", line 1632, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 1505, in __init__
FileNotFoundError: [Errno 2] No such file or directory
$
DImuthuUpe commented 1 year ago

It seems like some issue in the virtual environment you created. The line which is getting failed is https://github.com/apache/airavata-mft/blob/master/python-cli/mft_cli/airavata_mft_cli/bootstrap.py#L20 which is the import of requests module. Can you try creating a new virtual environment and repeating the installation steps?

slimandslam commented 1 year ago

What are you using to check if your daemon is running? Every time I've done an install, I see this (below) but there is never anything running. 

$ mft init
Setting up MFT Services
Consul process id: 41547
Standalone Service stoping ...
./standalone-service-daemon.sh: line 55: kill: (41361) - No such process
Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started
$
DImuthuUpe commented 1 year ago

I think the process created for the previous mft server run was abruptly killed or exited. That's why you are seeing " No such process" error. One possibility is that you do not have a compatible java runtime installed on your machine. We can try the attached mode in mft to narrow down the issue. Please send the output after running the following commands. 

cd ~/.mft/Standalone-Service-0.01/bin
chmod +x standalone-service.sh
./standalone-service.sh
slimandslam commented 1 year ago

What are you using to check to see if the daemon is running? Doe CLI commands depend on the daemon process?

J

DImuthuUpe commented 1 year ago

mft init command kills the currently running mft service daemon process (if exists) and starts a new one. When the daemon process starts, it saves a process id file at file ~/.mft/Standalone-Service-0.01/bin/service-pid. When a daemon restart was invoked, it looks for a process with the id in service-pid file to kill first. However, in your case, this process exited due to some error, and logs were not even created. I assume that you do not have a compatible java virtual machine in your computer. The commands I mentioned earlier should reveal what the error is

slimandslam commented 1 year ago

Do the mft CLI commands depend on the daemon in order to work?

J

On Tue, Apr 4, 2023 at 9:20 AM Dimuthu Wannipurage @.***> wrote:

mft init command kills the currently running mft service daemon process (if exists) and starts a new one. When the daemon process starts, it saves a process id file at file ~/.mft/Standalone-Service-0.01/bin/service-pid. When a daemon restart was invoked, it looks for a process with the id in service-pid file to kill first. However, in your case, this process exited due to some error, and logs were not even created. I assume that you do not have a compatible java virtual machine in your computer. The commands I mentioned earlier should reveal what the error is

— Reply to this email directly, view it on GitHub https://github.com/apache/airavata-mft/issues/93#issuecomment-1496059617, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABYBR225HME6BPKGWO2LN3W7QUZBANCNFSM6AAAAAAWNXYORA . You are receiving this because you were mentioned.Message ID: @.***>

DImuthuUpe commented 1 year ago

Yes, it requires the daemon to be up and running. CLI talks to the daemon through its APIs

slimandslam commented 1 year ago

Is there a way for the CLI to return a sensible error message like: “cannot connect to demon” ?

J

On Tue, Apr 4, 2023 at 10:03 AM Dimuthu Wannipurage < @.***> wrote:

Yes, it requires the daemon to be up and running. CLI talks to the daemon through its APIs

— Reply to this email directly, view it on GitHub https://github.com/apache/airavata-mft/issues/93#issuecomment-1496133423, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABYBR4MWR6FXRUQIM5DDHDW7QZ3LANCNFSM6AAAAAAWNXYORA . You are receiving this because you were mentioned.Message ID: @.***>

DImuthuUpe commented 1 year ago

Good suggestion. I created the issue https://github.com/apache/airavata-mft/issues/94 to track this enhancement

slimandslam commented 1 year ago

Thanks!

On Tue, Apr 4, 2023 at 12:23 PM Dimuthu Wannipurage < @.***> wrote:

Good suggestion. I created the issue #94 https://github.com/apache/airavata-mft/issues/94 to track this enhancement

— Reply to this email directly, view it on GitHub https://github.com/apache/airavata-mft/issues/93#issuecomment-1496337252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABYBRYIPW2MYR2IXRAEJWLW7RKJRANCNFSM6AAAAAAWNXYORA . You are receiving this because you were mentioned.Message ID: @.***>

slimandslam commented 1 year ago

In my case, I had Java 11 installed, but Java 8 was the selected runtime (my bad). However, there were two things that made it harder to debug than it should have been. One thing was that the CLI wasn't responding with "cannot contact daemon" (or something similar). I guess that will be handled in https://github.com/apache/airavata-mft/issues/94 The other thing is that your script said:

Standalone Service stopped ...
Starting Standalone Service ...
Standalone Service started ...
MFT Started

But it almost certainly exited immediately after this. You can test it by running it with the Java 8 runtime. Perhaps your script should check for the version of the Java runtime just before starting the service since having multiple Java runtimes is not uncommon.

DImuthuUpe commented 1 year ago

These are very good suggestions @slimandslam. Especially checking the java runtime before starting the script is very important. Did you manage to start MFT after changing the java version to 11?

DImuthuUpe commented 1 year ago

@slimandslam I created another issue to track your suggestion. Please feel free to create issues you encounter on both the usability and operational level. We are actively working on improving the usability of the CLI as it is still at the very early stage of development

DImuthuUpe commented 1 year ago

Closing as the issue no longer appears. Please reopen if required