search

whole-tale / globus_handler

Whole Tale Globus Transfer Handler plugin
0 stars 1 forks source link

Globus transfer handling fails to initialize #1

Open Xarthisius opened 3 years ago

Xarthisius commented 3 years ago

Step to reproduce

  1. Register Globus resource
  2. Create a Tale, add globus resources to dataSet
  3. Launch a Tale
  4. cat ../data/<globus_resource>

Expected outcome

See the content of the file

Actual outcome

Despair and tracebacks:

[2021-05-02 18:38:29,560] ERROR: Traceback (most recent call last):
[2021-05-02 18:38:29,561] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/transfer_manager.py", line 24, in run
[2021-05-02 18:38:29,561] ERROR:     self.transferHandler.run()
[2021-05-02 18:38:29,561] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/tm_utils.py", line 34, in run
[2021-05-02 18:38:29,561] ERROR:     self.transfer()
[2021-05-02 18:38:29,562] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 30, in transfer
[2021-05-02 18:38:29,562] ERROR:     self._maybeStartServer()
[2021-05-02 18:38:29,562] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 106, in _maybeStartServer
[2021-05-02 18:38:29,562] ERROR:     self.server.start()
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 57, in start
[2021-05-02 18:38:29,563] ERROR:     (self.endpointId, self.endpointName) = self._createEndpoint()
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 223, in _createEndpoint
[2021-05-02 18:38:29,563] ERROR:     _runGCCommand('%s/globusconnectpersonal' % self.gcDir, '-setup', setupKey, '-dir',
[2021-05-02 18:38:29,563] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 19, in _runGCCommand
[2021-05-02 18:38:29,564] ERROR:     raise Exception(msg)
[2021-05-02 18:38:29,564] ERROR: Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-setup', '...', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9') failed with exit code 2:
[2021-05-02 18:38:29,564] ERROR: b'usage: register [-h] [--attributes [file:PATH|JSON_DATA]] [-n NAME]\n                [--description DESCRIPTION] [--owner OWNER] [-z]\n

NOTE

Switching order of args from --setup ... --dir ... to --dir ... --setup ... fixes the issue. Globuspersonalconnect passes everything that comes after --setup to separate routine called register. Update: then I get a different error:

Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-debug', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', 'e916cf72-97a0-4e49-9f03-3019ae33b3b0') failed with exit code 1: 
b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"

$ cat register.log 
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup
[WARNING] relaytool.py::136 [invoke_relaytool] relaytool setup exit with status 1, indicating failure. use debug mode for full output
[INFO] cli.py::200 [main] relaytool stdout:
-----
('',)-----
[INFO] cli.py::201 [main] relaytool stderr:
-----
('',)
-----
[INFO] cli.py::202 [main] full trace of relaytool failure
Traceback (most recent call last):
  File "lib/cli.py", line 198, in main
  File "lib/relaytool.py", line 151, in run_relaytool_setup
  File "lib/relaytool.py", line 146, in invoke_relaytool
lib.relaytool.RelayToolFailureError: ('relaytool setup failed', CompletedProcess(args='/opt/globusconnectpersonal/gt_amd64/bin/relaytool', returncode=1, stdout=b'', stderr=b''))
[ERROR] cli.py::203 [main] relaytool failed, full error info available in register.log
craig-willis commented 2 years ago

I get this error in globusconnectpersonal 3.1.x but not 3.0.4, which we have in our v0.9 image:

docker run -it --user=girder --entrypoint=/bin/bash wholetale/girder:v0.9

/opt/globusconnectpersonal/globusconnectpersonal -debug -setup 29e108db-88ac-48bd-a961-cfca7382ceba -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9
Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.
Creating the directory... Done
Configuration directory: /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta
Contacting relay.globusonline.org:2223
Done!
Xarthisius commented 2 years ago

This is really annoying cause when I run this manually:

$ cat ala.py 
import subprocess
import threading

cmd = "/opt/globusconnectpersonal/globusconnectpersonal -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9 -setup 39a06eeb-62e6-41fa-9c6f-3c7ed1514617"

p = subprocess.run(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(p.stdout)

class GCThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self, name="Globus Connect")
        self.daemon = True

    def run(self):
        p = subprocess.run(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        print(p.stdout)

t = GCThread()
t.start()
t.join(timeout=10)

I get 

girder@1e556b22a355:~$ python3.9 ala.py 
b'setup completed successfully\n'
b'setup completed successfully\n'
craig-willis commented 2 years ago

Re-opening this issue because the problem still exists with GCP 3.1.x and we should try to resolve at some point.

Noting that setup appears to succeed when run directly from the CLI:


$ docker run -it --user girder craigwillis/girder:gcp-3.1.6 bash

girder@81853cdda67c /opt/globusconnectpersonal/globusconnectpersonal -version
Globus Connect Personal 3.1.6

girder@81853cdda67c:/girder$ /opt/globusconnectpersonal/globusconnectpersonal -dir /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9 -setup 52b10576-2690-438f-80c1-f87e74fb42c5
Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.
Creating the directory... Done
setup completed successfully

But fails when run from the globus_handler as above:

[2022-02-21 15:10:56,252] WARNING: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', '52b10576-2690-438f-80c1-f87e74fb42c5') failed with exit code 1:
b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"
b''
[2022-02-21 15:10:56,254] ERROR: Traceback (most recent call last):
[2022-02-21 15:10:56,254] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/transfer_manager.py", line 24, in run
[2022-02-21 15:10:56,254] ERROR:     self.transferHandler.run()
[2022-02-21 15:10:56,255] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/tm_utils.py", line 34, in run
[2022-02-21 15:10:56,255] ERROR:     self.transfer()
[2022-02-21 15:10:56,255] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 30, in transfer
[2022-02-21 15:10:56,256] ERROR:     self._maybeStartServer()
[2022-02-21 15:10:56,256] ERROR:   File "/girder/plugins/wt_data_manager/server/lib/handlers/globus.py", line 106, in _maybeStartServer
[2022-02-21 15:10:56,256] ERROR:     self.server.start()
[2022-02-21 15:10:56,256] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 57, in start
[2022-02-21 15:10:56,257] ERROR:     (self.endpointId, self.endpointName) = self._createEndpoint()
[2022-02-21 15:10:56,257] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 223, in _createEndpoint
[2022-02-21 15:10:56,257] ERROR:     _runGCCommand('%s/globusconnectpersonal' % self.gcDir, '-dir', self.confDir, '-setup', setupKey)
[2022-02-21 15:10:56,257] ERROR:   File "/girder/plugins/globus_handler/server/server.py", line 19, in _runGCCommand
[2022-02-21 15:10:56,258] ERROR:     raise Exception(msg)
[2022-02-21 15:10:56,258] ERROR: Exception: Command ('/opt/globusconnectpersonal/globusconnectpersonal', '-dir', '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9', '-setup', '52b10576-2690-438f-80c1-f87e74fb42c5') failed with exit code 1:
[2022-02-21 15:10:56,258] ERROR: b"Could not read the directory '/home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9' with configuration files.\nCreating the directory... Done\nrelaytool setup exit with status 1, indicating failure. use debug mode for full output\nrelaytool failed, full error info available in register.log\n\nSetup did not complete successfully.\nYou may want to check /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log for more information\n"
[2022-02-21 15:10:56,258] ERROR: b''
$ cat /home/girder/.WholeTale/0baa7c04-1ff6-4cf5-82b3-00dd47e828d9/lta/register.log
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup
[WARNING] relaytool.py::136 [invoke_relaytool] relaytool setup exit with status 1, indicating failure. use debug mode for full output
[INFO] cli.py::200 [main] relaytool stdout:
-----
('',)-----
[INFO] cli.py::201 [main] relaytool stderr:
-----
('',)
-----
[INFO] cli.py::202 [main] full trace of relaytool failure
Traceback (most recent call last):
  File "lib/cli.py", line 198, in main
  File "lib/relaytool.py", line 151, in run_relaytool_setup
  File "lib/relaytool.py", line 146, in invoke_relaytool
lib.relaytool.RelayToolFailureError: ('relaytool setup failed', CompletedProcess(args='/opt/globusconnectpersonal/gt_amd64/bin/relaytool', returncode=1, stdout=b'', stderr=b''))
[ERROR] cli.py::203 [main] relaytool failed, full error info available in register.log
[INFO] cli.py::190 [main] got --setup-key, skip endpoint creation...
[INFO] cli.py::196 [main] starting relaytool setup
Xarthisius commented 2 years ago

For easy reproducing https://gist.github.com/Xarthisius/79f5fc8b8186d5b9d4340ce5d81c2486

Xarthisius commented 2 years ago

Note to self: This works...

$ docker exec -ti -u girder $(docker ps --filter=name=girder -q) girder-shell
import pathlib
from girder.plugins.globus_handler.clients import Clients
from girder.plugins.globus_handler.server import Server
clients = Clients()
s = Server(clients)
key = s._getEndpointKey()
s.confDir = '%s/.WholeTale/%s' % (pathlib.Path.home(), key)
s._createEndpoint()