search

opencadc / vostools

VOSpace command line and FUSE clients
https://www.canfar.net/en/docs/storage
12 stars 18 forks source link

Frequent vcp failures #189

Closed drtobybrown closed 3 years ago

drtobybrown commented 3 years ago

vcp transfers from local (research cluster and laptop) to VOSpace fail frequently. Typically I am transferring large folders (several GB) containing 100s if not 1000s of files. I haven't been keeping a log but I would say over 50% of attempts fail. Often transfers complete on the 3rd, 4th, or 5th attempt. There is no drop in local connection and downloads seem more robust.

The errors error messages vary and are not particularly instructive.

e.g., cadcutils.exceptions.HttpException: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I can't find documentation to guide vcp best practice. Steps to reproduce, try and transfer a big folder...

drtobybrown commented 3 years ago

I received this error today..

$ vcp release.v1.2 vos:VERTICO/share/products/
Traceback (most recent call last):
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/commands/vcp.py", line 335, in copy
    send_md5=True, head=head)
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 1873, in copy
    put_urls = self.get_node_url(destination, 'PUT')
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 2164, in get_node_url
    md5_checksum=md5_checksum)
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 2325, in _put
    md5_checksum=md5_checksum)
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 2357, in transfer
    security_methods=security_methods)
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 2957, in transfer
    format(transfer_url))
RuntimeError: BUG: No protocol/endpoint returned for transfer URL https://ws-cadc.canfar.net/vault/xfer/sm2lg8axkmiwik5a
ERROR:: BUG: No protocol/endpoint returned for transfer URL https://ws-cadc.canfar.net/vault/xfer/sm2lg8axkmiwik5a
drtobybrown commented 3 years ago

Followed by this...

Traceback (most recent call last):
  File "/home/brownt16/.local/lib/python3.6/site-packages/vos/vos.py", line 1891, in copy
    put_url, data=reader)
  File "/home/brownt16/.local/lib/python3.6/site-packages/requests/sessions.py", line 590, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/home/brownt16/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/brownt16/.local/lib/python3.6/site-packages/cadcutils/net/ws.py", line 482, in send
    **kwargs)
  File "/home/brownt16/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/brownt16/.local/lib/python3.6/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='ws-uv.canfar.net', port=443): Read timed out. (read timeout=120)
drtobybrown commented 3 years ago

Upgrading to vos 3.3a with pip results in raw_input("Config file not found. Would you like to create an empty one? (yes: enter, no: ctrl+c)") NameError: name 'raw_input' is not defined

andamian commented 3 years ago

@drtobybrown - please upgrade to the newly release 3.3.2 and let me know if you still see these problems or if their frequency is lower. I'm going to close this ticket for now but we can re-open it if the patch doesn't do it.