BUG: Remote upload fails with `requests.exceptions.ConnectionError: [Errno 32] Broken pipe`

corneliusroemer commented 2 years ago

Current Behavior

After migration of neherlab, when I try to upload using nextstrain remote upload https://nextstrain.org/groups... the upload fails and I get the following uncaught Python error: requests.exceptions.ConnectionError: [Errno 32] Broken pipe

Expected behavior

Upload works

How to reproduce

nextstrain remote upload https://nextstrain.org/groups/neherlab/ncov/lithuania_2022-06-19 lithuania/2022-06-19/ncov_lithuania_2022-06-19.json lithuania/2022-06-19/ncov_lithuania_2022-06-19_root-sequence.json lithuania/2022-06-19/ncov_lithuania_2022-06-19_tip-frequencies.json 2>&1

Your environment: if browsing Nextstrain locally

Operating system: macOS and CentOS

❯ nextstrain version
nextstrain.cli 3.2.5

Additional context

It could be that I'm using the command to upload wrong. I followed the migration guide: https://docs.nextstrain.org/en/latest/guides/share/groups/migrate-from-s3.html

In any case, there should not be an uncaught exception that doesn't explain what the user is doing wrong.

Full error trace back

~/Downloads/tmp via 🐍 v3.8.13 via 🅒 nextstrain took 3s 
❯ nextstrain remote upload https://nextstrain.org/groups/neherlab/ncov/lithuania_2022-06-19 lithuania/2022-06-19/ncov_lithuania_2022-06-19.json lithuania/2022-06-19/ncov_lithuania_2022-06-19_root-sequence.json lithuania/2022-06-19/ncov_lithuania_2022-06-19_tip-frequencies.json 

Uploading lithuania/2022-06-19/ncov_lithuania_2022-06-19.json as https://nextstrain.org/groups/neherlab/ncov/lithuania_2022-06-19
Traceback (most recent call last):
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/requests/adapters.py", line 526, in send
    low_conn.send(i)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/http/client.py", line 972, in send
    self.sock.sendall(data)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/bin/nextstrain", line 10, in <module>
    sys.exit(main())
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/nextstrain/cli/__main__.py", line 10, in main
    return cli.run( argv[1:] )
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/nextstrain/cli/__init__.py", line 33, in run
    return opts.__command__.run(opts)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/nextstrain/cli/command/remote/upload.py", line 64, in run
    for local_file, remote_file in uploads:
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/nextstrain/cli/remote/nextstrain_dot_org.py", line 237, in upload
    put(endpoint, file, media_type)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/nextstrain/cli/remote/nextstrain_dot_org.py", line 218, in put
    response = http.put(
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/requests/sessions.py", line 647, in put
    return self.request("PUT", url, data=data, **kwargs)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Caskroom/mambaforge/base/envs/nextstrain/lib/python3.8/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: [Errno 32] Broken pipe

tsibley commented 2 years ago

This should be resolved by https://github.com/nextstrain/nextstrain.org/pull/565 once it's merged.