search

boto / s3transfer

Amazon S3 Transfer Manager for Python
Apache License 2.0
209 stars 133 forks source link

RuntimeError: cannot schedule new futures after interpreter shutdown #197

Open lphuberdeau opened 3 years ago

lphuberdeau commented 3 years ago

Reverting to boto3 1.17.53, which is the last version using s3transfer<0.4 solves the issue.

The calls to upload_fileobj and download_fileobj are performed within a thread via asyncio's run_in_executor.

Traceback (most recent call last):
...
    s3.download_fileobj(self.bucket_name, key, fp)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 355, in _submit
    self._submit_download_request(
  File "/venv/lib/python3.8/site-packages/s3transfer/download.py", line 384, in _submit_download_request
    self._transfer_coordinator.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 320, in submit
    future = executor.submit(task, tag=tag)
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
...
    s3.upload_fileobj(io.BytesIO(data), self.bucket_name, key)
  File "/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 536, in upload_fileobj
    future = manager.upload(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 326, in upload
    return self._submit_transfer(
  File "/venv/lib/python3.8/site-packages/s3transfer/manager.py", line 500, in _submit_transfer
    self._submission_executor.submit(
  File "/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 467, in submit
    future = ExecutorFuture(self._executor.submit(task))
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 181, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown
jpl-jengelke commented 2 years ago

Actually, reverting to an earlier Boto3 version didn't solve the problem. (It was verified using pip freeze that s3transfer was in the 3.x version range.) It did start working when I disabled threading on the s3 operations using TransferConfig().