search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.06k stars 1.87k forks source link

S3 download fails with logs that are difficult to interpret: Event needs-retry.s3.HeadObject #2550

Closed briancaffey closed 4 years ago

briancaffey commented 4 years ago

I'm trying to download a file from S3 during a celery task that is running on a container in a Fargate task. It is working, but fails sometimes with logs that are difficult for me to interpret.

I'm using django-storages, so the file is accessed like this: 

def report_file_to_df(report_file):
    # get report file and read into df
    f = report_file.data_file
    excel_file = pd.ExcelFile(f, engine='xlrd')
    return excel_file_to_df(excel_file)

The bucket and fargate task are both in us-east-1.

The file does exist and has data; I can download it an verify this, even though some of the logs make it look like the file is empty.

The files I am working with are about 15MB in size, and most of them are able to be downloaded and processed just fine.

For some files this is failing with the following logs: 

/usr/local/lib/python3.8/site-packages/celery/platforms.py:800: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
[2020-08-18 11:58:48,740: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-08-18 11:58:48,742: DEBUG/MainProcess] | Worker: Building graph...
[2020-08-18 11:58:48,742: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer}
[2020-08-18 11:58:48,754: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-08-18 11:58:48,755: DEBUG/MainProcess] | Consumer: Building graph...
[2020-08-18 11:58:48,787: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Tasks, Control, Agent, Heart, event loop}
-------------- worker-default@ip-10-0-0-119.ec2.internal v4.4.7 (cliffs)
--- ***** -----
-- ******* ---- Linux-4.14.158-129.185.amzn2.x86_64-x86_64-with-glibc2.2.5 2020-08-18 11:58:48
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: backend:0x7f901383bc10
- ** ---------- .> transport: redis://redis.address.use1.cache.amazonaws.com:6379/1
- ** ---------- .> results: redis://redis.address.use1.cache.amazonaws.com:6379/2
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: ON
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. apps.app.tasks.process_uploaded_report_file
[2020-08-18 11:58:48,799: DEBUG/MainProcess] | Worker: Starting Hub
[2020-08-18 11:58:48,799: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:48,799: DEBUG/MainProcess] | Worker: Starting Pool
[2020-08-18 11:58:48,911: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:48,911: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-08-18 11:58:48,911: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-08-18 11:58:48,916: DEBUG/ForkPoolWorker-1] Using selector: EpollSelector
[2020-08-18 11:58:48,925: INFO/MainProcess] Connected to redis://redis.address.use1.cache.amazonaws.com:6379/1
[2020-08-18 11:58:48,926: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:48,926: DEBUG/MainProcess] | Consumer: Starting Events
[2020-08-18 11:58:48,934: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:48,934: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-08-18 11:58:48,935: INFO/MainProcess] mingle: searching for neighbors
[2020-08-18 11:58:49,956: INFO/MainProcess] mingle: all alone
[2020-08-18 11:58:49,957: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:49,957: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-08-18 11:58:49,962: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:49,962: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-08-18 11:58:49,967: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:49,967: DEBUG/MainProcess] | Consumer: Starting Control
[2020-08-18 11:58:49,972: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:49,972: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-08-18 11:58:49,974: DEBUG/MainProcess] ^-- substep ok
[2020-08-18 11:58:49,974: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-08-18 11:58:49,974: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2020-08-18 11:58:49,975: INFO/MainProcess] worker-default@ip-10-0-0-119.ec2.internal ready.
[2020-08-18 11:58:49,975: DEBUG/MainProcess] basic.qos: prefetch_count->1
[2020-08-18 11:58:50,088: INFO/MainProcess] Received task: apps.app.tasks.process_uploaded_report_file[97e8774a-c137-4b29-8a72-6bc409c2f903]
[2020-08-18 11:58:50,089: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f9012a22280> (args:('apps.app.tasks.process_uploaded_report_file', '97e8774a-c137-4b29-8a72-6bc409c2f903', {'lang': 'py', 'task': 'apps.app.tasks.process_uploaded_report_file', 'id': '97e8774a-c137-4b29-8a72-6bc409c2f903', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97e8774a-c137-4b29-8a72-6bc409c2f903', 'parent_id': None, 'argsrepr': '(2198,)', 'kwargsrepr': '{}', 'origin': 'gen8@ip-10-0-1-129.ec2.internal', 'reply_to': '74809a9d-a164-3593-a408-292a7ee323c6', 'correlation_id': '97e8774a-c137-4b29-8a72-6bc409c2f903', 'hostname': 'worker-default@ip-10-0-0-119.ec2.internal', 'delivery_info': {'exchange': '', 'routing_key': 'default', 'priority': 0, 'redelivered': None}, 'args': [2198], 'kwargs': {}}, b'[[2198], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-08-18 11:58:50,091: DEBUG/MainProcess] Task accepted: apps.app.tasks.process_uploaded_report_file[97e8774a-c137-4b29-8a72-6bc409c2f903] pid:15
[2020-08-18 11:58:50,187: INFO/ForkPoolWorker-1] Task apps.app.tasks.process_uploaded_report_file[97e8774a-c137-4b29-8a72-6bc409c2f903] succeeded in 0.09556395399999928s: None
[2020-08-18 11:58:51,979: INFO/MainProcess] Received task: apps.core.tasks._caller[2e1f7404-9b42-405a-884a-0c1952d9fa42]
[2020-08-18 11:58:51,979: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f9012a22280> (args:('apps.core.tasks._caller', '2e1f7404-9b42-405a-884a-0c1952d9fa42', {'lang': 'py', 'task': 'apps.core.tasks._caller', 'id': '2e1f7404-9b42-405a-884a-0c1952d9fa42', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97e8774a-c137-4b29-8a72-6bc409c2f903', 'parent_id': '97e8774a-c137-4b29-8a72-6bc409c2f903', 'argsrepr': '(2198,)', 'kwargsrepr': '{}', 'origin': 'gen15@ip-10-0-0-119.ec2.internal', 'reply_to': '9f3f6ccf-a080-311b-b175-94374a43a954', 'correlation_id': '2e1f7404-9b42-405a-884a-0c1952d9fa42', 'hostname': 'worker-default@ip-10-0-0-119.ec2.internal', 'delivery_info': {'exchange': '', 'routing_key': 'default', 'priority': 0, 'redelivered': None}, 'args': [2198], 'kwargs': {}}, b'[[2198], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-08-18 11:58:51,981: DEBUG/MainProcess] Task accepted: apps.core.tasks._caller[2e1f7404-9b42-405a-884a-0c1952d9fa42] pid:15
[2020-08-18 11:58:52,013: DEBUG/ForkPoolWorker-1] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
[2020-08-18 11:58:52,015: DEBUG/ForkPoolWorker-1] Changing event name from before-call.apigateway to before-call.api-gateway
[2020-08-18 11:58:52,016: DEBUG/ForkPoolWorker-1] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
[2020-08-18 11:58:52,018: DEBUG/ForkPoolWorker-1] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
[2020-08-18 11:58:52,018: DEBUG/ForkPoolWorker-1] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
[2020-08-18 11:58:52,019: DEBUG/ForkPoolWorker-1] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
[2020-08-18 11:58:52,020: DEBUG/ForkPoolWorker-1] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
[2020-08-18 11:58:52,023: DEBUG/ForkPoolWorker-1] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
[2020-08-18 11:58:52,023: DEBUG/ForkPoolWorker-1] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
[2020-08-18 11:58:52,023: DEBUG/ForkPoolWorker-1] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
[2020-08-18 11:58:52,024: DEBUG/ForkPoolWorker-1] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
[2020-08-18 11:58:52,033: DEBUG/ForkPoolWorker-1] Loading JSON file: /usr/local/lib/python3.8/site-packages/boto3/data/s3/2006-03-01/resources-1.json
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: env
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: assume-role
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: assume-role-with-web-identity
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: sso
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: shared-credentials-file
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: custom-process
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: config-file
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: ec2-credentials-file
[2020-08-18 11:58:52,037: DEBUG/ForkPoolWorker-1] Looking for credentials via: boto-config
[2020-08-18 11:58:52,038: DEBUG/ForkPoolWorker-1] Looking for credentials via: container-role
[2020-08-18 11:58:52,039: DEBUG/ForkPoolWorker-1] Starting new HTTP connection (1): 169.254.170.2:80
[2020-08-18 11:58:52,041: DEBUG/ForkPoolWorker-1] http://163.124.180.14:80 "GET /v2/credentials/48679b14-11a3-4f00-887f-23dc86e0a603 HTTP/1.1" 200 1397
[2020-08-18 11:58:52,042: DEBUG/ForkPoolWorker-1] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/endpoints.json
[2020-08-18 11:58:52,049: DEBUG/ForkPoolWorker-1] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f9013175a60>
[2020-08-18 11:58:52,057: DEBUG/ForkPoolWorker-1] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/s3/2006-03-01/service-2.json
[2020-08-18 11:58:52,064: DEBUG/ForkPoolWorker-1] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f9013286700>
[2020-08-18 11:58:52,064: DEBUG/ForkPoolWorker-1] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f8fda2274c0>
[2020-08-18 11:58:52,065: DEBUG/ForkPoolWorker-1] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f90132864c0>
[2020-08-18 11:58:52,066: DEBUG/ForkPoolWorker-1] Setting s3 timeout as (60, 60)
[2020-08-18 11:58:52,067: DEBUG/ForkPoolWorker-1] Loading JSON file: /usr/local/lib/python3.8/site-packages/botocore/data/_retry.json
[2020-08-18 11:58:52,068: DEBUG/ForkPoolWorker-1] Registering retry handlers for service: s3
[2020-08-18 11:58:52,069: DEBUG/ForkPoolWorker-1] Loading s3:s3
[2020-08-18 11:58:52,071: DEBUG/ForkPoolWorker-1] Loading s3:Bucket
[2020-08-18 11:58:52,071: DEBUG/ForkPoolWorker-1] Renaming Bucket attribute name
[2020-08-18 11:58:52,072: DEBUG/ForkPoolWorker-1] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7f8ff777b550>
[2020-08-18 11:58:52,073: DEBUG/ForkPoolWorker-1] Loading s3:Object
[2020-08-18 11:58:52,074: DEBUG/ForkPoolWorker-1] Event creating-resource-class.s3.Object: calling handler <function lazy_call.<locals>._handler at 0x7f8ff777b5e0>
[2020-08-18 11:58:52,074: DEBUG/ForkPoolWorker-1] Calling s3:head_object with {'Bucket': 'my-bucket', 'Key': 'media/private/reportfiles/my_file.xlsx'}
[2020-08-18 11:58:52,075: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7f9013053430>
[2020-08-18 11:58:52,075: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7f90130533a0>
[2020-08-18 11:58:52,075: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,075: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f8ff7334220>>
[2020-08-18 11:58:52,075: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7f90130531f0>
[2020-08-18 11:58:52,076: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f9013053700>
[2020-08-18 11:58:52,076: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,076: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7f9013054a60>
[2020-08-18 11:58:52,076: DEBUG/ForkPoolWorker-1] Making request for OperationModel(name=HeadObject) with params: {'url_path': '/my-bucket/media/private/reportfiles/my_file.xlsx', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f8ff7309670>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'my-bucket'}}}
[2020-08-18 11:58:52,076: DEBUG/ForkPoolWorker-1] Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ff73095e0>>
[2020-08-18 11:58:52,077: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f8ff776f4c0>>
[2020-08-18 11:58:52,077: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7f90130530d0>
[2020-08-18 11:58:52,077: DEBUG/ForkPoolWorker-1] Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f8ff73342b0>>
[2020-08-18 11:58:52,077: DEBUG/ForkPoolWorker-1] Defaulting to S3 virtual host style addressing with path style addressing fallback.
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] Checking for DNS compatible bucket for: https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] URI updated to: https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] Calculating signature using v4 auth.
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] CanonicalRequest:
HEAD
/media/private/reportfiles/my_file.xlsx
host:my-bucket.s3.amazonaws.com
x-amz-content-sha256:<sha>
x-amz-date:20200818T115852Z
x-amz-security-token:<token>
host;x-amz-content-sha256;x-amz-date;x-amz-security-token
<sha>
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] StringToSign:
AWS4-HMAC-SHA256
<signature>
[2020-08-18 11:58:52,078: DEBUG/ForkPoolWorker-1] Signature:
<signature>
[2020-08-18 11:58:52,079: DEBUG/ForkPoolWorker-1] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx, headers={'User-Agent': b'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource', 'X-Amz-Date': b'20200818T115852Z', 'X-Amz-Security-Token': b'<token>', 'X-Amz-Content-SHA256': b'<sha>', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAZUMVIPFTNPT6MIPS/20200818/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=05ab3f4df656190e17b89709c98837fbda793db3fe99cc5a9f35198c6a6e5113'}>
[2020-08-18 11:58:52,079: DEBUG/ForkPoolWorker-1] Starting new HTTPS connection (1): my-bucket.s3.amazonaws.com:443
[2020-08-18 11:58:52,137: DEBUG/ForkPoolWorker-1] https://my-bucket.s3.amazonaws.com:443 "HEAD /media/private/reportfiles/my_file.xlsx HTTP/1.1" 200 0
[2020-08-18 11:58:52,138: DEBUG/ForkPoolWorker-1] Response headers: {'x-amz-id-2': '<value1>', 'x-amz-request-id': '<request-id>', 'Date': 'Tue, 18 Aug 2020 11:58:53 GMT', 'Last-Modified': 'Mon, 17 Aug 2020 17:20:20 GMT', 'ETag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'Cache-Control': 'max-age=86400', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'Content-Length': '15726922', 'Server': 'AmazonS3'}
[2020-08-18 11:58:52,138: DEBUG/ForkPoolWorker-1] Response body:
b''
[2020-08-18 11:58:52,139: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8ff7334100>
[2020-08-18 11:58:52,139: DEBUG/ForkPoolWorker-1] No retry needed.
[2020-08-18 11:58:52,139: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,139: DEBUG/ForkPoolWorker-1] Response: {'ResponseMetadata': {'RequestId': '<request-id>', 'HostId': '<value1>', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '<value>', 'x-amz-request-id': '<request-id>', 'date': 'Tue, 18 Aug 2020 11:58:53 GMT', 'last-modified': 'Mon, 17 Aug 2020 17:20:20 GMT', 'etag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'cache-control': 'max-age=86400', 'accept-ranges': 'bytes', 'content-type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'content-length': '15726922', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'AcceptRanges': 'bytes', 'LastModified': datetime.datetime(2020, 8, 17, 17, 20, 20, tzinfo=tzutc()), 'ContentLength': 15726922, 'ETag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'CacheControl': 'max-age=86400', 'ContentType': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'Metadata': {}}
[2020-08-18 11:58:52,141: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,141: DEBUG/ForkPoolWorker-1] DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f8ff729c670>}) about to wait for the following futures []
[2020-08-18 11:58:52,141: DEBUG/ForkPoolWorker-1] DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f8ff729c670>}) done waiting for dependent futures
[2020-08-18 11:58:52,142: DEBUG/ForkPoolWorker-1] Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f8ff729c670>}) with kwargs {'client': <botocore.client.S3 object at 0x7f8ff73094c0>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7f8ff72fb880>, 'osutil': <s3transfer.utils.OSUtils object at 0x7f8ff72fbbe0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7f8ff72fb910>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7f8ff729c670>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070>}
[2020-08-18 11:58:52,142: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7f9013053430>
[2020-08-18 11:58:52,142: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7f90130533a0>
[2020-08-18 11:58:52,142: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,142: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f8ff7334220>>
[2020-08-18 11:58:52,143: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7f90130531f0>
[2020-08-18 11:58:52,143: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7f9013053700>
[2020-08-18 11:58:52,143: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,143: DEBUG/ForkPoolWorker-1] Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x7f9013054a60>
[2020-08-18 11:58:52,143: DEBUG/ForkPoolWorker-1] Making request for OperationModel(name=HeadObject) with params: {'url_path': '/my-bucket/media/private/reportfiles/my_file.xlsx', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f8ff7309670>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'my-bucket'}}}
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7f90128e3ca0>
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ff73095e0>>
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f8ff776f4c0>>
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7f90130530d0>
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f8ff73342b0>>
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] Checking for DNS compatible bucket for: https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,144: DEBUG/ForkPoolWorker-1] URI updated to: https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] Calculating signature using v4 auth.
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] CanonicalRequest:
HEAD
/media/private/reportfiles/my_file.xlsx
host:my-bucket.s3.amazonaws.com
x-amz-content-sha256:<sha>
x-amz-date:20200818T115852Z
x-amz-security-token:<token>
host;x-amz-content-sha256;x-amz-date;x-amz-security-token
<sha>
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] StringToSign:
AWS4-HMAC-SHA256
<signature>
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] Signature:
<signature>
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7f90128e3d30>
[2020-08-18 11:58:52,145: DEBUG/ForkPoolWorker-1] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx, headers={'User-Agent': b'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource', 'X-Amz-Date': b'20200818T115852Z', 'X-Amz-Security-Token': b'<token>', 'X-Amz-Content-SHA256': b'<sha>', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAZUMVIPFTNPT6MIPS/20200818/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=05ab3f4df656190e17b89709c98837fbda793db3fe99cc5a9f35198c6a6e5113'}>
[2020-08-18 11:58:52,154: DEBUG/ForkPoolWorker-1] https://my-bucket.s3.amazonaws.com:443 "HEAD /media/private/reportfiles/my_file.xlsx HTTP/1.1" 200 0
[2020-08-18 11:58:52,154: DEBUG/ForkPoolWorker-1] Response headers: {'x-amz-id-2': '<value3>', 'x-amz-request-id': '<request-id-3>', 'Date': 'Tue, 18 Aug 2020 11:58:53 GMT', 'Last-Modified': 'Mon, 17 Aug 2020 17:20:20 GMT', 'ETag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'Cache-Control': 'max-age=86400', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'Content-Length': '15726922', 'Server': 'AmazonS3'}
[2020-08-18 11:58:52,154: DEBUG/ForkPoolWorker-1] Response body:
b''
[2020-08-18 11:58:52,155: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8ff7334100>
[2020-08-18 11:58:52,155: DEBUG/ForkPoolWorker-1] No retry needed.
[2020-08-18 11:58:52,155: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,155: DEBUG/ForkPoolWorker-1] Submitting task GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=0-8388607'}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff72fb910> for transfer request: 0.
[2020-08-18 11:58:52,156: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,156: DEBUG/ForkPoolWorker-1] GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=0-8388607'}}) about to wait for the following futures []
[2020-08-18 11:58:52,156: DEBUG/ForkPoolWorker-1] Submitting task GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=8388608-'}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff72fb910> for transfer request: 0.
[2020-08-18 11:58:52,156: DEBUG/ForkPoolWorker-1] GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=0-8388607'}}) done waiting for dependent futures
[2020-08-18 11:58:52,156: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,157: DEBUG/ForkPoolWorker-1] Executing task GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=0-8388607'}}) with kwargs {'client': <botocore.client.S3 object at 0x7f8ff73094c0>, 'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'extra_args': {'Range': 'bytes=0-8388607'}, 'callbacks': [], 'max_attempts': 5, 'start_index': 0, 'download_output_manager': <s3transfer.download.DownloadSeekableOutputManager object at 0x7f8ff7334310>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
[2020-08-18 11:58:52,157: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7f9013053430>
[2020-08-18 11:58:52,157: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7f90130533a0>
[2020-08-18 11:58:52,157: DEBUG/ForkPoolWorker-1] GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=8388608-'}}) about to wait for the following futures []
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=8388608-'}}) done waiting for dependent futures
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f8ff7334220>>
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] Executing task GetObjectTask(transfer_id=0, {'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'extra_args': {'Range': 'bytes=8388608-'}}) with kwargs {'client': <botocore.client.S3 object at 0x7f8ff73094c0>, 'bucket': 'my-bucket', 'key': 'media/private/reportfiles/my_file.xlsx', 'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'extra_args': {'Range': 'bytes=8388608-'}, 'callbacks': [], 'max_attempts': 5, 'start_index': 8388608, 'download_output_manager': <s3transfer.download.DownloadSeekableOutputManager object at 0x7f8ff7334310>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
[2020-08-18 11:58:52,158: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7f90130531f0>
[2020-08-18 11:58:52,159: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7f9013053430>
[2020-08-18 11:58:52,159: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7f9013053700>
[2020-08-18 11:58:52,159: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7f90130533a0>
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7f9013054a60>
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f8ff7334220>>
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Making request for OperationModel(name=GetObject) with params: {'url_path': '/my-bucket/media/private/reportfiles/my_file.xlsx', 'query_string': {}, 'method': 'GET', 'headers': {'Range': 'bytes=0-8388607', 'User-Agent': 'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f8ff7309670>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'my-bucket'}}}
[2020-08-18 11:58:52,160: DEBUG/ForkPoolWorker-1] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7f90130531f0>
[2020-08-18 11:58:52,161: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7f90128e3ca0>
[2020-08-18 11:58:52,161: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7f9013053700>
[2020-08-18 11:58:52,161: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ff73095e0>>
[2020-08-18 11:58:52,162: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,162: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f8ff776f4c0>>
[2020-08-18 11:58:52,162: DEBUG/ForkPoolWorker-1] Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7f9013054a60>
[2020-08-18 11:58:52,162: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7f90130530d0>
[2020-08-18 11:58:52,162: DEBUG/ForkPoolWorker-1] Making request for OperationModel(name=GetObject) with params: {'url_path': '/my-bucket/media/private/reportfiles/my_file.xlsx', 'query_string': {}, 'method': 'GET', 'headers': {'Range': 'bytes=8388608-', 'User-Agent': 'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f8ff7309670>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'my-bucket'}}}
[2020-08-18 11:58:52,163: DEBUG/ForkPoolWorker-1] Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f8ff73342b0>>
[2020-08-18 11:58:52,163: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7f90128e3ca0>
[2020-08-18 11:58:52,163: DEBUG/ForkPoolWorker-1] Checking for DNS compatible bucket for: https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,163: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8ff73095e0>>
[2020-08-18 11:58:52,163: DEBUG/ForkPoolWorker-1] URI updated to: https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,164: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f8ff776f4c0>>
[2020-08-18 11:58:52,164: DEBUG/ForkPoolWorker-1] Calculating signature using v4 auth.
[2020-08-18 11:58:52,164: DEBUG/ForkPoolWorker-1] Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7f90130530d0>
[2020-08-18 11:58:52,164: DEBUG/ForkPoolWorker-1] CanonicalRequest:
GET
/media/private/reportfiles/my_file.xlsx
host:my-bucket.s3.amazonaws.com
range:bytes=0-8388607
x-amz-content-sha256:<sha>
x-amz-date:20200818T115852Z
x-amz-security-token:<token>
host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token
<sha>
[2020-08-18 11:58:52,164: DEBUG/ForkPoolWorker-1] Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f8ff73342b0>>
[2020-08-18 11:58:52,165: DEBUG/ForkPoolWorker-1] StringToSign:
AWS4-HMAC-SHA256
fc8f60c14f5f95615b1170b5d3e18a57c7d6baa725cb291248e22f8934606f32
[2020-08-18 11:58:52,165: DEBUG/ForkPoolWorker-1] Checking for DNS compatible bucket for: https://s3.amazonaws.com/my-bucket/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,165: DEBUG/ForkPoolWorker-1] Signature:
<signature1>
[2020-08-18 11:58:52,165: DEBUG/ForkPoolWorker-1] URI updated to: https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx
[2020-08-18 11:58:52,165: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7f90128e3d30>
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] Calculating signature using v4 auth.
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] CanonicalRequest:
GET
/media/private/reportfiles/my_file.xlsx
host:my-bucket.s3.amazonaws.com
range:bytes=8388608-
x-amz-content-sha256:<sha>
x-amz-date:20200818T115852Z
x-amz-security-token:<token>
host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token
<sha>
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] StringToSign:
AWS4-HMAC-SHA256
<signature>
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] Signature:
<signature>
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7f90128e3d30>
[2020-08-18 11:58:52,167: DEBUG/ForkPoolWorker-1] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx, headers={'Range': b'bytes=8388608-', 'User-Agent': b'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource', 'X-Amz-Date': b'20200818T115852Z', 'X-Amz-Security-Token': b'<token>', 'X-Amz-Content-SHA256': b'<sha>', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAZUMVIPFTNPT6MIPS/20200818/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<signature-1>'}>
[2020-08-18 11:58:52,166: DEBUG/ForkPoolWorker-1] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://my-bucket.s3.amazonaws.com/media/private/reportfiles/my_file.xlsx, headers={'Range': b'bytes=0-8388607', 'User-Agent': b'Boto3/1.14.39 Python/3.8.5 Linux/4.14.158-129.185.amzn2.x86_64 exec-env/AWS_ECS_FARGATE Botocore/1.17.43 Resource', 'X-Amz-Date': b'20200818T115852Z', 'X-Amz-Security-Token': b'<token>', 'X-Amz-Content-SHA256': b'<sha>', 'Authorization': b'AWS4-HMAC-SHA256 Credential=ASIAZUMVIPFTNPT6MIPS/20200818/us-east-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<signature-2>'}>
[2020-08-18 11:58:52,168: DEBUG/ForkPoolWorker-1] Starting new HTTPS connection (2): my-bucket.s3.amazonaws.com:443
[2020-08-18 11:58:52,258: DEBUG/ForkPoolWorker-1] https://my-bucket.s3.amazonaws.com:443 "GET /media/private/reportfiles/my_file.xlsx HTTP/1.1" 206 7338314
[2020-08-18 11:58:52,258: DEBUG/ForkPoolWorker-1] Response headers: {'x-amz-id-2': '<value2>', 'x-amz-request-id': '<request-id-2>', 'Date': 'Tue, 18 Aug 2020 11:58:53 GMT', 'Last-Modified': 'Mon, 17 Aug 2020 17:20:20 GMT', 'ETag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'Cache-Control': 'max-age=86400', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 8388608-15726921/15726922', 'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'Content-Length': '7338314', 'Server': 'AmazonS3'}
[2020-08-18 11:58:52,258: DEBUG/ForkPoolWorker-1] Response body:
<botocore.response.StreamingBody object at 0x7f8ff7249d60>
[2020-08-18 11:58:52,259: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8ff7334100>
[2020-08-18 11:58:52,259: DEBUG/ForkPoolWorker-1] No retry needed.
[2020-08-18 11:58:52,259: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,264: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 8388608}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,264: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,264: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8388608}) about to wait for the following futures []
[2020-08-18 11:58:52,264: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8388608}) done waiting for dependent futures
[2020-08-18 11:58:52,265: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 8388608}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 8388608}
[2020-08-18 11:58:52,271: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,272: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 8650752}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,272: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,272: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8650752}) about to wait for the following futures []
[2020-08-18 11:58:52,272: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8650752}) done waiting for dependent futures
[2020-08-18 11:58:52,272: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 8650752}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 8650752}
[2020-08-18 11:58:52,273: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,274: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 8912896}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,274: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,274: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8912896}) about to wait for the following futures []
[2020-08-18 11:58:52,274: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8912896}) done waiting for dependent futures
[2020-08-18 11:58:52,275: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 8912896}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 8912896}
[2020-08-18 11:58:52,275: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,276: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 9175040}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,276: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,277: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9175040}) about to wait for the following futures []
[2020-08-18 11:58:52,277: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9175040}) done waiting for dependent futures
[2020-08-18 11:58:52,277: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 9175040}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 9175040}
[2020-08-18 11:58:52,277: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,278: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 9437184}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,278: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,279: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9437184}) about to wait for the following futures []
[2020-08-18 11:58:52,279: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9437184}) done waiting for dependent futures
[2020-08-18 11:58:52,279: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 9437184}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 9437184}
[2020-08-18 11:58:52,280: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,281: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 9699328}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,281: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,282: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9699328}) about to wait for the following futures []
[2020-08-18 11:58:52,282: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9699328}) done waiting for dependent futures
[2020-08-18 11:58:52,282: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 9699328}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 9699328}
[2020-08-18 11:58:52,282: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,283: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 9961472}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,283: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,284: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 10223616}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,284: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,284: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9961472}) about to wait for the following futures []
[2020-08-18 11:58:52,284: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 9961472}) done waiting for dependent futures
[2020-08-18 11:58:52,284: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 9961472}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 9961472}
[2020-08-18 11:58:52,285: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,285: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10223616}) about to wait for the following futures []
[2020-08-18 11:58:52,285: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10223616}) done waiting for dependent futures
[2020-08-18 11:58:52,285: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 10223616}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 10223616}
[2020-08-18 11:58:52,285: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,286: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 10485760}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,286: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,286: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10485760}) about to wait for the following futures []
[2020-08-18 11:58:52,287: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10485760}) done waiting for dependent futures
[2020-08-18 11:58:52,287: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 10485760}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 10485760}
[2020-08-18 11:58:52,287: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,288: DEBUG/ForkPoolWorker-1] https://my-bucket.s3.amazonaws.com:443 "GET /media/private/reportfiles/my_file.xlsx HTTP/1.1" 206 8388608
[2020-08-18 11:58:52,288: DEBUG/ForkPoolWorker-1] Response headers: {'x-amz-id-2': '<header>', 'x-amz-request-id': '<request-id>', 'Date': 'Tue, 18 Aug 2020 11:58:53 GMT', 'Last-Modified': 'Mon, 17 Aug 2020 17:20:20 GMT', 'ETag': '"282b727827a19e44f38be8ea8491fdf0-2"', 'Cache-Control': 'max-age=86400', 'Accept-Ranges': 'bytes', 'Content-Range': 'bytes 0-8388607/15726922', 'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet', 'Content-Length': '8388608', 'Server': 'AmazonS3'}
[2020-08-18 11:58:52,288: DEBUG/ForkPoolWorker-1] Response body:
<botocore.response.StreamingBody object at 0x7f8ff7258550>
[2020-08-18 11:58:52,290: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8ff7334100>
[2020-08-18 11:58:52,290: DEBUG/ForkPoolWorker-1] No retry needed.
[2020-08-18 11:58:52,290: DEBUG/ForkPoolWorker-1] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8ff7334160>>
[2020-08-18 11:58:52,291: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 10747904}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,291: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,292: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10747904}) about to wait for the following futures []
[2020-08-18 11:58:52,292: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 10747904}) done waiting for dependent futures
[2020-08-18 11:58:52,292: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 10747904}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 10747904}
[2020-08-18 11:58:52,292: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,293: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 11010048}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,294: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,294: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11010048}) about to wait for the following futures []
[2020-08-18 11:58:52,294: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11010048}) done waiting for dependent futures
[2020-08-18 11:58:52,295: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 11010048}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 11010048}
[2020-08-18 11:58:52,295: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,296: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 0}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,296: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,296: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
[2020-08-18 11:58:52,296: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
[2020-08-18 11:58:52,297: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 0}
[2020-08-18 11:58:52,297: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,298: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 11272192}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,298: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,299: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11272192}) about to wait for the following futures []
[2020-08-18 11:58:52,299: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11272192}) done waiting for dependent futures
[2020-08-18 11:58:52,299: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 11272192}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 11272192}
[2020-08-18 11:58:52,299: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,300: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 262144}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,300: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,301: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 262144}) about to wait for the following futures []
[2020-08-18 11:58:52,301: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 262144}) done waiting for dependent futures
[2020-08-18 11:58:52,301: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 262144}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 262144}
[2020-08-18 11:58:52,301: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,302: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 11534336}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,302: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,303: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11534336}) about to wait for the following futures []
[2020-08-18 11:58:52,303: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11534336}) done waiting for dependent futures
[2020-08-18 11:58:52,303: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 11534336}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 11534336}
[2020-08-18 11:58:52,304: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,304: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 11796480}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,304: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,305: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11796480}) about to wait for the following futures []
[2020-08-18 11:58:52,305: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 11796480}) done waiting for dependent futures
[2020-08-18 11:58:52,305: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 11796480}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 11796480}
[2020-08-18 11:58:52,305: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,306: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 12058624}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,306: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,306: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 12320768}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,307: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,307: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 524288}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,307: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,308: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12058624}) about to wait for the following futures []
[2020-08-18 11:58:52,308: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12058624}) done waiting for dependent futures
[2020-08-18 11:58:52,308: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 12058624}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 12058624}
[2020-08-18 11:58:52,308: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,309: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12320768}) about to wait for the following futures []
[2020-08-18 11:58:52,309: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12320768}) done waiting for dependent futures
[2020-08-18 11:58:52,309: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 12320768}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 12320768}
[2020-08-18 11:58:52,309: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,309: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 524288}) about to wait for the following futures []
[2020-08-18 11:58:52,310: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 524288}) done waiting for dependent futures
[2020-08-18 11:58:52,310: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 524288}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 524288}
[2020-08-18 11:58:52,310: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,311: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 12582912}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,311: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,311: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12582912}) about to wait for the following futures []
[2020-08-18 11:58:52,311: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12582912}) done waiting for dependent futures
[2020-08-18 11:58:52,311: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 12582912}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 12582912}
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 786432}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 786432}) about to wait for the following futures []
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 786432}) done waiting for dependent futures
[2020-08-18 11:58:52,312: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 786432}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 786432}
[2020-08-18 11:58:52,313: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,314: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 12845056}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,314: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,314: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12845056}) about to wait for the following futures []
[2020-08-18 11:58:52,314: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 12845056}) done waiting for dependent futures
[2020-08-18 11:58:52,314: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 12845056}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 12845056}
[2020-08-18 11:58:52,315: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,315: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 1048576}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,315: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,316: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 1310720}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,316: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,316: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1048576}) about to wait for the following futures []
[2020-08-18 11:58:52,316: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1048576}) done waiting for dependent futures
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 1048576}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 1048576}
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1310720}) about to wait for the following futures []
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1310720}) done waiting for dependent futures
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 1310720}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 1310720}
[2020-08-18 11:58:52,317: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,318: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 13107200}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,318: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,318: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13107200}) about to wait for the following futures []
[2020-08-18 11:58:52,318: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13107200}) done waiting for dependent futures
[2020-08-18 11:58:52,318: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 13107200}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 13107200}
[2020-08-18 11:58:52,319: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,319: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 13369344}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,319: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,320: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 13631488}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,320: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,321: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 13893632}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,321: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,322: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 14155776}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,322: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,323: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 14417920}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,323: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,323: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13369344}) about to wait for the following futures []
[2020-08-18 11:58:52,323: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13369344}) done waiting for dependent futures
[2020-08-18 11:58:52,323: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 13369344}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 13369344}
[2020-08-18 11:58:52,324: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,324: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13631488}) about to wait for the following futures []
[2020-08-18 11:58:52,324: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13631488}) done waiting for dependent futures
[2020-08-18 11:58:52,324: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 13631488}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 13631488}
[2020-08-18 11:58:52,324: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13893632}) about to wait for the following futures []
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 13893632}) done waiting for dependent futures
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 13893632}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 13893632}
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14155776}) about to wait for the following futures []
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14155776}) done waiting for dependent futures
[2020-08-18 11:58:52,325: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 14155776}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 14155776}
[2020-08-18 11:58:52,326: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,326: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14417920}) about to wait for the following futures []
[2020-08-18 11:58:52,326: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14417920}) done waiting for dependent futures
[2020-08-18 11:58:52,326: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 14417920}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 14417920}
[2020-08-18 11:58:52,326: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,327: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 1572864}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,327: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,327: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1572864}) about to wait for the following futures []
[2020-08-18 11:58:52,328: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1572864}) done waiting for dependent futures
[2020-08-18 11:58:52,328: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 1572864}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 1572864}
[2020-08-18 11:58:52,328: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,328: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 14680064}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,328: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,329: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14680064}) about to wait for the following futures []
[2020-08-18 11:58:52,329: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14680064}) done waiting for dependent futures
[2020-08-18 11:58:52,329: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 14680064}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 14680064}
[2020-08-18 11:58:52,329: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,330: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 1835008}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,331: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,331: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1835008}) about to wait for the following futures []
[2020-08-18 11:58:52,331: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 1835008}) done waiting for dependent futures
[2020-08-18 11:58:52,331: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 1835008}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 1835008}
[2020-08-18 11:58:52,331: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,332: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 14942208}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,332: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,332: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14942208}) about to wait for the following futures []
[2020-08-18 11:58:52,332: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 14942208}) done waiting for dependent futures
[2020-08-18 11:58:52,333: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 2097152}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,333: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 14942208}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 14942208}
[2020-08-18 11:58:52,333: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,334: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 15204352}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,335: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,334: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,335: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2097152}) about to wait for the following futures []
[2020-08-18 11:58:52,335: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2097152}) done waiting for dependent futures
[2020-08-18 11:58:52,335: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 2097152}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 2097152}
[2020-08-18 11:58:52,336: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,336: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 15204352}) about to wait for the following futures []
[2020-08-18 11:58:52,336: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 15204352}) done waiting for dependent futures
[2020-08-18 11:58:52,337: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 15204352}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 15204352}
[2020-08-18 11:58:52,337: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,338: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 2359296}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,338: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,338: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2359296}) about to wait for the following futures []
[2020-08-18 11:58:52,338: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2359296}) done waiting for dependent futures
[2020-08-18 11:58:52,338: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 2359296}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 2359296}
[2020-08-18 11:58:52,339: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,339: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 15466496}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,339: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,340: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,340: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 15466496}) about to wait for the following futures []
[2020-08-18 11:58:52,340: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 15466496}) done waiting for dependent futures
[2020-08-18 11:58:52,340: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 15466496}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 15466496}
[2020-08-18 11:58:52,340: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,341: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 2621440}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,341: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,341: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2621440}) about to wait for the following futures []
[2020-08-18 11:58:52,341: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2621440}) done waiting for dependent futures
[2020-08-18 11:58:52,341: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 2621440}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 2621440}
[2020-08-18 11:58:52,342: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,342: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 2883584}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,342: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,343: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2883584}) about to wait for the following futures []
[2020-08-18 11:58:52,343: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 2883584}) done waiting for dependent futures
[2020-08-18 11:58:52,343: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 2883584}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 2883584}
[2020-08-18 11:58:52,343: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,344: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 3145728}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,344: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,344: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3145728}) about to wait for the following futures []
[2020-08-18 11:58:52,344: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3145728}) done waiting for dependent futures
[2020-08-18 11:58:52,344: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 3145728}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 3145728}
[2020-08-18 11:58:52,345: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,345: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 3407872}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,345: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,346: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3407872}) about to wait for the following futures []
[2020-08-18 11:58:52,346: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3407872}) done waiting for dependent futures
[2020-08-18 11:58:52,346: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 3407872}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 3407872}
[2020-08-18 11:58:52,346: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 3670016}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3670016}) about to wait for the following futures []
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3670016}) done waiting for dependent futures
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 3670016}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 3670016}
[2020-08-18 11:58:52,347: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,348: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 3932160}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,348: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,349: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3932160}) about to wait for the following futures []
[2020-08-18 11:58:52,349: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 3932160}) done waiting for dependent futures
[2020-08-18 11:58:52,349: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 3932160}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 3932160}
[2020-08-18 11:58:52,350: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,350: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 4194304}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,350: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,350: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4194304}) about to wait for the following futures []
[2020-08-18 11:58:52,351: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4194304}) done waiting for dependent futures
[2020-08-18 11:58:52,351: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 4194304}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 4194304}
[2020-08-18 11:58:52,351: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,352: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 4456448}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,352: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,353: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4456448}) about to wait for the following futures []
[2020-08-18 11:58:52,353: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4456448}) done waiting for dependent futures
[2020-08-18 11:58:52,353: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 4456448}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 4456448}
[2020-08-18 11:58:52,353: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,354: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 4718592}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,354: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,354: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4718592}) about to wait for the following futures []
[2020-08-18 11:58:52,354: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4718592}) done waiting for dependent futures
[2020-08-18 11:58:52,354: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 4718592}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 4718592}
[2020-08-18 11:58:52,355: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,355: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 4980736}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,355: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,356: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 5242880}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,356: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,357: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 5505024}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,357: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,358: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 5767168}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,358: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 6029312}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4980736}) about to wait for the following futures []
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 4980736}) done waiting for dependent futures
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 4980736}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 4980736}
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,359: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5242880}) about to wait for the following futures []
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5242880}) done waiting for dependent futures
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 5242880}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 5242880}
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5505024}) about to wait for the following futures []
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5505024}) done waiting for dependent futures
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 5505024}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 5505024}
[2020-08-18 11:58:52,360: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5767168}) about to wait for the following futures []
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 5767168}) done waiting for dependent futures
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 5767168}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 5767168}
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6029312}) about to wait for the following futures []
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6029312}) done waiting for dependent futures
[2020-08-18 11:58:52,361: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 6029312}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 6029312}
[2020-08-18 11:58:52,362: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,362: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 6291456}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,362: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,362: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6291456}) about to wait for the following futures []
[2020-08-18 11:58:52,362: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6291456}) done waiting for dependent futures
[2020-08-18 11:58:52,363: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 6291456}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 6291456}
[2020-08-18 11:58:52,363: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 6553600}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6553600}) about to wait for the following futures []
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6553600}) done waiting for dependent futures
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 6553600}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 6553600}
[2020-08-18 11:58:52,364: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,365: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 6815744}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,365: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,365: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6815744}) about to wait for the following futures []
[2020-08-18 11:58:52,365: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 6815744}) done waiting for dependent futures
[2020-08-18 11:58:52,366: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 6815744}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 6815744}
[2020-08-18 11:58:52,366: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,368: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 7077888}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,368: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,369: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7077888}) about to wait for the following futures []
[2020-08-18 11:58:52,369: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7077888}) done waiting for dependent futures
[2020-08-18 11:58:52,369: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 7077888}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 7077888}
[2020-08-18 11:58:52,369: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,370: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 7340032}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,370: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,370: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7340032}) about to wait for the following futures []
[2020-08-18 11:58:52,370: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7340032}) done waiting for dependent futures
[2020-08-18 11:58:52,370: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 7340032}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 7340032}
[2020-08-18 11:58:52,371: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 7602176}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,371: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,371: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,371: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7602176}) about to wait for the following futures []
[2020-08-18 11:58:52,372: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7602176}) done waiting for dependent futures
[2020-08-18 11:58:52,372: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 7602176}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 7602176}
[2020-08-18 11:58:52,372: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,372: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 7864320}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,373: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,373: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7864320}) about to wait for the following futures []
[2020-08-18 11:58:52,373: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 7864320}) done waiting for dependent futures
[2020-08-18 11:58:52,373: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 7864320}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 7864320}
[2020-08-18 11:58:52,373: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,374: DEBUG/ForkPoolWorker-1] Submitting task IOWriteTask(transfer_id=0, {'offset': 8126464}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,374: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,374: DEBUG/ForkPoolWorker-1] Submitting task CompleteDownloadNOOPTask(transfer_id=0, {}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f8ff729c070> for transfer request: 0.
[2020-08-18 11:58:52,374: DEBUG/ForkPoolWorker-1] Acquiring 0
[2020-08-18 11:58:52,374: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8126464}) about to wait for the following futures []
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] IOWriteTask(transfer_id=0, {'offset': 8126464}) done waiting for dependent futures
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] Executing task IOWriteTask(transfer_id=0, {'offset': 8126464}) with kwargs {'fileobj': <tempfile.SpooledTemporaryFile object at 0x7f8ff777f130>, 'offset': 8126464}
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] CompleteDownloadNOOPTask(transfer_id=0, {}) about to wait for the following futures []
[2020-08-18 11:58:52,375: DEBUG/ForkPoolWorker-1] CompleteDownloadNOOPTask(transfer_id=0, {}) done waiting for dependent futures
[2020-08-18 11:58:52,376: DEBUG/ForkPoolWorker-1] Executing task CompleteDownloadNOOPTask(transfer_id=0, {}) with kwargs {}
[2020-08-18 11:58:52,376: DEBUG/ForkPoolWorker-1] Releasing acquire 0/None
[2020-08-18 11:58:53,033: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:58:58,032: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:03,033: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:08,032: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:13,033: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:18,032: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:23,033: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:28,032: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:33,044: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2020-08-18 11:59:38,033: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
worker: Warm shutdown (MainProcess)
[2020-08-18 11:59:39,677: DEBUG/MainProcess] | Worker: Closing Hub...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Worker: Closing Pool...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Worker: Closing Consumer...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Connection...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Events...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2020-08-18 11:59:39,678: DEBUG/MainProcess] | Consumer: Closing Control...
[2020-08-18 11:59:39,679: DEBUG/MainProcess] | Consumer: Closing Heart...
[2020-08-18 11:59:39,679: DEBUG/MainProcess] | Consumer: Closing event loop...
[2020-08-18 11:59:39,679: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2020-08-18 11:59:39,679: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2020-08-18 11:59:39,680: DEBUG/MainProcess] | Consumer: Stopping Control...
[2020-08-18 11:59:39,713: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2020-08-18 11:59:39,713: DEBUG/MainProcess] Canceling task consumer...
[2020-08-18 11:59:39,713: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2020-08-18 11:59:39,716: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2020-08-18 11:59:39,716: DEBUG/MainProcess] | Consumer: Stopping Events...
[2020-08-18 11:59:39,716: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2020-08-18 11:59:39,716: DEBUG/MainProcess] | Worker: Stopping Pool...
rollue commented 3 years ago

@briancaffey Did you manage to solve this? I'm having exactly the same problem from my AWS fargate django container :(

briancaffey commented 3 years ago

@mhoonjeon sorry, I think this was an issue with celery and I think it was resolved by upgrading celery packages. Are you still having the issue? Sorry for the late reply!

rollue commented 3 years ago

Thanks @briancaffey.

For me though, this was a problem with multiple rows in one of the django table sharing the same image . Specifically, my Event model was set to use a default image when a user does not upload one. This means thousands of event instances was referencing the same image in S3 bucket as their image.

Removing duplicate in DB rows solved this for me. Hope this helps someone else.