search

boto / s3transfer

Amazon S3 Transfer Manager for Python
Apache License 2.0
209 stars 133 forks source link

Add more logs to ProcessPoolDownloader #127

Closed kyleknap closed 5 years ago

kyleknap commented 5 years ago

Here is a sample snippet of what the logs now look like with this update:

2019-02-04 14:52:06,953 - s3transfer.processpool - DEBUG - Starting the TransferMonitorManager.
2019-02-04 14:52:06,965 - s3transfer.processpool - DEBUG - Starting the GetObjectSubmitter.
2019-02-04 14:52:06,967 - s3transfer.processpool - DEBUG - Starting 10 GetObjectWorkers.
2019-02-04 14:52:07,094 - s3transfer.processpool - DEBUG - Submitting download file request: DownloadFileRequest(transfer_id=0, bucket='mybucketfoo', key='1GB', filename='1GB', extra_args={}, expected_size=None).
2019-02-04 14:52:09,840 - s3transfer.processpool - DEBUG - Notifying 128 job(s) to complete for transfer_id 0.
2019-02-04 14:52:28,024 - s3transfer.processpool - DEBUG - 127 jobs remaining for transfer_id 0.
2019-02-04 14:52:28,536 - s3transfer.processpool - DEBUG - 126 jobs remaining for transfer_id 0.
2019-02-04 14:52:33,823 - s3transfer.processpool - DEBUG - 125 jobs remaining for transfer_id 0.
2019-02-04 14:52:35,206 - s3transfer.processpool - DEBUG - 124 jobs remaining for transfer_id 0.
... [SHORTENED] ...
2019-02-04 14:58:04,012 - s3transfer.processpool - DEBUG - 9 jobs remaining for transfer_id 0.
2019-02-04 14:58:08,168 - s3transfer.processpool - DEBUG - 8 jobs remaining for transfer_id 0.
2019-02-04 14:58:09,386 - s3transfer.processpool - DEBUG - 7 jobs remaining for transfer_id 0.
2019-02-04 14:58:09,458 - s3transfer.processpool - DEBUG - 6 jobs remaining for transfer_id 0.
2019-02-04 14:58:10,151 - s3transfer.processpool - DEBUG - 5 jobs remaining for transfer_id 0.
2019-02-04 14:58:11,549 - s3transfer.processpool - DEBUG - 4 jobs remaining for transfer_id 0.
2019-02-04 14:58:14,727 - s3transfer.processpool - DEBUG - 3 jobs remaining for transfer_id 0.
2019-02-04 14:58:15,231 - s3transfer.processpool - DEBUG - 2 jobs remaining for transfer_id 0.
2019-02-04 14:58:15,747 - s3transfer.processpool - DEBUG - 1 jobs remaining for transfer_id 0.
2019-02-04 14:58:16,834 - s3transfer.processpool - DEBUG - 0 jobs remaining for transfer_id 0.
2019-02-04 14:58:16,835 - s3transfer.processpool - DEBUG - Shutting down the GetObjectSubmitter.
2019-02-04 14:58:16,835 - s3transfer.processpool - DEBUG - Submitter shutdown signal received.
2019-02-04 14:58:16,840 - s3transfer.processpool - DEBUG - Shutting down the GetObjectWorker.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,841 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,842 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,842 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,842 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,842 - s3transfer.processpool - DEBUG - Worker shutdown signal received.
2019-02-04 14:58:16,867 - s3transfer.processpool - DEBUG - Shutting down the TransferMonitorManager.

Let me know if there are any other places you think we should add logs.