Improve logging

[helen-m-lin]

Is your feature request related to a problem? Please describe. As a service maintainer, I want to easily review job summary logs in AWS cloudwatch so that I can troubleshoot issues faster.

• Currently there are no summary logs for each bucket, and the s3 and docdb operations are being logged as raw results:

  INFO:root:{'ResponseMetadata': {'RequestId': '*************', 'HostId': '*************', 'HTTPStatusCode': 200, 
  '**************************', 'x-amz- 'date': 'Fri, 17 May 2024 20:08:09 GMT', 'x-amz-version-id': '**************-side-encryption': '***', 
  'etag': '"*******************************"', ength': '0'}, 'RetryAttempts': 0}, 'ETag': '"*******************************',
  'VersionId': '*************************'}

Describe the solution you'd like

• [ ] If applicable, update logs in populate_s3_with_metadata_files.py and aind_bucket_indexer.py to include summary counts (num created, updated, deleted, etc.) after each bucket.
• [ ] If applicable, log record ID, operation, and location before any logging.info(response)

Describe alternatives you've considered Leaving it as is does not affect running the jobs.

Additional context Add any other context or screenshots about the feature request here.

[helen-m-lin]

@jtyoung84 also suggested we should set up logging in the init/ top level modules rather than passing any log_flag