search

unity-sds / unity-data-services

Apache License 2.0
0 stars 3 forks source link

Add INFO logging for each uploaded file #436

Closed ngachung closed 1 week ago

wphyojpl commented 1 month ago

Instead of trying to add a custom level, we can use log.critical("<uploaded file>") which will always display whether we set the log level at debug, info, or error. The wording is a bit alarming coz it's critical, but that's what SWOT did when I worked for them.

wphyojpl commented 1 month ago

Auxiliary File test with this config:

logging.basicConfig(level=20, format="%(asctime)s [%(levelname)s] [%(name)s::%(lineno)d] %(message)s")

Result:

OK
2024-10-14 10:19:50,010 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:50,145 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
2024-10-14 10:19:50,453 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:50,560 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:50,611 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/son/granddaughter/test_file_3.json
2024-10-14 10:19:51,196 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:51,278 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:51,367 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/son/grandson/test_file_2.json
2024-10-14 10:19:51,850 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:51,925 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:51,971 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/son/test_file_0.json
2024-10-14 10:19:52,504 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:52,688 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:52,732 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/test_file_0.json
2024-10-14 10:19:53,287 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:53,370 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:53,472 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/daughter/granddaughter/test_file_5.json
2024-10-14 10:19:54,156 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:54,239 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:54,283 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/daughter/grandson/test_file_4.json
2024-10-14 10:19:54,801 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:54,958 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:19:55,004 [AUDIT] [mdps_ds_lib.stage_in_out.upload_arbitrary_files_as_granules::80] uploading auxiliary file: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpzw94_anh/auxiliary_base/daughter/test_file_1.json

Testing normal Upload:

2024-10-14 10:21:28,148 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:21:28,234 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
2024-10-14 10:21:28,527 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:21:28,629 [INFO] [botocore.credentials::1278] Found credentials in shared credentials file: ~/.aws/credentials
2024-10-14 10:21:28,680 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::151] multithread processing starting with process_count: 12
2024-10-14 10:21:28,683 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3372, exit_code: None
2024-10-14 10:21:28,685 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3373, exit_code: None
2024-10-14 10:21:28,688 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3374, exit_code: None
2024-10-14 10:21:28,692 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3375, exit_code: None
2024-10-14 10:21:28,695 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3376, exit_code: None
2024-10-14 10:21:28,698 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3377, exit_code: None
2024-10-14 10:21:28,702 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3378, exit_code: None
2024-10-14 10:21:28,705 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3379, exit_code: None
2024-10-14 10:21:28,707 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3380, exit_code: None
2024-10-14 10:21:28,711 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3381, exit_code: None
2024-10-14 10:21:28,714 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3382, exit_code: None
2024-10-14 10:21:28,717 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::158] starting consumer pid: 3383, exit_code: None
2024-10-14 10:21:28,719 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3372. exit_code: None
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
2024-10-14 10:21:28,803 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file01.nc
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
2024-10-14 10:21:28,812 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file02.nc
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
Error loading: /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd_attach_to_process/attach_x86_64.dylib
2024-10-14 10:21:28,827 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file03.nc
2024-10-14 10:21:29,276 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file02.nc.cas
2024-10-14 10:21:29,276 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file01.nc.cas
2024-10-14 10:21:29,277 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file03.nc.cas
2024-10-14 10:21:29,495 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file02.nc.stac.json
2024-10-14 10:21:29,507 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file01.nc.stac.json
2024-10-14 10:21:29,507 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file03.nc.stac.json
2024-10-14 10:21:29,723 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file04.nc
2024-10-14 10:21:29,800 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file05.nc
2024-10-14 10:21:29,801 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file06.nc
2024-10-14 10:21:29,807 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file07.nc
2024-10-14 10:21:29,811 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file08.nc
2024-10-14 10:21:29,931 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file04.nc.cas
2024-10-14 10:21:30,144 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file04.nc.stac.json
2024-10-14 10:21:30,177 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file05.nc.cas
2024-10-14 10:21:30,179 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file07.nc.cas
2024-10-14 10:21:30,185 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file08.nc.cas
2024-10-14 10:21:30,221 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file06.nc.cas
2024-10-14 10:21:30,385 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file09.nc
2024-10-14 10:21:30,400 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file08.nc.stac.json
2024-10-14 10:21:30,400 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file05.nc.stac.json
2024-10-14 10:21:30,404 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file07.nc.stac.json
2024-10-14 10:21:30,438 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file06.nc.stac.json
2024-10-14 10:21:30,590 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file09.nc.cas
2024-10-14 10:21:30,637 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading data: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file10.nc
2024-10-14 10:21:30,814 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file09.nc.stac.json
2024-10-14 10:21:30,845 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file10.nc.cas
2024-10-14 10:21:31,060 [AUDIT] [mdps_ds_lib.stage_in_out.upload_granules_by_complete_catalog_s3::67] uploading metadata: /var/folders/33/xhq97d6s0dq78wg4h2smw23m0000gq/T/tmpdssdsz_x/some_granules/./test_file10.nc.stac.json
2024-10-14 10:21:31,223 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3373. exit_code: None
2024-10-14 10:21:32,778 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3374. exit_code: None
2024-10-14 10:21:33,974 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3375. exit_code: 0
2024-10-14 10:21:33,974 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3376. exit_code: 0
2024-10-14 10:21:33,974 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3377. exit_code: None
2024-10-14 10:21:33,988 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3378. exit_code: 0
2024-10-14 10:21:33,989 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3379. exit_code: 0
2024-10-14 10:21:33,989 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3380. exit_code: 0
2024-10-14 10:21:33,989 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3381. exit_code: 0
2024-10-14 10:21:33,989 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3382. exit_code: 0
2024-10-14 10:21:33,989 [INFO] [mdps_ds_lib.lib.processing_jobs.multithread_processor::164] joining consumers: 3383. exit_code: 0
wphyojpl commented 1 month ago

The down side of splitting the repo is.. we have to update this repo after this PR is merged.