search

ets / tap-spreadsheets-anywhere

GNU Affero General Public License v3.0
31 stars 63 forks source link

Azure sync process logs quite noisy #57

Open acarter24 opened 1 year ago

acarter24 commented 1 year ago

Is there any way to reduce the http request/responses being printed to the logs? This was for a single table with two rows.

2023-05-18T12:32:39.201092Z [info     ] Environment 'dev' is active
2023-05-18T12:32:42.127043Z [warning  ] No state was found, complete import.
2023-05-18T12:32:42.528555Z [info     ] INFO Using supplied catalog /home/andycarter/vscode_projects/empiric_elt/empiric_meltano/.meltano/run/tap-spreadsheets-anywhere/tap.properties.json. cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.528749Z [info     ] INFO Processing 1 selected streams from Catalog cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.528866Z [info     ] INFO Syncing stream:test       cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.528960Z [info     ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer?restype=REDACTED&comp=REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529053Z [info     ] Request method: 'GET'          cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529154Z [info     ] Request headers:               cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529258Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529364Z [info     ]     'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529473Z [info     ]     'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529576Z [info     ]     'x-ms-date': 'REDACTED'    cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529690Z [info     ]     'x-ms-client-request-id': '15924b0a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.529840Z [info     ]     'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.530106Z [info     ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.714693Z [info     ] time=2023-05-18 13:32:42 name=target_postgres level=INFO message=Schema 'raw__tap_spreadsheets_anywhere' does not exist. Creating... CREATE SCHEMA IF NOT EXISTS raw__tap_spreadsheets_anywhere cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-05-18T12:32:42.717406Z [info     ] INFO Response status: 200      cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.717761Z [info     ] Response headers:              cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.718086Z [info     ]     'Transfer-Encoding': 'chunked' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.718241Z [info     ]     'Content-Type': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.718404Z [info     ]     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.719016Z [info     ]     'x-ms-request-id': 'fc5c41ae-701e-0014-2d84-89af86000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.719263Z [info     ]     'x-ms-client-request-id': '15924b0a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.719416Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.719569Z [info     ]     'Date': 'Thu, 18 May 2023 12:32:38 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.719833Z [info     ] INFO Checking 2 resolved objects for any that match regular expression "in/emp.txt" and were modified since 2017-05-01 00:00:00+00:00 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.720014Z [info     ] INFO Processing 1 resolved objects that met our criteria. Enable debug verbosity logging for more details. cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.720178Z [info     ] INFO Syncing file "in/emp.txt". cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.721358Z [info     ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer/in/emp.txt' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722021Z [info     ] Request method: 'HEAD'         cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722222Z [info     ] Request headers:               cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722348Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722473Z [info     ]     'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722585Z [info     ]     'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722705Z [info     ]     'x-ms-date': 'REDACTED'    cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.722902Z [info     ]     'x-ms-client-request-id': '15b1cf70-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.723121Z [info     ]     'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.723812Z [info     ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.776114Z [info     ] time=2023-05-18 13:32:42 name=target_postgres level=INFO message=Table '"test"' does not exist. Creating... CREATE TABLE IF NOT EXISTS raw__tap_spreadsheets_anywhere."test" ("_smart_source_bucket" character varying, "_smart_source_file" character varying, "_smart_source_lineno" numeric, "personidlastnamefirstnameaddresscity" character varying) cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-05-18T12:32:42.884202Z [info     ] INFO Response status: 200      cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.884708Z [info     ] Response headers:              cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.885120Z [info     ]     'Content-Length': '86'     cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.885423Z [info     ]     'Content-Type': 'text/plain' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.885700Z [info     ]     'Content-MD5': 'REDACTED'  cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.885974Z [info     ]     'Last-Modified': 'Fri, 12 May 2023 14:36:14 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.886300Z [info     ]     'Accept-Ranges': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.886755Z [info     ]     'ETag': '"0x8DB52F63D10EF6A"' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.887060Z [info     ]     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.887367Z [info     ]     'x-ms-request-id': '5d901676-301e-003a-0b84-89fd91000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.887883Z [info     ]     'x-ms-client-request-id': '15b1cf70-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.888275Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.888610Z [info     ]     'x-ms-creation-time': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.889107Z [info     ]     'x-ms-lease-status': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.890025Z [info     ]     'x-ms-lease-state': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.890645Z [info     ]     'x-ms-blob-type': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.891199Z [info     ]     'x-ms-server-encrypted': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.891704Z [info     ]     'x-ms-access-tier': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.892250Z [info     ]     'x-ms-access-tier-inferred': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.892839Z [info     ]     'Date': 'Thu, 18 May 2023 12:32:39 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.893863Z [info     ] INFO Request URL: 'https://devstorefmzyvn6gotqas.blob.core.windows.net/blobcontainer/in/emp.txt' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.894584Z [info     ] Request method: 'GET'          cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.895417Z [info     ] Request headers:               cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.898002Z [info     ]     'x-ms-range': 'REDACTED'   cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.899292Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.899755Z [info     ]     'Accept': 'application/xml' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.900309Z [info     ]     'User-Agent': 'azsdk-python-storage-blob/12.16.0 Python/3.10.6 (Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35)' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.901188Z [info     ]     'x-ms-date': 'REDACTED'    cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.903005Z [info     ]     'x-ms-client-request-id': '15cae50a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.903593Z [info     ]     'Authorization': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.904158Z [info     ] No body was attached to the request cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.917702Z [info     ] INFO Response status: 206      cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.918325Z [info     ] Response headers:              cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.925455Z [info     ]     'Content-Length': '86'     cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.926205Z [info     ]     'Content-Type': 'text/plain' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.926697Z [info     ]     'Content-Range': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.927164Z [info     ]     'Last-Modified': 'Fri, 12 May 2023 14:36:14 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.927599Z [info     ]     'Accept-Ranges': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.928037Z [info     ]     'ETag': '"0x8DB52F63D10EF6A"' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.928536Z [info     ]     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.929027Z [info     ]     'x-ms-request-id': '5d901692-301e-003a-2384-89fd91000000' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.929394Z [info     ]     'x-ms-client-request-id': '15cae50a-f578-11ed-a56a-00155d6836b4' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.929803Z [info     ]     'x-ms-version': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.930234Z [info     ]     'x-ms-creation-time': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.930665Z [info     ]     'x-ms-blob-content-md5': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.933171Z [info     ]     'x-ms-lease-status': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.934547Z [info     ]     'x-ms-lease-state': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.935142Z [info     ]     'x-ms-blob-type': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.935642Z [info     ]     'x-ms-server-encrypted': 'REDACTED' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.936053Z [info     ]     'Date': 'Thu, 18 May 2023 12:32:39 GMT' cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:42.936905Z [info     ] INFO Wrote 2 records for stream "test". cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2023-05-18T12:32:43.049936Z [info     ] time=2023-05-18 13:32:43 name=target_postgres level=INFO message=Loading 2 rows into 'raw__tap_spreadsheets_anywhere."test"' cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-05-18T12:32:43.080390Z [info     ] time=2023-05-18 13:32:43 name=target_postgres level=INFO message=Loading into raw__tap_spreadsheets_anywhere."test": {"inserts": 2, "updates": 0, "size_bytes": 126} cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-05-18T12:32:43.139285Z [info     ] Incremental state has been updated at 2023-05-18 12:32:43.139165.
2023-05-18T12:32:43.165229Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
acarter24 commented 1 year ago

After investigation this is due to the logging policy in azure.core, closing.

acarter24 commented 1 year ago

I was able to silence the info from Azure Blob with this added the top of file_utils.py

logger=logging.getLogger('azure.core.pipeline.policies.http_logging_policy')
logger.setLevel(logging.WARNING)

It makes for much cleaner logs now.