opendatadiscovery / odd-collectors

Apache License 2.0
8 stars 9 forks source link

Fix Snowflake adapter warnings + add more debug logs #107

Closed ValeriyWorld closed 6 days ago

ValeriyWorld commented 1 week ago

Warning fixes: 1) Dependency warning related to snowflake-connector-python package. 2) Pydantic warning related to Pipe class - it had attribute schema, the same name was used in parent class BaseModel that had method with the same name (it is deprecated though, but still not deleted).

Debug logging enhancement: 1) Added logger.debug() messages in the mapping functions, which are useful to understand exactly where the collector crashed. image Error case: image 2) Due to avoid logging duplication (current mapping logic have some nesting logic) - changed properties to the methods decorated with @ttl_cache.

Tests updates: 1) Updated tests with new adapter and client realisation + added a few pipes to the test. 2) Added Snowflake tests to our run-tests workflow.