BritishGeologicalSurvey / etlhelper

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.
https://britishgeologicalsurvey.github.io/etlhelper/
GNU Lesser General Public License v3.0
104 stars 25 forks source link

Make information on column types accessible #128

Closed volcan01010 closed 2 years ago

volcan01010 commented 2 years ago

Description

This merge request adds a describe_columns function that can return column metadata.

Closes #127

To test

ximenesuk commented 2 years ago

I have implemented this function for SQL Server. Due to the parameter handling there has been a little refactoring. The tests are passing locally. (I can't see any README updates in this PR.)

volcan01010 commented 2 years ago

Maybe SQLite does have a concept of schemas. Or at least of two-part table names.

https://stackoverflow.com/questions/49660302/does-sqlite-have-the-concept-of-a-schema-in-naming-tables-views

Judging by this, we should set the schema to main if one isn't specified.

ximenesuk commented 2 years ago

Maybe SQLite does have a concept of schemas. Or at least of two-part table names.

Yes, I was going to raise this as I saw "schemas" being used in an example. We need to work out how to use them dynamically as the format for schemas with pragmas is:

schema.pragma_name(...)
volcan01010 commented 2 years ago

A tweak for the Oracle SQL query - we need to wrap the schema name in ^, $, otherwise the REGEX is too greedy and matches BGS_BGSPROD when I searched for just BGS.

SELECT * FROM all_tab_columns
    WHERE LOWER(table_name) = LOWER('DIC_SEN_SENSOR')
    AND REGEXP_LIKE(LOWER(owner), COALESCE(LOWER('^' || 'BGS' || '$'), '.*'))
ximenesuk commented 2 years ago

Over to you @volcan01010

volcan01010 commented 2 years ago

This is nearly ready to merge, but I can't get a connection to MS SQL Server to work for the integration tests. Once we have that fixed and confirm that tests pass there, we can release.

ximenesuk commented 2 years ago

I've rerun the whole test suite including MS SQL Server and all tests pass.