search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

Source MSSQL: export meta data from INFORMATION_SCHEMA #14481

Open sivankumar86 opened 2 years ago

sivankumar86 commented 2 years ago

Tell us about the problem you're trying to solve

I want to export meta data from mssql to snowflake to normalize the data using dbt jobs.

Describe the solution you’d like

We have CDC jobs which exporting json row incrementally. We have job which calculate scd 2 and need to infer the json row datatype from source meta

Describe the alternative you’ve considered or used

need to write a airflow job which export the meta every day to snowflake ( i do not want to do) and want to keep everything in airbyte

Additional context

Add any other context or screenshots about the feature request here.

sivankumar86 commented 2 years ago

I think, Need to change this line. Could someone confirm if it is ok to exclude ? so that, I can give pull request.

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-mssql/src/main/java/io/airbyte/integrations/source/mssql/MssqlSource.java#L210

misteryeo commented 2 years ago

Tagging @grishick to take a look!

grishick commented 2 years ago

feel free to submit a pull request. From the first look it seems like a safe change

sivankumar86 commented 2 years ago

@grishick Thank you for taking a look. I have submitted a pull request.

bleonard commented 2 years ago

Note: This PR was closed without merging https://github.com/airbytehq/airbyte/pull/15341