dipeshkd
commented
1 year ago I ran the mssql discover docker with below
docker run --init -i -w /data/932356e1-1114-49ba-997d-0d105b08c306/0 --log-driver json-file --name source-mssql-discover-test --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-mssql:2.0.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.50.28 -e WORKER_JOB_ID=932356e1-1114-49ba-997d-0d105b08c306 airbyte/source-mssql:2.0.1 discover --config source_config.jsonhere is the output.
It seems to be taking a long time to get the table columns. It took 6 mins to get the columns. Not sure why its taking so long.
2023-09-09 12:22:45 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.m.MssqlSource(main):569 starting source: class io.airbyte.integrations.source.mssql.MssqlSource"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {discover=null, config=source_config.json}"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):130 Running integration: io.airbyte.integrations.base.ssh.SshWrappedSource"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):131 Command: DISCOVER"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):132 Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"WARN","message":"WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"WARN","message":"WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"WARN","message":"WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL"}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting..."}}
2023-09-09 12:22:46 {"type":"LOG","log":{"level":"INFO","message":"INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed."}}
2023-09-09 12:22:47 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.j.AbstractJdbcSource(discoverInternal):170 Internal schemas to exclude: [spt_fallback_db, spt_monitor, cdc, spt_values, INFORMATION_SCHEMA, spt_fallback_usg, MSreplication_options, sys, spt_fallback_dev]"}}
2023-09-09 12:22:50 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.j.AbstractJdbcSource(discoverPrimaryKeys):277 Discover primary keys for tables: <LONG LIST OF TABLES>}}
2023-09-09 12:28:34 {"type":"LOG","log":{"level":"INFO","message":"INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated..."}}
2023-09-09 12:28:34 {"type":"LOG","log":{"level":"INFO","message":"INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed."}}
2023-09-09 12:28:34 <LONG LIST OF COLUMNS>
2023-09-09 12:28:34 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.b.IntegrationRunner(runInternal):226 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource"}}
2023-09-09 12:28:34 {"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.m.MssqlSource(main):571 completed source: class io.airbyte.integrations.source.mssql.MssqlSource"}}
tjsoliveira
evantahler
Connector Name
source-mssql
Connector Version
2.0.1
What step the error happened?
Configuring a new connector
Revelant information
Hi,
I am on Airbyte version 0.50.28 (Also tested this on 0.50.24) I am no longer able to create a connection with MSSQL as a source. I have tested this with S3 and local Postgres destination. This was working earlier in the week.
The connection takes a long time when trying to connect to my production MSSQL
I get the following error
looking at the logs from worker it seems to stop at discoverPrimaryKeys I have removed the list of tables with ""
Relevant log output
Contribute