ValeriyWorld
commented
1 month ago @CathyBreslow Hello! I have run mssql and odbc adapters on some test database with a few tables + views. Seems like both of adapters are working. 1) mssql: Successful collector logs:
ODD Platform result:
At first I would suggest you to check if the version of collector is up to date. It can be easily checked when you run a collector, it is the first row of the logs. The most up to date version is 0.1.63, which can be pulled with ghcr.io/opendatadiscovery/odd-collector:latest (make sure that Docker didn't cache the old image, in this case delete the old one and pull once more). The next thing you may have issue with is the following: there is already a datasource created with the name you specified. For example you have already some datasource name: "<datasource_name>", in this way platform is trying to create a new adapter for new configuration, but can not, because the one with this name already exists. So try to change name attribute of the plugin.
2) odbc: This adapters worked fine as well in my quick test, but it collects a lot of system views that can not be needed, so can make a mess in the platform. If it will be a problem we can discuss this later. Also successful collector logs:
ODD Platform result:
I did some investigation and found out, that there are no proper error handling in odbc adapter, so if you did some mistake in the configuration, for example in the host - it can lead to this not good explaining "Could not map table" error for now. Information about drivers you may find in the ODBC Data Source Administrator -> Drivers.
So you need the exactly the name for driver, and may put it in the {}, or left w/o them, both seems to work fine. As I can see the problem could be in the driver name, you version - "ODBC Driver 17s for SQL Server", seems like s in 17s is redundant. My odbc plugin configuration looked like this than I ran a collector:
- type: "odbc"
name: "odbc_adapter"
description: "" # Optional string
database: "<db_name>"
user: "<user_name>"
password: "<db_password>"
host: "<host>"
port: "1433"
driver: "{ODBC Driver 17 for SQL Server}"I guess you should get the result after driver name fix. But also I may assume that the same "Could not map table" error can appear in case of lack of user's permissions, because adapter is getting metadata from system tables and views too.
For quick summarisation: you better try mssql adapter once more, as I write, try to change the name property in collector_config.yaml - it is used for datasource name generation and must be unique for different configurations. And for odbc - try to fix driver name.
CathyBreslow
Hello! I successfully connected ODD to our production Postgres data warehouse, and I need to connect to our SQL server databases for lineage. I'm having issues connecting to one of our SQL servers using both adapters.
I'm getting errors trying to create a MSSQL collector using either the MSSQL adapter or the ODBC adapter. I can connect to this database using DBeaver software using the exact same credentials so I know it's not necessarily a network problem.
Here's the error using the MSSQL Adapter: 2024-07-18 14:49:38.474 | ERROR | odd_collector_sdk.collector:run:137 - Could not create data sources. Reason: 500, message='Internal Server Error', url=URL('http://odd-platform:8080/ingestion/datasources') This is the exact text in my collector_config.yaml for this collector: (sensitive data replaced)
default_pulling_interval: 60 token: "xxxx" plugins:
type: mssql name: "XYZ" description: "MyServerName" # Optional string database: "MyDatabase" host: "MyServerName" port: 1433 user: "UserName" password: "PW"
Here's the error when I try to use the ODBC connector: 2024-07-18 15:32:26.212 | INFO | odd_collector_sdk.job:log_execution:21 - [Syteline - ODBC] collecting metadata started. 2024-07-18 15:32:26.214 | ERROR | odd_collector_sdk.job:log_execution:25 - [Syteline - ODBC] failed. Could not map table
And here's the ODBC adapter info: default_pulling_interval: 10 token: "xxxx" plugins:
I am more than happy to try any tests for fixes you can give me!
Thank you, Cathy Breslow