Request to have CDMConnector optimised for Spark

darwin-eu / CDMConnector

A pipe friendly way to interact with an OMOP Common Data Model

https://darwin-eu.github.io/CDMConnector/

Apache License 2.0

12 stars 9 forks source link

Request to have CDMConnector optimised for Spark #11

Open KAMDeS opened 7 months ago

KAMDeS commented 7 months ago

Currently CDMConnector doesn't seem to work for Spark - setting up a Connector object takes hours rather than minutes and most of the functionality seems not work. Would like to request it to work for Spark.

ablack3 commented 6 months ago

ok noted. I will work on it.

villesiltala commented 5 months ago

Any updates on this? Will this bring full support for Spark SQL? Based on the documentation, at least some of the connections are established using an ODBC connection. Will this Spark support enable using either the JDBC or ODBC driver for connecting to a Databricks based Spark cluster? As you may know, DatabaseConnector and SqlRender already support Spark SQL.

laacdm commented 3 months ago

I ran CDMConnector after making a connection to Databricks/Spark using the DatabaseConnector R-Package together with a JDBC driver.

The connection was made after a long wait and got the following warning message:

"Not all functionality is supported when DatabaseConnector as your database driver! Some issues may occur".