Add Databricks profile - Githubissues

OHDSI / WebAPI

OHDSI WebAPI contains all OHDSI services that can be called from OHDSI applications

Apache License 2.0

126 stars 156 forks source link

Add Databricks profile #2339

Closed anthonysena closed 4 months ago

anthonysena commented 5 months ago

Per feedback from @konstjar, I've added a new profile "webapi-databricks" to use the Databricks JDBC Driver reference. Some notes on this PR:

Using the latest Databricks driver (v2.6.36) was problematic in Apache Tomcat 8.5.x. When using this driver, the servlet container would crash and WebAPI failed to start. v2.6.34 did not exhibit this behavior hence that is why it is referenced in the pom.xml
Added com.databricks.client.jdbc.Driver to the list of supported drivers to support this newer JDBC driver.

anthonysena commented 5 months ago

From discussion with @konstjar on the Atlas WG call, we decided to create a "webapi-databricks" profile and retain the "webapi-spark" profile for backwards compatibility. I'll revise this PR accordingly.

konstjar commented 5 months ago

@anthonysena

My filling is that it is not enough to add the profile.

There are few places in WebAPI where we have some custom logic for Spark dialect: https://github.com/OHDSI/WebAPI/blob/master/src/main/java/org/ohdsi/webapi/cohortcharacterization/DropCohortTableListener.java#L52 https://github.com/OHDSI/WebAPI/blob/3f9e90c5a62dac4557f34f85b2d0ce70d58b439d/src/main/java/org/ohdsi/webapi/util/CancelableJdbcTemplate.java#L81 https://github.com/OHDSI/WebAPI/blob/3f9e90c5a62dac4557f34f85b2d0ce70d58b439d/src/main/java/org/ohdsi/webapi/cohortcharacterization/GenerateCohortCharacterizationTasklet.java#L106 https://github.com/OHDSI/WebAPI/blob/3f9e90c5a62dac4557f34f85b2d0ce70d58b439d/src/main/java/org/ohdsi/webapi/util/PreparedStatementRenderer.java#L291 https://github.com/OHDSI/WebAPI/blob/3f9e90c5a62dac4557f34f85b2d0ce70d58b439d/src/main/java/org/ohdsi/webapi/pathway/PathwayStatisticsTasklet.java#L173

I think that we should add "jdbc:databricks" handling there. In the past we had few issues with Characterization and Pathway analysis executions.

anthonysena commented 5 months ago

Thanks @konstjar - you've raised some important points here with respect to the code base and the "spark" profile.

Recalling our conversation on the Atlas WG call, I thought that we wanted to adopt the "databricks" profile since some users may be using the "spark" driver which may target an older build of Apache Spark for Databricks while others may need the newer driver? When setting up the new or old Spark data source in Atlas/WebAPI, the source would be configured with the dbms == "spark" but the connection string might differ based on the driver. So in this way, the only check that requires extension is the one that is looking at the JDBC connection string, correct? In that case we can have a static method to detect the dialect based on either "databricks" or "spark" in the JDBC connection string and that would work?

anthonysena commented 4 months ago

@konstjar - I've updated this PR with the idea that when a person configures a data source to use spark/databricks, they would create the entry in the source table with the dialect == spark and for the connection string they could either provide the jdbc:spark or jdbc:databricks connection details. So, the checks in the code that you pointed out earlier in this PR remain the same with the exception of the one that checks the connection string details. I've modified this to check both jdbc URL formats and that should hopefully handle backwards compatibility. If you agree, this should be ready to merge.

konstjar commented 4 months ago

@anthonysena Looks good to me.

anthonysena commented 4 months ago

Thanks @konstjar - can you approve the PR then?