Added params to savePaths for spark dialect.

OHDSI / WebAPI

OHDSI WebAPI contains all OHDSI services that can be called from OHDSI applications

Apache License 2.0

126 stars 156 forks source link

Added params to savePaths for spark dialect. #2276

Closed chrisknoll closed 1 year ago

chrisknoll commented 1 year ago

Fixes #2275.

chrisknoll commented 1 year ago

@tomwhite , @TomWhite-MedStar , please let me know if this PR addresses your issue. I believe the issue was that the SqlRender params weren't being set for the schema and that led to the issue.

TomWhite-MedStar commented 1 year ago

@chrisknoll , Thanks. Yes, I tested this patch and it solved the issue for Pathway analysis.

Oddly, I'm still seeing issues with SqlRender (and show columns) for other modules (esp. for Feature Extraction in Characterization). However, the patch does solve the issue with Pathway analysis.

chrisknoll commented 1 year ago

Ok, thanks, please let us know where those other issues are, they probably follow a similar pattern of:

if (sparkDb) then { handleSparkInsert (sql) }
else {  prepareRenderTanslate(sql, params, values) }

We need to find those places that are doing the special spark handling and ensure their sql statements have their parameters filled.

TomWhite-MedStar commented 1 year ago

@chrisknoll , Here are the lingering show columns statements that are failing on Databricks. It looks as though some may be from Feature Extraction module, but I'm not sure:

show columns in #analysis_ref
show columns in #cov_ref
show columns in #hfrs_scoring
show columns in #chads2_scoring
show columns in #chads2_concepts
show columns in #charlson_concepts
show columns in #charlson_scoring
show columns in #chads2vasc_scoring
show columns in #chads2vasc_concepts
show columns in #dcsi_scoring

chrisknoll commented 1 year ago

Yes, those are all FeatureExtraction analyses (I'm 99 certain), but it could be that something in SqlRender is intercepting inserts for spark.....basically anywhere you see an inset statement, it will attempt to do a show columns on it because of a limitation in Spark where you can'tdo an insert into... with column names. So, it will require a little digging to uncover exactly where it is being called from.