Open cocozuloaga opened 3 months ago
I changed the input_datetime to timestamp format in the yaml file, but still getting the error. The log file is attached. Thanks so much!
It seems like this line has something to do with it, but can't figure out much:
: [CSV_ENFORCE_SCHEMA_NOT_SUPPORTED] The CSV option enforceSchema
cannot be set when using rescuedDataColumn
or failOnUnknownFields
, as columns are read by name rather than ordinal. SQLSTATE: 0A000
So... @cocozuloaga I made it work with some hacks, but I guess this is just a temporary solution.
I edited the_tuva_project.macros.load_seed.sql
, and there I made 'enforceSchema' = 'false',
(line 194). Then, I also changed all the datetime variables for other tables, e.g. lab_result, observation, to detect "databricks" and change it to timestamp.
Then, all the synthetic datasets are loaded to the databricks beautifully.
@yubinmimi glad that worked and you were able to load the seeds! We'll test in our Databricks environment as soon as we have it ready and push a fix to tuva_demo so that the hack you described is not necessary. Thanks for the feedback!
Describe the bug Eligibility seed file not loading in Databricks because DATETIME data type is not supported.
To Reproduce Run Tuva Demo on Databricks.