We were not lower-casing the result of asking DuckDB for its schema.
So we were searching inside its schema results for the exact camel
case string in duckdb but Athena would need a lower case string.
Impossible to work in both. So double-word depth-two-fields would
never be matched.
For depth-two fields, we were also assuming that we would be given
fields in the same order that the SQL table would return them.
So if you didn't provide them in that precise order (like if you used
start, end but the SQL table had end, start -- you'd miss one).
This was the exact case for Encounter fields in duckdb.
This commit allows the following new features/conveniences:
Case-insensitive schema lookups (i.e. table name and column name
arguments to get_column_datatype_query() can be any case)
Case-insensitive schema validation (i.e. the expected column names
passed to validate_table_schema() can be any case)
The resulting validated schema will use the original case used by
the expected-fields dictionary. So that the user will not be
surprised and we'll use camel case if they do, lower case if they do.
Checklist
[x] Consider if documentation (like in docs/) needs to be updated
[x] Consider if tests should be added
[x] Update template repo if there are changes to study configuration
This fixes a couple bugs:
This commit allows the following new features/conveniences:
Checklist
docs/) needs to be updated