Each connection in atlas is configured as a connection string, alongside fields for schema, results writeable, vocabulary.
Some data lake providers allow connections to be conducted with different computation warehouses.
In effect, Data storage and Data compute are separated.
In ATLAS, this compute flexibility should be available via a drop down (like the switch for data source, pictured below) allowing switch of execution warehouse appropriate for the scale of job to generated
Taking Snowflake as an example, small, medium and large warehouses are specified by a variable in connection string url:
snowflakeacc.com/?warehouse=snowflake_warehouse&db=somedata.....
Flexibility in connection string does not exist, however does appear to be possible given the separated out username and password fields.
Configuration page should have a field "warehouse" or "wildcard" or "Extra connection variables" which if populated with a comma separated or quoted series of strings, will introduce the user selected item from the list (on generation pages for Cohorts et al) near the Generate buttons that allows the user to change this variable.
A default can still be set for testing connections, SOP as first item in the wildcard list. Reccomendation for this would be alongside the source daemons box.
Applications outside of my current snowflake example: Other dbms that allows use of connection string variables to alter sessions, would be able to implement such switches in the same field.
Actual behavior
At current, this variable must be included in the connection string - resulting in duplication on the order of
connections*Warehouse
number of connections to have each data set available with different warehouses.
Expected behavior
Each connection in atlas is configured as a connection string, alongside fields for schema, results writeable, vocabulary. Some data lake providers allow connections to be conducted with different computation warehouses. In effect, Data storage and Data compute are separated.
In ATLAS, this compute flexibility should be available via a drop down (like the switch for data source, pictured below) allowing switch of execution warehouse appropriate for the scale of job to generated
Taking Snowflake as an example, small, medium and large warehouses are specified by a variable in connection string url:
snowflakeacc.com/?warehouse=snowflake_warehouse&db=somedata.....Flexibility in connection string does not exist, however does appear to be possible given the separated out username and password fields.Configuration page should have a field "warehouse" or "wildcard" or "Extra connection variables" which if populated with a comma separated or quoted series of strings, will introduce the user selected item from the list (on generation pages for Cohorts et al) near the Generate buttons that allows the user to change this variable.
A default can still be set for testing connections, SOP as first item in the wildcard list. Reccomendation for this would be alongside the source daemons box.
Applications outside of my current snowflake example: Other dbms that allows use of connection string variables to alter sessions, would be able to implement such switches in the same field.
Actual behavior
At current, this variable must be included in the connection string - resulting in duplication on the order of
connections*Warehousenumber of connections to have each data set available with different warehouses.Steps to reproduce behavior
Not a bug, but an enhancement request