search

dataform-co / dataform-web-tracking

Dataform is a collaborative data modelling platform that enables analysts and engineers to manage complex data models in SQL
https://dataform.co
MIT License
2 stars 0 forks source link

[Bug] Web API RunCreateRequest doesn't properly use Environment configOverride settings #330

Open andrewscarani opened 2 years ago

andrewscarani commented 2 years ago

When the environment name has been specified in the RunCreateRequest command on the Dataform Web API, it fails to pull-in the configOverride settings (i.e. schemaSuffix) specified in the environment config. This is problematic as it does not allow one to use the Web API to trigger jobs in environments other than production with the correct configuration and defeats the entire purpose of specifying an environment name to begin with.

See docs for expected behavior: https://docs.dataform.co/dataform-web/api/reference#/definitions/v1RunCreateRequest:~:text=The%20name%20of%20the%20environment%20in%20the%20%60environments.json%60%20file%20to%20pull%20project%20configuration%20options%20from%20(optional).

I noticed that this is a bump on this originally reported issue that was closed by @BenBirt: https://github.com/dataform-co/dataform-web-tracking/issues/305

Using schemaSuffix is a very common way of separating environments in DBT and other tools, so it would be great if Dataform could support this properly. Otherwise, it's just one more drop in the bucket in-favor of switching to DBT.

BenBirt commented 2 years ago

The legacy Dataform webapp is in maintenance mode only, we encourage users to use the new GCP version wherever possible.

andrewscarani commented 2 years ago

@BenBirt I would love to do that, but the new GCP version is currently in Preview, is not recommended for production use, and has a long running list of legacy features not supported (without any documentation as to if/when they will be supported again):

https://cloud.google.com/dataform/docs/migration#:~:text=We%20do%20not%20recommend%20migrating%20your%20daily%20development%20of%20Dataform%20production%20projects%20to%20Google%20Cloud%20at%20present%20as%20not%20all%20features%20of%20legacy%20Dataform%20are%20available%20in%20Google%20Cloud%20at%20this%20time.

https://cloud.google.com/dataform/docs/migration#legacy_dataform_features_not_supported_in_at_this_time