search

dataform-co / dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://cloud.google.com/dataform/docs
Apache License 2.0
851 stars 163 forks source link

Improve error messages when workflow settings have non-string values for default project or dataset #1846

Open kolina opened 2 months ago

kolina commented 2 months ago

Currently if defautProject or defaultDataset have an integer type in workflow_settings.yaml, a compilation will throw a not very obvious error like t.target.schema.includes is not a function. I've added a validation of these fields to be strings (if not, an explicit error is thrown).

kolina commented 2 months ago

I also hoped that it should have been the case, but unfortunately this code https://github.com/dataform-co/dataform/blob/05341b7c2e8f6e55a7002ed29b4ff4a7e2cbf12b/common/protos/index.ts#L45-L46 doesn't validate types and actually allows to create protoCastObject where defaultDataset has an integer type and this check later doesn't work https://github.com/dataform-co/dataform/blob/05341b7c2e8f6e55a7002ed29b4ff4a7e2cbf12b/common/protos/index.ts#L53

The thing that actually verifies types is protoType.verify(). WDYT about using it?

Ekrekr commented 1 week ago

.verify() is a lie, it doesn't actually do anything - there are no situations where the call will actually fail.