Starting as DRAFT for now. Needs integration tests and some additional robustness.
This PR adds a coercions module for implicitly coercing values along the lines of what Spark's from_json function provides. Hooks are provided with this PR, but we are only implementing coercions that are relevant to our own deployments at the moment.
The two that are biting us right now are:
The implicit coercion of nested JSON properties to the delta string data type
The implicit coercion of ISO 8601 timestamp strings to the delta timestamp data type
I implemented this separately from the transforms module because the intention is quite different and that module could use some cleanup. Ultimately, I could see us potentially getting rid of the dependency on the jmespath crate, extracting the CoercionTree into a more general data structure that can be used for a single pass walk over a deserialized serde_json::Value to do transforms and coercions together but that's for another time.
Starting as DRAFT for now. Needs integration tests and some additional robustness.
This PR adds a
coercionsmodule for implicitly coercing values along the lines of what Spark'sfrom_jsonfunction provides. Hooks are provided with this PR, but we are only implementing coercions that are relevant to our own deployments at the moment.The two that are biting us right now are:
stringdata typetimestampdata typeI implemented this separately from the
transformsmodule because the intention is quite different and that module could use some cleanup. Ultimately, I could see us potentially getting rid of the dependency on the jmespath crate, extracting theCoercionTreeinto a more general data structure that can be used for a single pass walk over a deserializedserde_json::Valueto do transforms and coercions together but that's for another time.