Before this change, the ingest logic looked roughly like this:
Initial transformation of JSON ("pre ingest transformer", field encodings, ...)
Generating CH table schema based on JSON + config (for "CREATE TABLE" or "ALTER TABLE"s)
Transformation of JSON (transformer object: flattening map and removing ignored fields)
Executing SQL query with the JSON data (after both initial and second transformation)
The issue with this logic is that the transformation (step 3) removes ignored fields, but the step 2 operated on a JSON with those fields present.
Rather than introducing additional logic to step 2 to handle ignored fields, this PR takes another approach: it moves the step 3 before the step 2. This way the CREATE TABLE/ALTER TABLE statements don't take into account the ignored fields.
Moving the transformation earlier allowed for simplifying JsonToColumns. Added two new integration tests.
Before this change, the ingest logic looked roughly like this:
The issue with this logic is that the transformation (step 3) removes ignored fields, but the step 2 operated on a JSON with those fields present.
Rather than introducing additional logic to step 2 to handle ignored fields, this PR takes another approach: it moves the step 3 before the step 2. This way the CREATE TABLE/ALTER TABLE statements don't take into account the ignored fields.
Moving the transformation earlier allowed for simplifying
JsonToColumns. Added two new integration tests.