snowplow / snowplow-rdb-loader

Stores Snowplow enriched events in Redshift, Snowflake and Databricks
Other
31 stars 17 forks source link

Kinesis transformer: fails to process event with multiple schema versions #1335

Open dkucharc opened 10 months ago

dkucharc commented 10 months ago

The transformer client is not able to decode server response containing multiple schema versions.

{
  "schema": "iglu:com.snowplowanalytics.snowplow.badrows/loader_iglu_error/jsonschema/2-0-0",
  "data": {
    "processor": {
      "artifact": "snowplow-transformer-kinesis",
      "version": "5.4.0"
    },
    "failure": [
      {
        "schemaCriterion": "iglu:com.custom/user_context/jsonschema/1-*-*",
        "error": {
          "error": "ResolutionError",
          "lookupHistory": [
            {
              "repository": "Iglu Client Embedded",
              "errors": [
                {
                  "error": "NotFound"
                }
              ],
              "attempts": 1,
              "lastAttempt": "2024-01-05T14:53:40.205Z"
            },
            {
              "repository": "Custom",
              "errors": [
                {
                  "error": "ClientFailure",
                  "message": "Could not decode server response. org.http4s.InvalidMessageBodyFailure: Invalid message body: Could not decode JSON: [\n  \"iglu:com.custom/user_context/jsonschema/1-1-0\",\n  \"iglu:com.custom/user_context/jsonschema/1-0-0\"\n]"
                }
              ],
              "attempts": 8,
              "lastAttempt": "2024-01-05T14:55:44.693Z"
            }
          ]
        }
      }
    ],
    "payload": {...}

Deployed stack: