search

authorjapps / zerocode

A community-developed, free, opensource, automated testing framework for microservices APIs, Kafka(Data Streams) and Load testing. Zerocode Open Source enables you to create, change and maintain your automated test scenarios via simple JSON or YAML files. Visit documentation below:
https://zerocode-tdd.tddfy.com
Apache License 2.0
909 stars 397 forks source link

Crash on Kafka consume when key is string with `recordType: JSON` #657

Closed Tobolov closed 7 months ago

Tobolov commented 7 months ago

When consuming a Kafka records with recordType: JSON, records with a plain string as a the key can't be deserialized.

Scenario to reproduce:

---
scenarioName: Demo
steps:
  - name: do_produce
    url: "kafka-topic:THE_TOPIC"
    operation: produce
    request:
      recordType: JSON
      records:
        - key: ${RANDOM.UUID}
          value:
            appName: App X
    verify:
      status: Ok
  - name: do_consume
    url: "kafka-topic:THE_TOPIC"
    operation: consume
    request:
      consumerLocalConfigs:
        recordType: JSON
        commitSync: true
        showRecordsConsumed: true
        maxNoOfRetryPollsOrTimeouts: 3
    verify:
      size: 1
      records:
        - key: ${$.do_produce.request.records[0].key}
          value:
            appName: App X

Exception:

ava.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('d' (code 100)): Expected space separating root-level values
 at [Source: (String)"1dd2e6c2-9d40-4c75-b735-c83911b6b652"; line: 1, column: 3]
    at org.jsmart.zerocode.core.kafka.client.BasicKafkaClient.execute(BasicKafkaClient.java:51)
    at org.jsmart.zerocode.core.engine.executor.ApiServiceExecutorImpl.executeKafkaService(ApiServiceExecutorImpl.java:59)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeApi(ZeroCodeMultiStepsScenarioRunnerImpl.java:485)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeRetry(ZeroCodeMultiStepsScenarioRunnerImpl.java:252)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeRetryWithSteps(ZeroCodeMultiStepsScenarioRunnerImpl.java:191)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeSteps(ZeroCodeMultiStepsScenarioRunnerImpl.java:173)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.runScenario(ZeroCodeMultiStepsScenarioRunnerImpl.java:136)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runLeafJsonTest(ZeroCodeUnitRunner.java:223)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runChild(ZeroCodeUnitRunner.java:127)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runChild(ZeroCodeUnitRunner.java:51)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.run(ZeroCodeUnitRunner.java:107)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('d' (code 100)): Expected space separating root-level values
 at [Source: (String)"1dd2e6c2-9d40-4c75-b735-c83911b6b652"; line: 1, column: 3]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2477)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:750)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:674)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportMissingRootWS(ParserMinimalBase.java:722)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._verifyRootSpace(ReaderBasedJsonParser.java:1830)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseUnsignedNumber(ReaderBasedJsonParser.java:1425)
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:805)
    at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4854)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3219)
    at org.jsmart.zerocode.core.kafka.helper.KafkaConsumerHelper.readJson(KafkaConsumerHelper.java:307)
    at org.jsmart.zerocode.core.kafka.receive.KafkaReceiver.appendNewRecords(KafkaReceiver.java:124)
    at org.jsmart.zerocode.core.kafka.receive.KafkaReceiver.receive(KafkaReceiver.java:96)
    at org.jsmart.zerocode.core.kafka.client.BasicKafkaClient.execute(BasicKafkaClient.java:38)
    ... 23 more

java.lang.RuntimeException: ZeroCode Step execution failed. Details:java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('d' (code 100)): Expected space separating root-level values
 at [Source: (String)"1dd2e6c2-9d40-4c75-b735-c83911b6b652"; line: 1, column: 3]

    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeRetry(ZeroCodeMultiStepsScenarioRunnerImpl.java:379)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeRetryWithSteps(ZeroCodeMultiStepsScenarioRunnerImpl.java:191)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.executeSteps(ZeroCodeMultiStepsScenarioRunnerImpl.java:173)
    at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.runScenario(ZeroCodeMultiStepsScenarioRunnerImpl.java:136)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runLeafJsonTest(ZeroCodeUnitRunner.java:223)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runChild(ZeroCodeUnitRunner.java:127)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runChild(ZeroCodeUnitRunner.java:51)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.run(ZeroCodeUnitRunner.java:107)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
    at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

Root cause: On KafkaConsumerHelper.java#L307, we run JsonNode keyNode = objectMapper.readTree(keyStr);. This expects the key to be a supported JSON element. The value1dd2e6c2-9d40-4c75-b735-c83911b6b652 is not a supported JSON element, however "1dd2e6c2-9d40-4c75-b735-c83911b6b652" would be.

JSON equivalent:

{
  "scenarioName": "Demo",
  "steps": [
    {
      "name": "do_produce",
      "url": "kafka-topic:THE_TOPIC",
      "operation": "produce",
      "request": {
        "recordType": "JSON",
        "records": [
          {
            "key": "${RANDOM.UUID}",
            "value": {
              "appName": "App X"
            }
          }
        ]
      },
      "verify": {
        "status": "Ok"
      }
    },
    {
      "name": "do_consume",
      "url": "kafka-topic:THE_TOPIC",
      "operation": "consume",
      "request": {
        "consumerLocalConfigs": {
          "recordType": "JSON",
          "commitSync": true,
          "showRecordsConsumed": true,
          "maxNoOfRetryPollsOrTimeouts": 3
        }
      },
      "verify": {
        "size": 1,
        "records": [
          {
            "key": "${$.do_produce.request.records[0].key}",
            "value": {
              "appName": "App X"
            }
          }
        ]
      }
    }
  ]
}
authorjapps commented 7 months ago

@Tobolov , understood the issue.

  1. You have a String key(with alpha-numeric) and JSON value already produced into the Kafka topic. Then, consuming failed due to deser failed at objectMapper.readTree(keyStr) as you have spotted.

Just checking about your test usecase, did you produce the messages that way(mentioned in No. 1) or another application has produced that way, then you are only consuming?

Tobolov commented 7 months ago

Thanks for the swift reply @authorjapps! Yep, issue correctly identified.

I first identified the issue when I had another application produce a record with an alpha-number key and a JSON value in response to a HTTP request.

That said, I will have also have tests quite similar in nature to the one included in the bug report. The only notable difference being the key will be some-prefix-${RANDOM.UUID}.

nirmalchandra commented 7 months ago

Thanks for the swift reply @authorjapps! Yep, issue correctly identified.

I first identified the issue when I had another application produce a record with an alpha-number key and a JSON value in response to a HTTP request.

That said, I will have also have tests quite similar in nature to the one included in the bug report. The only notable difference being the key will be some-prefix-${RANDOM.UUID}.

@Tobolov , we will be sorting this out soon in the next release.

Until the fix is in place, the below work-around could be used(temporarily fix only if feasible) : While producing(either your test scenario or another application), produce like this: e.g.

"key": "\"ALPHA-001\"",

instead of "key": "ALPHA-001",

I am aware that this may not be very convinient if the message is produced by another application(internal or external to your system), but may be an alternative(for temporarily until the PR is merged) if your one of your test step is doing the PRODUCE.

A working example is here for reference, for the the this test scenario.

authorjapps commented 7 months ago

TODO: Documentation: Update here.

Available in version: 1.3.43.

Release Details: https://github.com/authorjapps/zerocode/releases/tag/1.3.43