search

confluentinc / kafka-connect-datagen

Connector that generates data for demos
Apache License 2.0
20 stars 87 forks source link

Connector silently fails to generate messages when schema is too long #98

Closed olivd44 closed 1 year ago

olivd44 commented 2 years ago

Hi,

When setting a large json schema either in schema.string or in schema.filename file, connector does not send any message.

No errors are raised, just messages are not generated.

Is there a known size limit in schemas that can be handled by datagen connector?

It used to work with 0.3.3 and does not work anymore from 0.4.x onward

Kind regards, Olivier

vdesabou commented 2 years ago

I can reproduce this issue, but it's not related to schema size, it is happening when "regex": "[a-zA-Z0-9]{x}": where x>15:

        {
            "name": "MY_STRING",
            "type": {
                "type": "string",
                "arg.properties": {
                    "regex": "[a-zA-Z0-9]{16}"
                }
            },
            "default": ""
        }
guitcastro commented 2 years ago

@vdesabou any workaround for this?

vdesabou commented 2 years ago

@guitcastro This was fixed by https://github.com/confluentinc/kafka-connect-datagen/pull/99 and available in 0.5.3 release.

This issue can be closed