Avro - issue generating when using externally defined custom types

[chvndb]

Describe the bug

When trying to generate html the follow error occurs:

    Something went wrong:
    Error: undefined type name: eu.account.AccountMetaData
        at parse (/usr/local/lib/node_modules/@asyncapi/generator/node_modules/@asyncapi/parser/lib/parser.js:124:11)
        at async Generator.generateFromString (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:265:22)
        at async /usr/local/lib/node_modules/@asyncapi/generator/cli.js:154:9

How to Reproduce

file: ./src/eu.account.account-metadata.avsc

    {
      "type": "record",
      "namespace": "eu.account",
      "name": "AccountMetaData",
      "fields": [
        {
          "name": "someTraceId",
          "type": "string"
        },
        {
          "name": "someTimestamp",
          "type": {
            "type": "long",
            "logicalType": "timestamp-millis"
          }
        }
      ]
    }

file: ./src/eu.account.account-created.avsc

    {
      "type": "record",
      "namespace": "eu.account",
      "name": "AccountCreated",
      "fields": [
        {
          "name": "accountId",
          "type": "string"
        },
        {
          "name": "metadata",
          "type": "eu.account.AccountMetaData"
        }
      ]
    }

file: ./src/asyncapi.yaml

    asyncapi: "2.4.0"
    id: "urn:acme:crm:account-events"
    info:
      title: Account events
      version: "1.0.0"
    channels:
      eu.account:
        publish:
          message:
            oneOf:
              - schemaFormat: "application/vnd.apache.avro;version=1.9.0"
                payload:
                  $ref: "eu.account.account-created.avsc"

command: docker run --rm -it -v ${PWD}:/asyncapi asyncapi/generator /asyncapi/src/asyncapi.yaml @asyncapi/html-template -o /asyncapi/build

Expected behavior

The generator to correctly parse the custom Avro type eu.account.AccountMetaData and generate html with the specifications inline.

Notes

When using avro-tools for example it can parse the avro schemas without a problem and generate java code. Is this a bug inn asyncapi or is this not supported?

[github-actions[bot]]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[github-actions[bot]]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[derberg]

@M3lkior @dalelane hey folks, I moved it from generator repo because looks like the problem is with conversion of avro to json schema. Can you help please?

[M3lkior]

hello @chvndb ;

Using a non referenced avsc file is actually not supported by the parser (and i think also by the generator) because the parser actually parse only the avsc file used in the $ref property

A Workaround is to use a single a single avsc file with all your avro references like

[
  {
    "type": "record",
    "namespace": "eu.account",
    "name": "AccountMetaData",
    "fields": [
      {
        "name": "someTraceId",
        "type": "string"
      },
      {
        "name": "someTimestamp",
        "type": {
          "type": "long",
          "logicalType": "timestamp-millis"
        }
      }
    ]
  },
  {
    "type": "record",
    "namespace": "eu.account",
    "name": "AccountCreated",
    "fields": [
      {
        "name": "accountId",
        "type": "string"
      },
      {
        "name": "metadata",
        "type": "eu.account.AccountMetaData"
      }
    ]
  }
]

@derberg how to deal with some external references never referenced in a asyncapi.yml file ?

[derberg]

oh, wait, you mean that eu.account.AccountMetaData in ./src/eu.account.account-created.avsc means that it is a reference to ./src/eu.account.account-metadata.avsc ? 🤔

[M3lkior]

oh, wait, you mean that eu.account.AccountMetaData in ./src/eu.account.account-created.avsc means that it is a reference to ./src/eu.account.account-metadata.avsc ? 🤔

yes exactly

[derberg]

oh, I had no idea 😆 thanks!

so yeah, at the moment AsyncAPI specification supports referencing through $ref only. This means main parser also supports dereferencing of $ref.

correct me if I'm wrong but this is not stopping this avro schema parser dereferencing in avro way? before converting from Avro to JSON Schema, dereferencing could take place, right? just like validation

[M3lkior]

oh, I had no idea 😆 thanks!

so yeah, at the moment AsyncAPI specification supports referencing through $ref only. This means main parser also supports dereferencing of $ref.

correct me if I'm wrong but this is not stopping this avro schema parser dereferencing in avro way? before converting from Avro to JSON Schema, dereferencing could take place, right? just like validation

dereferencing ?

i just know that the avro file contents are pushed by the generator ; the avro-parser can not know what avro files are used in the spec

[chvndb]

Would it be possible to provide a reference file that lists "external" avro files which then be used when compiling?

[M3lkior]

Would it be possible to provide a reference file that lists "external" avro files which then be used when compiling?

i think this is not currently supported by the specification ;

You can not use the above workaround with all your external avro structures delcared in the same avsc file ?

[derberg]

dereferencing

nevermind, I meant resolving references, this eu.account.AccountMetaData but that is basically not possible with how things work now. I guess the only way is someone does it as a step before parsing with parser

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

[derberg]

I guess this one be closed, there is nothing that could be addressed in schema parser to fix it

[chvndb]

@M3lkior sorry for the very late response. We in fact "solved" it by automatically inlining the Avro references into one avsc file, but this is far from ideal. We would still prefer a way of keeping the references and somehow tell the parser the dependencies.

[hguidetti]

@chvndb as you can see, I have the same problem. Can you give an example of how you "automatically inline the Avro references in one asvc file" ? I'm not sure I understand how you workaround this issue. Thanks in advance.

[chvndb]

@hguidetti well we created a parser that scans all our avsc files. If an avsc file contains a reference to another schema, the parser will fetch the content of that schema and paste the content inline into the original avsc. The output avsc files do not contain any references anymore and can then be used as input the asyncapi. I cannot share any code as it is proprietary, but it is a fairly simple script.

It is annoying step that ideally would be done by the parser itself.

[hguidetti]

OK Thanks a lot. In fact it is a kind of preprocessor. It is a good idea (the less bad of the worths). We will implement something like that as well.

[chvndb]

@hguidetti oor you can use AVDL, which inlines references when transformed into AVSC.

[derberg]

you would need to discuss with https://github.com/asyncapi/parser-js maintainers how to do it so that parser and avro schema parser supports what you need

it is not trivial topic