Support AsyncAPI schema

[n0nvme]

Is your feature request related to a problem? Please describe. from asyncapi docs:

AsyncAPI is an open source initiative that seeks to improve the current state of Event-Driven Architectures

It's like OpenAPI but for components that use message brokers instead of HTTP protocol. After some research, I've found that there are no convenient tools to generate pydantic models from AsyncAPI spec.

Describe the solution you'd like It's gonna be great to have the ability to generate pydantic models from AsyncAPI schemas.

Describe alternatives you've considered AsyncAPI has its own generator, but templates are written in javascript and I'm not sure it can support all pydantic features.

Additional context AsyncAPI started as an adaptation of the OpenAPI specification and is compatible with OpenAPI schema object syntax. So I think it is possible to reuse some code from the OpenAPI parser.

Links: https://www.asyncapi.com/docs/getting-started/coming-from-openapi https://github.com/asyncapi/generator

Funding

• You can sponsor this specific effort via a Polar.sh pledge below
• We receive the pledge once the issue is completed & verified

[ShmakovVA]

'+'

[blackPanther39]

'+'

[koxudaxi]

@n0nvme Thank you for creating the PR. It's a good idea. I have added enhancement tag on the issue 😄

[ghandic]

👍

[xkortex]

Seconded. That would be great to have. I'm looking to use AsyncApi as an intermediate representation of sorts for generating stubs for Celery tasks.

However it seems there's a bit of a snag in that datamodel-code-generator itself cannot seem to run on the asyncapi jsonschema files. You could probably work around that, but I think it would be valuable to have that self-bootstrapping ability to be able to validate asyncapi documents. Might have to sort out #447 on the way to generating model code from AsyncAPI specs.

[Kludex]

Something I can do to push this forward? I'm super interested.

[koxudaxi]

I'm sorry for late reply.

@xkortex

However it seems there's a bit of a snag in that datamodel-code-generator itself cannot seem to run on the asyncapi jsonschema files.

I just checked the problem. patternPropetry doesn't work fine. I will fix it.

@Kludex I want to know the minimum requirement for the feature. We don't need to generate code for all parts of the schema. And, Is there an example of the API definition of json/yaml?

[Kludex]

I want to know the minimum requirement for the feature.

Being able to generate Python code from the following schemas: https://github.com/asyncapi/spec-json-schemas/tree/master/schemas

[phillipuniverse]

@koxudaxi FYI from the 2.6.0 schema, running this command:

datamodel-codegen --input asyncapi-schema-2.6.0.json --input-file-type jsonschema --output asyncapi.py --output-model-type pydantic.BaseModel

yields this error:

Traceback (most recent call last):
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/__main__.py", line 388, in main
    generate(
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/__init__.py", line 435, in generate
    results = parser.parse()
              ^^^^^^^^^^^^^^
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/base.py", line 1135, in parse
    self.parse_raw()
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1603, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1679, in _parse_file
    obj = JsonSchemaObject.parse_obj(model)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 270, in __init__
    super().__init__(**data)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for JsonSchemaObject
patternProperties -> ^x-
  value is not a valid dict (type=type_error.dict)

[phillipuniverse]

Ok looks like the problem is the spot where the value for patternProperties is true here in the AsyncAPI spec

If you change the true to {} then it starts to generate, but then you get this error:

Traceback (most recent call last):
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/http.py", line 6, in <module>
    import httpx

...
...
    from datamodel_code_generator.http import get_body
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/http.py", line 8, in <module>
    raise Exception(
Exception: Please run $pip install datamodel-code-generator[http] to resolve URL Reference

No big deal, I didn't have httpx, so I installed it:

poetry add --group dev 'datamodel-code-generator[http]'

Then the next error:

/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py:334: UserWarning: format of 'regex' not understood for 'string' - using default
  warn(f'format of {format__!r} not understood for {type_!r} - using default' '')
Modular references require an output directory, not a file

So then I re-ran with a folder instead of asyncapi.py:

poetry run datamodel-codegen --input asyncapi-schema-2.6.0.json --input-file-type jsonschema --output asyncapi_schema --output-model-type pydantic.BaseModel

And that generated something!

A little counterintuitive where the schemas got placed but not too big of a deal

[phillipuniverse]

I tried the same workaround with AsyncAPI 3.0.0 but so far no dice. I did my above changes:

1. Change patternProperties reference
2. use the http extra
3. Run with the folder

poetry run datamodel-codegen --input asyncapi-schema-3.0.0.json --input-file-type jsonschema --output asyncapi_schema --output-model-type pydantic.BaseModel

And here's my stack trace:

/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py:338: UserWarning: format of 'regex' not understood for 'string' - using default
  warn(f'format of {format__!r} not understood for {type_!r} - using default' '')
Traceback (most recent call last):
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/__main__.py", line 428, in main
    generate(
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/__init__.py", line 462, in generate
    results = parser.parse()
              ^^^^^^^^^^^^^^
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/base.py", line 1153, in parse
    self.parse_raw()
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1704, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1795, in _parse_file
    self.parse_raw_obj(key, model, path)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1639, in parse_raw_obj
    self.parse_obj(name, self.SCHEMA_OBJECT_TYPE.parse_obj(raw), path)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1665, in parse_obj
    self.parse_ref(obj, path)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1599, in parse_ref
    self.parse_ref(property_value, path)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1580, in parse_ref
    self.parse_ref(obj.items, path)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1577, in parse_ref
    self.resolve_ref(obj.ref)
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1566, in resolve_ref
    self._parse_file(
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1762, in _parse_file
    with self.root_id_context(raw):
  File "/Users/phillip/.pyenv/versions/3.11.6/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/phillip/Library/Caches/pypoetry/virtualenvs/shipwell-backend-core-cpwchxt0-py3.11/lib/python3.11/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1627, in root_id_context
    root_id: Optional[str] = root_raw.get('$id')
                             ^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'get'