Open ofek opened 2 years ago
@koxudaxi Do you have bandwidth for this? Is there anything we can do to help here?
@Kludex I appreciate your help :bow: I'm sorry. I know we should support Pydantic v2 soon :cry: But, Due to personal circumstances, I have yet to be able to accomplish much. Additionally, I have been spending time addressing this error, so the work is not progressing :(
Are you asking if the content you're talking about pertains to the output model, or is it about a library that the CLI depends on?
@ofek @Kludex I have started supporting Pydantic v2. I will merge and release the PR in a few days.
Thanks π
that's awesome thank you very much!
@ofek @Kludex
I have released the new version 0.21.0
The version can be run in Pydantic V2.
Also, There is --output-model-type pydantic_v2.BaseModel
option to generate pydantic v2 model
The detail is hereβ https://github.com/koxudaxi/datamodel-code-generator/releases/tag/0.21.0
__root__
field (a.k.a Custom Root Types)__root__
field (a.k.a Custom Root Types) is removed in pydantic v2.
The model is changed to RootModel
https://docs.pydantic.dev/2.0/migration/#changes-to-pydanticfield
typing.Set
. this feature is discussed in https://github.com/pydantic/pydantic-core/issues/296pydantic.Config
-> pydantic.ConfigDict
components:
schemas:
SharedConfig:
properties:
proxy:
properties:
http:
type: string
https:
type: string
no_proxy:
items:
type: string
type: array
type: object
service:
type: string
skip_proxy:
type: boolean
timeout:
type: number
required: []
paths:
/shared:
get:
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/SharedConfig'
β C:\Users\ofek\Desktop\code\integrations-core\datadog_checks_dev\datadog_checks\dev\tooling\confi β
β guration\consumers\model\model_consumer.py:117 in _process_section β
β β
β 114 β β β β field_constraints=True, β
β 115 β β β ) β
β 116 β β β print(f'Parsing the OpenAPI schema `{schema_name}`') β
β β± 117 β β β parsed_section = section_parser.parse() β
β 118 β β except Exception as e: β
β 119 β β β raise β
β 120 β β β print(f'Error parsing the OpenAPI schema `{schema_name}`: {e}') β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\base.py:995 in parse β
β β
β 992 β β format_: Optional[bool] = True, β
β 993 β β settings_path: Optional[Path] = None, β
β 994 β ) -> Union[str, Dict[Tuple[str, ...], Result]]: β
β β± 995 β β self.parse_raw() β
β 996 β β β
β 997 β β if with_import: β
β 998 β β β if self.target_python_version != PythonVersion.PY_36: β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\openapi.py:549 in parse_raw β
β β
β 546 β β β β β obj_name, β
β 547 β β β β β raw_obj, β
β 548 β β β β ) in schemas.items(): # type: str, Dict[Any, Any] β
β β± 549 β β β β β self.parse_raw_obj( β
β 550 β β β β β β obj_name, β
β 551 β β β β β β raw_obj, β
β 552 β β β β β β [*path_parts, '#/components', 'schemas', obj_name], β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:1535 in parse_raw_obj β
β β
β 1532 β β raw: Dict[str, Any], β
β 1533 β β path: List[str], β
β 1534 β ) -> None: β
β β± 1535 β β self.parse_obj(name, JsonSchemaObject.parse_obj(raw), path) β
β 1536 β β
β 1537 β def parse_obj( β
β 1538 β β self, β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:1552 in parse_obj β
β β
β 1549 β β β if isinstance(data_type, EmptyDataType) and obj.properties: β
β 1550 β β β β self.parse_object(name, obj, path) β
β 1551 β β elif obj.properties: β
β β± 1552 β β β self.parse_object(name, obj, path) β
β 1553 β β elif obj.patternProperties: β
β 1554 β β β self.parse_root_type(name, obj, path) β
β 1555 β β elif obj.type == 'object': β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:909 in parse_object β
β β
β 906 β β ) β
β 907 β β class_name = reference.name β
β 908 β β self.set_title(class_name, obj) β
β β± 909 β β fields = self.parse_object_fields(obj, path, get_module_name(class_name, None)) β
β 910 β β if fields or not isinstance(obj.additionalProperties, JsonSchemaObject): β
β 911 β β β data_model_type_class = self.data_model_type β
β 912 β β else: β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:864 in parse_object_fields β
β β
β 861 β β β β ) β
β 862 β β β β continue β
β 863 β β β β
β β± 864 β β β field_type = self.parse_item(modular_name, field, [*path, field_name]) β
β 865 β β β β
β 866 β β β if self.force_optional_for_required_fields or ( β
β 867 β β β β self.apply_default_values_for_required_fields and field.has_default β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:1038 in parse_item β
β β
β 1035 β β elif item.is_object or item.patternProperties: β
β 1036 β β β object_path = get_special_path('object', path) β
β 1037 β β β if item.properties: β
β β± 1038 β β β β return self.parse_object( β
β 1039 β β β β β name, item, object_path, singular_name=singular_name β
β 1040 β β β β ) β
β 1041 β β β elif item.patternProperties: β
β β
β C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator β
β \parser\jsonschema.py:936 in parse_object β
β β
β 933 β β β data_model_type_class = self.data_model_root_type β
β 934 β β β
β 935 β β self.set_additional_properties(class_name, obj) β
β β± 936 β β data_model_type = data_model_type_class( β
β 937 β β β reference=reference, β
β 938 β β β fields=fields, β
β 939 β β β custom_base_class=obj.custom_base_path or self.base_class, β
β°βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ―
TypeError: 'DataModelType' object is not callable
@ofek I have tested your example. It works fine. I used python v3.11.4 and Linux(ubuntu) Could you share your CLI arguments?
β― datamodel-codegen --input openapi.yaml --input-file-type openapi
# generated by datamodel-codegen:
# filename: openapi.yaml
# timestamp: 2023-07-03T22:58:36+00:00
from __future__ import annotations
from typing import List, Optional
from pydantic import BaseModel
class Proxy(BaseModel):
http: Optional[str] = None
https: Optional[str] = None
no_proxy: Optional[List[str]] = None
class SharedConfig(BaseModel):
proxy: Optional[Proxy] = None
service: Optional[str] = None
skip_proxy: Optional[bool] = None
timeout: Optional[float] = None
OK, I see you use codegen as a library.
from datamodel_code_generator import DataModelType
from datamodel_code_generator.parser.openapi import OpenAPIParser
SPEC = """\
components:
schemas:
SharedConfig:
properties:
proxy:
properties:
http:
type: string
https:
type: string
no_proxy:
items:
type: string
type: array
type: object
service:
type: string
skip_proxy:
type: boolean
timeout:
type: number
required: []
paths:
/shared:
get:
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/SharedConfig'
"""
def main():
parser = OpenAPIParser(SPEC, data_model_type=DataModelType.PydanticV2BaseModel)
parser.parse()
if __name__ == '__main__':
main()
β― C:\Users\ofek\AppData\Local\Programs\Python\Python311\python.exe ..\..\..\pd.py
C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\pydantic\main.py:292: UserWarning: Pydantic serializer warnings:
Expected `Union[list[definition-ref], definition-ref, bool]` but got `JsonSchemaObject` - serialized value may not be as expected
return self.__pydantic_serializer__.to_python(
Traceback (most recent call last):
File "C:\Users\ofek\Desktop\pd.py", line 45, in <module>
main()
File "C:\Users\ofek\Desktop\pd.py", line 41, in main
parser.parse()
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\base.py", line 995, in parse
self.parse_raw()
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\openapi.py", line 549, in parse_raw
self.parse_raw_obj(
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 1535, in parse_raw_obj
self.parse_obj(name, JsonSchemaObject.parse_obj(raw), path)
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 1552, in parse_obj
self.parse_object(name, obj, path)
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 909, in parse_object
fields = self.parse_object_fields(obj, path, get_module_name(class_name, None))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 864, in parse_object_fields
field_type = self.parse_item(modular_name, field, [*path, field_name])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 1038, in parse_item
return self.parse_object(
^^^^^^^^^^^^^^^^^^
File "C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 936, in parse_object
data_model_type = data_model_type_class(
^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'DataModelType' object is not callable
@ofek Thank you for your example code. The parser needs five parameters for v2 output model type.
I will update the documents
...
def main():
from datamodel_code_generator import DataModelType, PythonVersion
from datamodel_code_generator.model import get_data_model_types
data_model_types = get_data_model_types(DataModelType.PydanticV2BaseModel,
target_python_version=PythonVersion.PY_38)
parser = OpenAPIParser(SPEC,
data_model_type=data_model_types.data_model,
data_model_root_type=data_model_types.root_model,
data_model_field_type=data_model_types.field_model,
data_type_manager_type=data_model_types.data_type_manager,
dump_resolve_reference_action=data_model_types.dump_resolve_reference_action,
)
parser.parse()
...
Thanks! Finally, do you know what this warning is and how it might be resolved?
β― C:\Users\ofek\AppData\Local\Programs\Python\Python311\python.exe ..\..\..\pd.py
C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\Lib\site-packages\pydantic\main.py:292: UserWarning: Pydantic serializer warnings:
Expected `Union[list[definition-ref], definition-ref, bool]` but got `JsonSchemaObject` - serialized value may not be as expected
return self.__pydantic_serializer__.to_python(
@ofek I'm checking the warning. I don't think critical issue because unittests work fine, same as v1. I'll reach out to you after I fix the warning.
If any help is needed, feel free to ping me. π
For optional fields calling this as a library does not append a = None
which is now required in v2 https://docs.pydantic.dev/2.0/migration/#required-optional-and-nullable-fields
@ofek
I'm sorry I couldn't catch some cases.
I have released the fixed version 0.21.1
.
If the version doesn't work with your input, please share your input schema.
@ofek I'm sorry I couldn't catch some cases. I have released the fixed version
0.21.1
. If the version doesn't work with your input, please share your input schema.
I get warnings with datamodel-codegen
0.21.1
Expected `Union[definition-ref, bool]` but got `JsonSchemaObject` - serialized value may not be as expected
for the following file:
Command that I use: datamodel-codegen --input ~/Downloads/TransferSearch_v1_Version_1.11_swagger_specification.json --snake-case-field --use-union-operator --reuse-model --target-python-version 3.11 --use-double-quotes
@AIGeneratedUsername I reported the error to pydantic team. https://github.com/pydantic/pydantic/issues/6422 They said it is not easy to fix the warning. I should disable the warning in the datamodel-code-generator.
Hi. Not sure whether I should open a new issue or use this one. Since this one is still open I thought I should use it. Tell me if otherwise.
There is a problem with the generation of RootModel.
I created a brand new project and installed only datamodel-code-generator
. Here are the dependencies I obtained:
annotated-types==0.6.0
argcomplete==3.2.1
black==23.12.1
click==8.1.7
datamodel-code-generator==0.25.2
dnspython==2.4.2
email-validator==2.1.0.post1
genson==1.2.2
idna==3.6
inflect==5.6.2
isort==5.13.2
Jinja2==3.1.2
MarkupSafe==2.1.3
mypy-extensions==1.0.0
packaging==23.2
pathspec==0.12.1
platformdirs==4.1.0
pydantic==2.5.3
pydantic_core==2.14.6
PyYAML==6.0.1
typing_extensions==4.9.0
Then I created this simple json schema file:
{
"type": "array",
"items": {
"type": "object",
"properties": {
"conf_name": {
"type": "string"
}
},
"required": [
"config_name"
]
}
}
When I ran datamodel-codegen
on it, I obtained the following result:
# generated by datamodel-codegen:
# filename: scratch_83.json
# timestamp: 2023-12-27T09:18:22+00:00
from __future__ import annotations
from typing import List, Optional
from pydantic import BaseModel
class ModelItem(BaseModel):
conf_name: Optional[str] = None
class Model(BaseModel):
__root__: List[ModelItem]
Using that code results in the following error:
TypeError: To define root models, use `pydantic.RootModel` rather than a field called '__root__'
Digging into Pydantic's doc on the usage of RootModel, I found that Pydantic v2 requires one of the following:
class Model(RootModel):
root: List[ModelItem]
# or
Model = RootModel[List[ModelItem]]
@esciara
--output-model-type pydantic_v2.BaseModel
you should use the option to generate pydantic v2 models.
--output-model-type {pydantic.BaseModel,pydantic_v2.BaseModel,dataclasses.dataclass,typing.TypedDict,msgspec.Struct}
Thanks for the tip @koxudaxi!
Should that not be the default by now? Or at least, should there not be a message during generation mentioning that one or the other version is used? Otherwise it would be difficult to know...
Even just warning that the default is generating v1 model is ok.
Just FYI https://pydantic-docs.helpmanual.io/blog/pydantic-v2/
Likely requires dropping < v2