Open dmontagu opened 5 years ago
@dmontagu Thank you for talking me a great plan. OK, let's think about how to develop features.
I had decided to create another code generator as different projects.
But, I'm thinking datamodel-code-generator
should support other features as an extension.
Users don't want to install and run a lot of tools.
I imagine the cli accepts features a sub-command or options
eg.
$ datamodel-codegen --input api.yaml fastapi-client
Btw, I tell you the base plan on this project.
I think the project should convert data models to python objects.
I expected source models are OpenAPI
and JsonSchema
, DDL of SQL
, GraphQL
, smithy and more.
I want to create Pydantic's BaseModel as output.
However, someone may need marshmallow
or python's dataclass
.
I may implement other output models.
That sounds good to me. I would be on-board with it accepting arbitrary (jinja) templates, but shipping with one or more, including at least pydantic.
I think handling APIs is a natural extension -- it just involves a (related) templated-generation step based on the operations, rather than the schemas.
As a separate note, if we support API client generation, I think we should design the model-generation-templates in such a way that, if your server-side models are all in a self-contained package (possibly with some constraints on the organization of that package), it would be possible to just include your server-side models code directly if desired. (Even if you don't need to generate the models, you still need most of the model-generation logic to generate the api-client operations from the openapi spec.)
This is currently how I use my fastapi_client package -- even though it can generate models, I actually just overwrite the generated models with my server-side models code, and mostly use the generator as a way to generate the APIs (for which there is not a server-side analogy). This way I can retain model methods, which obviously are not supported via OpenAPI schema; it helps make the generated client into something closer to an SDK.
@dmontagu I agree.
As a separate note, if we support API client generation, I think we should design the model-generation-templates in such a way that, if your server-side models are all in a self-contained package
How did you think about technical design for other features? (eg. fastapi client) Now, Paser class receive the model class.
parser = OpenAPIParser(
BaseModel,
CustomRootType,
filename=input_filename,
base_class=namespace.base_class,
)
I think we should change to
parser = OpenAPIParser(
BaseModel,
CustomRootType,
filename=input_filename,
base_class=namespace.base_class,
client=FastAPIClient, # or AsyncClient
server=FastAPIServer
)
Also, we may distribute a separate package like this.
$ pip install datamodel-code-generator[fastapi]
I don't know the idea is good. but, I don't want a package heavyweight. I see it's a balance.
it would be possible to just include your server-side models code directly if desired. (Even if you don't need to generate the models, you still need most of the model-generation logic to generate the api-client operations from the openapi spec.)
I feel we must think about the usability for people at first. next is extensibility.
This way I can retain model methods, which obviously are not supported via OpenAPI schema; it helps make the generated client into something closer to an SDK.
I'm interested. It's a high practicality client 😄
@dmontagu How did we start the implement? I want to merge your api-client. but, I think I should develop the pycharm-plugin first. Would you create a PR? or Are you busy supporting pydantic community for releasing version 1?
@koxudaxi Here's my tentative breakdown of the steps we'll need to cover (probably in this order):
SyncApis
and AsyncApis
classes (or something equivalent) for use as the primary external interface to the generated client. I think the first thing to do is to port the generation logic and templates for the apis from openapi-generator format to jinja. I may not be able to get to that today/for a little while, but I'll take responsibility for doing it and making a PR as soon as I can.
Does that make sense to you?
@dmontagu OK. I seem good the steps.
datamodel-code-generator has the Import
and Imports
classes for generating import statements dynamically.
We can use the classes to follow the steps when parsing an input file.
https://github.com/koxudaxi/datamodel-code-generator/blob/4eed7b29eddf1081a648f7451a05cdc791af3a99/datamodel_code_generator/types.py#L7 https://github.com/koxudaxi/datamodel-code-generator/blob/4eed7b29eddf1081a648f7451a05cdc791af3a99/datamodel_code_generator/types.py#L19
from datamodel_code_generator.types import Import, Imports
pydantic_import = Import(import_='BaseModel', from_='pydantic')
optional_import = Import(import_='Optional', from_='typing')
list_import = Import(import_='List', from_='typing')
datetime_import = Import(import_='datetime')
imports = Imports()
imports.append(pydantic_import)
imports.append(list_import)
imports.append(optional_import)
imports.append(datetime_import)
print(imports.dump())
from pydantic import BaseModel
from typing import List, Optional
import datetime
I may not be able to get to that today/for a little while, but I'll take responsibility for doing it and making a PR as soon as I can.
Thank you. No problem. Would you create the PR when you have time to do?
@koxudaxi that import management is pretty cool!
However, I am concerned about cyclic schema-references (at least with pydantic), which can happen even in cases where there are no cyclic object references. For example, imagine modeling a factory process, where a number of workers are ordered in a sequence, worker types may repeat, and each worker has a reference to the next worker.
This could be modeled in a single file via:
from pydantic import BaseModel
from typing import Union
class WorkerA(BaseModel):
worker_id: str
next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]
class WorkerB(BaseModel):
worker_id: str
next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]
class WorkerC(BaseModel):
worker_id: str
next_worker: Union["WorkerA", "WorkerB", "WorkerC", None]
WorkerA.update_forward_refs()
WorkerB.update_forward_refs()
WorkerC.update_forward_refs()
But note that the "naive approach" where one file is generated per model and there are no forward refs will cause import errors.
I initially thought that this would prevent pydantic from working with an arbitrary openapi schema, but having worked through the above example, I now believe this can be handled in general by putting every BaseModel type-hint in double quotes, and calling update_forward_refs
on all models from another file after all models have been imported successfully.
But I think we might need to be careful like this.
Separately, I spent some time reading through the datamodel-code-generator code, and I'm having a little bit of a hard time understanding everything it is doing -- it seems very abstract to me! (But that's not surprising given the goal of supporting arbitrary models...).
In order to support API generation, I think we'd first need something equivalent to the DataModel
class, but for the paths
field of an OpenAPI spec, rather than the schemas
. (I think I'm understanding it correctly, but maybe not quite yet..).
@koxudaxi I'm interested in how you are thinking about the relationship between this project and OpenAPI -- do you want this project to be broader in scope than OpenAPI? If so, I think it might be awkward to include API generation, since that seems very OpenAPI-specific.
I'm interested to hear your thoughts.
@dmontagu
Thank you for your great concern.
I think the way is right to resolve the problem of cyclic schema-references.
However, It may be difficult to detect all invalid situation with cyclic schema-references before implement the code generator.
I seem not unfortunate to implement and run unittest of a few patterns.
@dmontagu Sorry, the datamodel-code-generator code may be complex because the code separates three layers. I show you the relation. parser(eg. openapi parser) --- middle-datamodel --- code-generator(eg. pydantic, fastapi-clinet)
I thought about the scope of the code generator yesterday.
In order to support API generation, I think we'd first need something equivalent to the DataModel class, but for the paths field of an OpenAPI spec, rather than the schemas. (I think I'm understanding it correctly, but maybe not quite yet..).
I think DataModel
is for the data class object.
Also, DataModel
inherit from TemplateBase
.
class DataModel(TemplateBase, ABC):
But, paths
have a side of Action
in OpenAPI.
We should create Action
. class
eg.
class ActionBase(TemplateBase, ABC): # or better name
.... # implement to render methods for creating `function` or `method`
class OpenAPIAction(ActionBase)
path: str
method: str ## http method
params: List[Param]
request_body: # ...
I'm interested in how you are thinking about the relationship between this project and OpenAPI -- do you want this project to be broader in scope than OpenAPI?
First, I wanted to create another project for OpenAPI generator includes Client/Server which uses datamodel-code-generator.
However, my thinking had changed by your suggestions on the issue.
Imagine if we need another code generator then, Do we create a new one every time?
We and users will confuse.
Also, I do not think to create another model generator project each data type source (eg. JsonSchema, GraphQL). because of the same reason.
btw, swagger-ui
project gets distressed to support openapi which may not expect next-generation schema type.
If we can support multi-type sources, it's the best way. I would solve the complex problem of implementation by architecture. The answer may be wrong. but, I feel a worthly challenge.
@dmontagu Have you been interested in the issue yet?
I thought about the issue for a long time.
Also, I use the code-generator for our daily project with co-workers.
I think code-generator is for pre-production ready.
This code-generator supports features enum
, allof
, anyof
...
I have the next plan, which is to create a code-generator for fastapi server. I said this code generator should have fastapi feature. But, It's wrong. If I add a feature of server-side into the code-generator then, It's too complex to use. Also, I will be not able to maintain the code. Your thought is correct.
Today, I detect to create a new project of code-generator for fastapi. I guess the fastapi generator can call this code generator as a module to build pydantic model.
I would set the first scope is to create code-generator for server-side in the next project. And, a second is to create code-generator for the client.
If you are interested in creating an API Client generator then, Would you build a new project with me or give me your code for API Client?
I have thought some more about it.
You can take a look at the fastapi_client repo. (Feel free to fork or whatever if you want to use the code.)
There are some pydantic features coming down the pipe that make building a typed client easier (e.g. parse_as_type). I think this would be very nice when writing tests of your server (not to mention broader use with other OpenAPI APIs).
I have two main frustrations with the current design of the client: 1) it uses generated models rather than the server side models. It’s nice to use exactly the server side models if possible so that you can reuse logic. 2) It uses openapi-generator, which doesn’t do a great job of handling complex edge cases.
The main benefit I see is getting a (typed) client for your API — currently you can’t get this at all, even if you are willing to copy server code.
I think my priorities may be somewhat orthogonal to yours, given I prefer to reuse the server models directly. But if you have implementation requests related to the client generator stuff I’d be happy to put some in some thought/effort.
@dmontagu Thank you for showing me your thoughts
I decided to delay create a new fastapi code generator by your opinion.
parse_as_type
is a very good feature. I'm waiting for published version 1 of pydantic.
OK, I will think about the client.
Any news on this? At Boloo we are faced with the desire to have an API Client, fully typed. @Bobronium and I thought we could extend your package by having it also generate an API client. We then noticed this thread.
The tool would generate typed endpoints from OpenAPI spec that will interact with a manually-written client (so you could decide how you do actual HTTP requests).
Generating models is a big (if not the biggest) part of it. We are starting the project and currently discovering possibilities. IMO, integrating with datamodel-code-generator
would be a great way of implementing it and solve a lot of issues we would discover along the way.
We would love to hear your thoughts on it.
Hello from @Boloo-group, @koxudaxi! Would love to collaborate with you on the proposal above :)
In our heads, this would be a separate open source project (we've already came up with a sweet name) that would would use this one as its main tool. That will likely require some changes to this project, like providing a stable API for such needs.
Would love to hear your ideas and thoughts on that.
Hi, @guushuizen @Bobronium Thank you for sharing the idea 👍
I think a typed API client generator is very good for everyone. But, I don't need the client generator in my current private project. So, I don't know what we need as a feature 😢
If you have some requests to define a stable API then I help you.
I expect someone uses datamodel-code-generator
as a library.
I feel it's a great opportunity to decide the public interface.
I really like the idea behind this -- jinja is a lot easier to work with than openapi-generator templates and the java-based generators.
Would you have any interest in incorporating api-client-generation capability into this codebase? I would be happy to put a little effort into transforming the API client and templated API generation from https://github.com/dmontagu/fastapi_client into something that would combine better with this code base. That way, it would 1) have fewer generation-time dependencies 2) an easier-to-work-with templating language, and 3) leverage the work you've already put in for model generation.
But I would also understand if you'd rather keep this repository more limited in scope and targeted specifically around model generation.