Decide on the tools and architecture of the backend

[tulir]

Before doing anything else we should decide a few things:

• What functionality is needed?
• How should the backend be structured?
• What tools should we use?

We'll most likely use Flask for the web server and SQLAlchemy for the database. We'll probably also want to use something like connexion or swagger-py-codegen

For the the server architecture, there are currently two main ideas:

1. A Git-like server where saved specs are immutable, but anyone can create a new revision based on a saved spec. This approach wouldn't require implementing any authentication.
2. An open-source implementation of the Swagger Hub spec

[tulir]

The Git-like approach is probably better when integrating into the APInf platform or other platforms that have their own authentication system. However, it would not be as easy to use as a standalone system.

[MikeRalphson]

Happy with Flask/SQLAlchemy. Connexion seems to have the edge over swagger-py-codegen (which is only an Alpha release) at least when it comes to generating runnable code from the swaggerhub API definition.

A Git-like server where saved specs are immutable, but anyone can create a new revision based on a saved spec. This approach wouldn't require implementing any authentication.

I can see a simple key/hash : value store being simpler to get up and running, but I suspect creating namespaces, aliases or some other mechanism to separate the API definitions would be almost as much work as authentication.

[mauriciovieira]

A Git-like server where saved specs are immutable, but anyone can create a new revision based on a saved spec. This approach wouldn't require implementing any authentication.

In this case, we might want to not handle version-control in database manually. Is there anything like the solutions for rails?

An open-source implementation of the Swagger Hub spec

I like this idea better. Since the endpoints and specs are already there, it can be a nice opportunity to do test-driven-development.

[brylie]

If we might be having user accounts, Django might be worth considering:

One tipping point I have heard multiple times is that if you need a login system for your users definitely consider Django over Flask.

Gnlfbz on /r/django discussion Why Django over Flask?

---

As soon [user accounts] become a requirement, I consider Django over Flask.

_wsgeorge on /r/django discussion Why Django over Flask?

There is also a Django extension, called Reversion, that makes it easy to store document revisions.

[brylie]

Another interesting comment:

I would go for DRF (though I haven't used Flash extensively), because it will include features that you may not be aware you need them until it's too late (such as multiple, pluggable authentication systems or the ease of its included serializers).

Bare-bones routing, implementation of HTTP verbs and DB lookups are the easy part (and completely do-able in vanilla Django or Flask). It's the extra bits that make an API usable and cleanly implementable that are hard and you get those with DRF.

virtron on /r/django discussion Flask vs Django Rest Framwork

[brylie]

This comment is somewhat relevant, although we have not discussed whether we will have a UI that needs forms:

When starting a new hobby project I always consider using Flask first unless it needs models. If I need models, there's a 98% probability that I need user management and forms too. And instead of installing flask-sqlalchemy, flask-wtf, flask-login and whatnot, I just use Django instead.

Retzudo on /r/flask discussion Django vs Flask: A practitioner’s perspective (2017)

This one is straightforward:

for database apps we use django.

scottocom on /r/flask discussion Django vs Flask: A practitioner’s perspective (2017)

[brylie]

I realize these are cherry-picked comments (aren't all quotes cherry picked?). In any case, the discussions are relevant to our current decision.

I am fine with either Django or Flask, but just wanted to share the Django side of things.

[xylix]

@brylie Before deciding on web frameworks we should probably decide which of these:

• A Git-like server where saved specs are immutable, but anyone can create a new revision based on a saved spec. This approach wouldn't require implementing any authentication.
• An open-source implementation of the Swagger Hub spec

solutions we are going with, or are we planning on doing something else?

[brylie]

The quotes above are intended to support the discussion of user authentication, document revisioning, etc., not to force a decision about framework at this point.

For the Git-like approach, how might an OpenAPI document be identified? Would namespacing be relevant/needed?

[tulir]

Namespacing would probably not be needed, at least not at the beginning. We could just store a hash or other unique value as the key and somehow link other revisions to make browsing the history/future possible.

[brylie]

One idea would be to store a revision history within the APInf codebase, since we are planning to integrate the designer interface into the APInf UI. That would help keep the OpenAPI Space project requirements small.

Also, we agreed that document revisions, in the sense of keeping a history of revisions linked in some way, was out of scope.

[xylix]

@brylie You mean something like OpenAPI Space would only store the freshest versions of a spec and when the user saves a new version the old version would be sent into an apinf.io side revision database? Would this eliminate the benefit of not requiring authentication for users of OpenAPI Space?

Also I don't see the point of splitting code that stores OpenAPI specs this way? Why not just have it all in one place, OpenAPI Space? The same amount of code will be required anyway to reach the same functionality, or at least that's what I am thinking.

[brylie]

You mean something like OpenAPI Space would only store the freshest versions of a spec and when the user saves a new version the old version would be sent into an apinf.io side revision database?

No, just that each revision would be it's own document. Each document could have a unique identifier (e.g. hash). The OpenAPI specs are only stored in OpenAPI Space.

I only suggested that APInf could store the revision history, e.g. as an array of hashes. That way OpenAPI Space doesn't need to link the revisions.

[tulir]

Yeah, we could leave linking the revisions to a later version or even leave it out completely. Even with linked revisions, APInf would need to remember what revision the user is on since anyone could create a new revision.

[brylie]

Jep. We already agreed that revisions are out-of-scope for the current project phase. We are 'on the fence' about authentication.

Basically, I can think of a way we could handle revisions, if needed, in APInf Platform.

[xylix]

@brylie Thanks for the clarification. Yeah that would make sense if we want to go for the "hash solution" for stored spec identification. If so, we need to design the API.

[mauriciovieira]

I vote for Django Rest Framework + Django Reversion + user authentication + document versioning (for each user) + test driven development

[MikeRalphson]

Is it possible to do design-first development with Django REST Framework - i.e. drive/generate the API implementation from an OpenAPI/Swagger definition? Maybe m'colleague @piersstorey can comment.

[brylie]

Is it possible to do design-first development with Django REST Framework - i.e. drive/generate the API implementation from an OpenAPI/Swagger definition?

Good question. I will also research this idea and wait for any comment from @piersstorey.

In any case, we can design first by creating a specification file and then developing code based on that file.

[brylie]

After searching around, I can't find any tooling to generate a Django REST Framework API from a Swagger/OpenAPI Spec file. The question has been raised on StackOverflow in 2016, with no answer.

[xylix]

@brylie Here is one quite big project that does that: https://github.com/marcgibbons/django-rest-swagger

edit: That tool does the exact reverse of what we are trying to do. I read the description wrong, sorry.

[brylie]

@xylix I think Django REST Swagger will generate an OpenAPI/Swagger spec from server code.

code --> spec file

I think @MikeRalphson was asking about any tools that would generate server code from a specification.

spec file --> code (server stub)

See the following projects for examples of the spec --> code idea:

• Connexion
• Swagger Codegen
• Swagger Py Codegen

[piersstorey]

@brylie @xylix @MikeRalphson Django REST Swagger generates OpenAPI/Swagger spec from server code eg: Django model (Data ORM class) -> views.py -> Swagger spec

[xylix]

There doesn't seem to be any well-known ways of generating django server side code from OpenAPI specs. I like the idea of using a design first approach so I think we should go with flask.

[brylie]

I like the idea of using a design first approach

Lets start with an OpenAPI spec design file. That way, we can get validation for the design ideas quickly.

Later, we can decide whether generating a server stub would be useful in our development process.

[tulir]

I created a pull request for a basic OpenAPI spec: #6

[MikeRalphson]

• https://www.git-pull.com/code_explorer/django-vs-flask.html
• HackerNews discussion https://news.ycombinator.com/item?id=14690638

[xylix]

I found a design first django tool. https://github.com/akx/lepo However it calls itself a "contract first API framework".

[brylie]

Lepo is Finnish for rest.

Haha, appropriate 😁

[mauriciovieira]

@xylix @tulir @brylie I took a look at Lepo, it just generates plain function-based views, and don't profit from the best of Django Rest Framework.

[mauriciovieira]

So, my vote to end this discussion is:

• Choose a person that will have the last word :-p (not me)
• Finally decide whether we will have user authentication - I go for ✅
• Start with a final spec. So we version Swagger Hub spec in this repository and then strip off the parts we think that are not going to be used. So in this case, we would not start with a basic spec ( #6 ).
• After (quickly) deciding on the OpenAPI spec, we can start using Lepo, and generate the skeleton of the backend using that codegen. But we also should keep in mind that that we will probably change 80 % of the generated code to use DRF's ModelViewSet (much better than lepo's generated method views)
• Use test driven development and aim for 90%+ of code coverage.

[brylie]

My 'vote' goes to Django REST Framework / Lepo.

[brylie]

Choose a person that will have the last word :-p (not me)

I move that @xylix should make the final decision. All in favor, please give a thumbs up on this comment.

[xylix]

We are planning to use django. Will confirm other details within 19 hours.

[xylix]

The SwaggerHub spec contains tons of definitions that are hard to understand without getting a look at their own implementations. We are going to create our own API spec based on SwaggerHub's, which will try to contain as much of the needed features as possible.

We will implement authentication using django's authentication system.

We won't use Lepo, because if we need to rewrite the code anyway it won't really help.

We won't start by writing tests but we will aim for high code coverage soon.

[MikeRalphson]

We won't use Lepo, because if we need to rewrite the code anyway it won't really help.

Is it worth forking Lepo to produce DRF ModelViewSets instead, and look at contributing that code back, or with a cut-down OpenAPI base, will there not be such a need for code-generation?

[brylie]

@akx what are some considerations and challenges regarding Lepo supporting DRF ModelViewSets?

[tulir]

We tried using ModelViewSets, but they seem to be a bit too abstract, since there's some work to do to convert the data being POSTed to our data models.

[akx]

Hey guys :)

Glad you stumbled upon Lepo – I haven't really advertised it anywhere and it's definitely in beta stage and nowhere near DRF's scope at present (nor do I think it'll ever be, really!). Disclaimers aside, then...

The reason I wrote Lepo in the first place was that @japsu was lamenting the lack of design-first API tools for Django. As noted earlier in the thread, DRF and related tools only do the reverse (and in my opinion, not that well.) So if you want something that you first write an OpenAPI spec for, then write the code, there aren't really any options for Django other than Lepo.

@mauriciovieira:

I took a look at Lepo, it just generates plain function-based views, and don't profit from the best of Django Rest Framework.

The codegen is really an afterthought/convenience tool in Lepo that gets you kickstarted/scaffolded with handlers for the various API calls described by the API.

To further explain what's going on in Lepo ––

In Django, views are callables with the signature (request, *args, **kwargs) -> Response, where the args and kwargs come from the URL router. Lepo adds a transform layer of sorts (the router and the views it generates) there; if you have an OpenAPI operation with parameters, say, name and comment, you'd instead write a handler with the signature (request, name, comment) -> Response | dict and map it through the operationId field in your OpenAPI doc. Lepo will take care of transforming the input data (be it query parameters, form data, raw POST or path fragments) into those args.

Anyway... Although it hasn't been tried, there's nothing that'd really stop you from using DRF's viewsets as the "backend" for Lepo/OpenAPI -described API calls. The glue code shouldn't be too complex, though of course you'd have to be careful to keep your DRF serializers etc. in sync with your OpenAPI document.

Oh, and for what it's worth, Lepo's (ahem, undocumented-but-tested) class-based handler API is loosely based on DRF viewsets anyway.

Hope this helps! Also, I'd be super glad to help you guys in any way if you decide to go down the Django/Lepo route!

[xylix]

We have been working at django-test, lepo-test and flask-test with @tulir for a couple of workdays now. We had problems using Django, partially because of our inexperience with it, so we decided that for getting this project properly going flask will work better for now. So we are now making a PR from flask-test into master which will be the implementation we are now planning to use, at least for now. It uses Flask with connexion and SQLAlchemy.

[tulir]

The Flask version is now working with support for all operations in the spec and has been merged into master.