[?] Basic technology: serving web applications

[bernhardreiter]

A complete intelmq solution will typically contain one or several "management" interfaces. A common approach (in these times) are graphical user interfaces via the web browser. While the core of intelmq may run without a graphical user interfaces, several settings will require one or benefit from it.

The existence of https://github.com/certtools/intelmq-manager shows the needs for configuration and operations management. A user interface will guide the administrators so that common actions are less error prone.

Additional use cases are:

• a statistics interface, to get overview reports.
• a contact management interface (Like envisoned for a contact database https://github.com/Intevation/intelmq/tree/certbund-contact )
• an interface to inspect how certain reports came to be, if the helpdesk is contacted over a special report.

One idea for larger organisations using intelmq: There may be users that are only allowed to performe a subset of actions, like they could change a contact that is reported to, but should not be able to change the bot configuration. Or they can create reports or see how a report came to be, but they cannot change anything on the system, just view it. So we may need the concept of users, roles and rights.

It makes sense that all web applications for intelmq use the same base technology. This issue is about selecting a base web technology that would cater a RESTful one-page application where the client does most of the work and the server only serves some pages, does authentication and serves and manipulates data. It should serve intelmq-manager and other potential needs mentioned above. (Thus it should be used to solve https://github.com/certtools/intelmq-manager/issues/80) As technology only python3 seems sensible.

[aaronkaplan]

I don't see that part of the core intelmq functionality. Having these features as part of the core part (or as part of this repo) would contradict the KISS principle of intelmq. I believe we need such features, but definitely outside of the core.

I believe we should concentrate on

1. defining these tools
2. defining the necessary interfaces with intelmq (whatever they might be)
3. document them

@bernhardreiter I am currently starting to write down feature requests for version 2.0. Will include those for 2.0. But right now, we need to ship 1.0 first.

[bernhardreiter]

@aaronkaplan your remark is good: I also do not see it as part of the core-technology (and source code), but as a recommended basic technology decision. Like: If you want to create a web-app that is useful for running intelmq(-core) use technology X.

I've separated this from the decision for certtools/intelmq-manager#80, because I believe it is good for intelmq installations to only come with one basic web server technology. And we probably will starting to share a user/role/permission system at one point, too. So it is decision for recommendation to be taken by the whole intelmq "ecosystem".

[bernhardreiter]

Selection criteria:

• technical it should be able to take over the intelmq manager serving needs.
• python3
• actively maintained, healthy community (multi-person)
• (optional) well packaged in common GNU/Linux distributions
• documentation
• should be able to make it easy to serve json files in a RESTful way or over GraphQL
• good security track record and overall design
• leans towards asynchronous JavaScript and JASON (AJAJ) or Single Page Applications
• Free Software (of course), with a license compatible to intelmq

[bernhardreiter]

Candidates (entries mentioning python3, lending towards micro services/restful and look active):

• Flask by Armin Ronacher Wikipedia entry, popular
• Bottle by Marcel Hellkamp Wikipedia entry, lightweight, little dependencies
• CherryPy, includes a webserver
• Pyramid, minimalistic, platform-independent MVC object publishing, allows hierarchical object traversal Wikipedia entry
• hug, by Timothy Crosley , code repo, designed for Python3 only
• circuits.web
• Morepath, by Martijn Faassen, combines routing with view lookup, opinionated , comparison to other web frameworks
• https://github.com/pecan/pecan , Object-dispatch routing

Less suitable (in my initial analysis)

• https://en.wikipedia.org/wiki/Django_%28web_framework%29, because to big
• https://en.wikipedia.org/wiki/Tornado_%28web_server%29, more focus on a web server
• https://en.wikipedia.org/wiki/TurboGears, designed around the model–view–controller (MVC) architecture
• http://www.growler.rocks/ , only one contributor
• http://pycnic.nullism.com/ , ulta light-weight, has not reached 0.1 yet
• Kyoukai, python3.5 only, two contributors.
• http://pythonhosted.org/wheezy.web/ , low on contributors

Lists to check:

• https://en.wikipedia.org/wiki/Comparison_of_web_frameworks#Python
• https://wiki.python.org/moin/WebFrameworks/

[bernhardreiter]

Now after finding the candidates, check the next IRCA step, which is "read existing reviews".

Some rewiews I found:

• http://www.slideshare.net/r1chardj0n3s/web-microframework-battle 2011 Richard Jones comparing 10 frameworks, from our candidates bottle won over flask over cherrypy.
• https://www.quora.com/Flask-vs-Bottle-what-are-their-advantages-and-disadvantages?share=1 links other answer, like the stackoverflow question. https://www.slant.co/versus/1398/1741/~flask_vs_bottle voting, but not much reasoning
• https://www.reddit.com/r/Python/comments/1u3sev/flask_vs_pyramid_which_do_you_recommend_for_a/
• http://codecondo.com/14-minimal-web-frameworks-for-python/ , 2014, more a list with short descriptions

Searching shows that there are not many reviews available for the less known options on the list. (So this step is concluded.)

[bernhardreiter]

Packaging status

well packaged: flask, bottle, cherrypy, pyramid somewhat packaged: circuits (on Debian/Ubuntu) not packaged yet: hug, morepath (checked CentOS 6-7, Debian Jessie, Ubuntu 14/16 LTS, OpenSuse 13)

[aaronkaplan]

@bernhardreiter we will look into this later .... 2.0. and most probably I'd like to see a set of different web interfaces / combineable . Not a completely monolithic system please. Anyway, assigned to 2.0.

[bernhardreiter]

After briefly comparing the candidates from the long list, here is my shortlist with rationale. (This concludes the step "C" from IRCA.)

Short list

1. hug

It seems to be the most simple solution that is sufficient. It has authentification and uses a python3 property (type annotations) to expose Python functions as HTTP RESTful url and command line interface. This seems to be a good fit for purposes within intelmq solutions as many functions will have to be there in a command line argument or terminal based administration.

Though mainly driven by one person and young, there seem to be enough contributers that hug makes the cut for being active and maintained. Its simplicity promises good design and good security properties. Its only 5500 SLOCs big, including test code and examples. The documentation makes a good and complete impression.

2. bottle

Bottle is also quite simple, though more complicated as hug and cannot easily use python3 specific improvements. It is popular, active with a large community, has a good documentation, many examples. It is well packaged.

3. morepath

Morepath looks promising, because its design is well reasoned and seems to have learned on experiences with Zope, grok and pyramid. Morepath seems to lean towards offering one good practice for solving challenges and it has a focus on RESTful design with stronger security by url routing (aka mapping) but with a model governing access. There is a lot of documentation which looks good. It is not well packaged yet.

followed by flask, pecan, circuits.web

The next group of candiates are not on the short list, but given here to show that flask, pecan and circuits.web are more interesting than pyramids and cherrypy. Why? cherrypy made an impression to be larger and bigger giving two many choices. Pyramids seems to give too many choices and if you want something like pyramids, morepath looks like a better alternative.

[bernhardreiter]

In the analysis step I'll try to write a small application with hug, exposing an intelmq-mailgen ticket as web and command line interface next.

[bernhardreiter]

Analysis hug

https://github.com/Intevation/intelmq-mailgen/tree/master/extras hug -f checkticket.py will serve an example mini-api to query the events that have been mailed out by one ticket.

Learned:

1. The hard part will be a good user presentation of an intelmq event, this is independent of the web-serving-framework
2. If we have the formatting function that has been used for the ticket# available, we can rerun it at the time where someone wants to see info about a ticket number. We would get a result that is close the send email. So there is no need to archive the full email.
3. hug has some interesting ideas where it is elegant, but it lacks some maturity. I've filed some reports to see how the community reacts. Examples:
   • no documentation how to setup, pool and teardown resources like a database connection https://github.com/timothycrosley/hug/issues/376
   • the cli and http "interface" exposure seems to break here and there, e.g. https://github.com/timothycrosley/hug/issues/399

[bernhardreiter]

Maybe something like https://github.com/dddomodossola/remi is good for prototyping.

[bernhardreiter]

So far I've build a hug/vuejs.org v2 Prototype to show event details for a ticket number in https://github.com/Intevation/intelmq-mailgen/tree/master/extras/checkticket-spa

In addition there is a https://github.com/dddomodossola/remi prototype in the same directory, based on events over a websocket connection.

@aaronkaplan told me that Cert.at plans to go forward with Django (or flask), because they can find people power for it more easily.

Overall my main lesson is that having the python backend functions and an idea how to present this to the users is more important than the serving and spa (single page application) logic technology.

Thus I'm stopping this issue for now. And we'll probably build prototypes using (serving technology) hug and (spa logic) vuejs.org v2 and then revisiting both elements based on more experience and ideas how to present this.

(I'm keeping the issue open, because I still think it makes sense to decide on a recommendation from intelmq itself and eventually move intelmq manager there.)

[bernhardreiter]

Lessons gained from experiments in #760 now led me in trying to get a more complete Single Page Application frame based on Vue2 next.

I'll try CoPilot because it seems most complete. The followup contender is https://vue-admin.fundon.me/#/, being more modular and lightweight, but less complete.

[bernhardreiter]

Authentication

Should probably be done with https://en.wikipedia.org/wiki/JSON_Web_Token (JWT, RFC7519, see reasoning at https://jwt.io/introduction/ ).

Python JWT server libraries

• https://github.com/jpadilla/pyjwt (supported and python3 ready)

[dmth]

Some thoughts on JWT at least three implementations exist for JWT in Python:

1. The aformentioned pyjwt https://github.com/jpadilla/pyjwt/
2. python-jose which is based on 1. but has more features than pyjwt https://github.com/mpdavis/python-jose
3. jwcrypto https://github.com/latchset/jwcrypto/

The first solution seems to be the most widely used one.

Caveats: When using JWT, be sure your lib is secure. There is something like a "None"-Hashing algorithm which overrides signature checking. Recent libs should not be vulnerable to this kind of Attack. https://auth0.com/blog/critical-vulnerabilities-in-json-web-token-libraries/

Support in hug: Not yet

[bernhardreiter]

Started https://github.com/Intevation/intelmq-fody which is based on Vue2, CoPilot and AdminLTE2.

[bernhardreiter]

https://github.com/Intevation/intelmq-fody now is a prototype for serving several events for a sent out ticket.

[bernhardreiter]

fody can be served by hug and mod_wsgi/apache, there is an example setup in the repo.

[bernhardreiter]

When working on an interface to the https://github.com/Intevation/intelmq/tree/certbund-contact database in https://github.com/Intevation/intelmq-fody/issues/13, I've seen that a RESTful Web service would usually use different URL patterns like /api/contact/15/edit and HATEOAS. hug does not offer examples for this and does not have a ready-made jwt module, so I've tried morepath. There is a prototype now in https://github.com/Intevation/intelmq-mailgen/tree/master/extras/checkticket-spa

Analysis morepath

https://morepath.readthedocs.io is well structured for a micro-framework. It uses models, basically python classes, to hand out data via several views. There is an jwt module.

Morepath's abilities to separate components and to produce link makes it suitable for products where a RESTful interface (with some HATEOAS) is really needed. Or where there are several views or operations on the same python objects. Thus morepath is very well suited serving APIs which are rich, will be used by several clients and offer more standardized interfaces (like for serving avatars or news-items).

Right now the aim of a certbund-contact contactdb interface is to stay close to how it is done in the database itself, there is not much processing to be done between the client and the database. Overall HATEOAS will make more general GUI interfaces, which are less usable for a special purpose.

As for intelmq we are just interested in offering an interface to intelmq itself via a single page webapplication we control a RESTful design seems to be a disadvantage for an intelqm interface.

conclusion -> trying further with hug

Meanwhile I've discovered that hug offers support for /api/contact/{contact_id} as well, looking at the added complexity of morepath, I'll continue with hug for now.

[aaronkaplan]

Please take a look at FastAPI: https://fastapi.tiangolo.com/tutorial/first-steps/

[bernhardreiter]

@aaronkaplan thanks for the hint, we will in https://github.com/certtools/intelmq-manager/issues/80 (Note that hug is used for fody-backend and the overall experience was good, its maintained, reliable and stable.)

[bernhardreiter]

Here are some general comparison reviews that feature FastAPI and competitors:

• https://steelkiwi.com/blog/top-10-python-web-frameworks-to-learn/ (interesting enough FastAPI ist not listed in the async section, and it had less stars and forks than hug or sanic. Also the use of typing is not mentioned.)
• https://geekflare.com/python-asynchronous-web-frameworks/ 2020-01-19 has competitors, too. Here FastAPI is considered almost big.

FastAPI is build upon https://www.starlette.io/ (which is an ASGI server that is considered beta by https://asgi.readthedocs.io/en/latest/implementations.html ) and Starlette will have to be run with either daphne or unicorn (or the beta hypercorn) all which have themselfs be run with an http<->https proxy to be secure.

FastAPI aims to create automatic docs for the APIs it is serving. It aims for OpenAPI, this I consider a drawback because API documentation should be for humans, not for machines in the base of an internal backend-frontend relationship anyway. So if you compare the OpenAPI examples with the documentation generated by hug, you see that they are much more verbose and harder to read for a human (you'd need another document generator to make them readable). The need and stated goal for OpenAPI seems to come out for REST, but REST is not a good match to create webapplications which serve human users. So the missmatch is understandable.

Reading https://fastapi.tiangolo.com/alternatives/ it can be seen that hug is close to FastAPI.

Strengths of hug/falcon/WSGI for IntelMQ (as backend for steering interfaces):

• More mature (stable revisions, even packaged in elder Debian/Ubuntu packages)
• Slightly lighter on the API documentation side and features (no OpenAPI, GraphQL)

  Strengths of FastAPI/Starlette/uvicorn/ASGI for IntelMQ (as backend for steering interfaces)

• Using more standard Python3 types for the annotations, allowing dynamic and static checking by external libraries like pydantic and mypy
• Lighter on the some features (no cli support)
• (Not relevant for internal backend for steering) potential to be faster because of asynchronous operation

[ghost]

More mature (stable revisions, even packaged in elder Debian/Ubuntu packages)

From a maintainer (and packagers) perspective this is a key argument.

[bernhardreiter]

As a note: https://github.com/certat/intelmq-webinput-csv uses Flask. Personally I believe that hug or FastAPI have advantages over Flask. And there is a need to gain experiences. With microframeworks, the dependency on a special framework is low.

[ghost]

See https://github.com/certtools/intelmq-manager/issues/80 / https://github.com/certtools/intelmq-manager/pull/197