search

certtools / intelmq-manager

IntelMQ Manager is a graphical interface to manage configurations for IntelMQ framework.
https://docs.intelmq.org/latest/user/manager/
103 stars 56 forks source link

Use python3 instead of php. #80

Closed bernhardreiter closed 3 years ago

bernhardreiter commented 8 years ago

Using python3 to serve the manager would get rid of a major dependency ("php"). Given that large frameworks always has defects it is desireable to remove php as python3 is already coming with intelqm.

Technically: Should be a reasonable effort because most stuff happens in Javascript anyway. Would need a python3 webframe work to move, which needs to be selected. (E.g. from stuff like Django to flask and bottle.)

aaronkaplan commented 8 years ago

On 30 Jun 2016, at 15:51, Bernhard E. Reiter notifications@github.com wrote:

Using python3 to serve the manager would get rid of a major dependency ("php"). Given that large frameworks always has defects it is desireable to remove php as python3 is already coming with intelqm.

Technically: Should be a reasonable effort because most stuff happens in Javascript anyway. Would need a python3 webframe work to move, which needs to be selected. (E.g. from stuff like Django to flask and bottle.)

For whatever it's worth, I prefer small light frameworks such as flask. Django is quite an overkill for this type of task IMHO.

My 2 cents, a.

SYNchroACK commented 8 years ago

@aaronkaplan you're 100% right IMHO.

+1

bernhardreiter commented 8 years ago

The decision for a web framework will have an effect on several intelmq components, so it should be taken centrally. I've created https://github.com/certtools/intelmq/issues/708 for it.

bernhardreiter commented 4 years ago

When fixing #111 is makes sense to also switch from php to python3. As https://github.com/Intevation/intelmq-fody-backend is working nicely with https://www.hug.rest/ this is the first choice for a micro-framework to use as there is some synergy. If there is a need to switch to a different one like falcon it would be much less efforts.

Rough plan:

Is is allowed to change the endpoints and their parameters if researching shows that something can be done better, because the serving and html pages and javascript parts are delivered together and considered internal implementation detail.

aaronkaplan commented 4 years ago

I don't quite agree and I am also not sure if it fits with future architectures. Please also take a look at fastAPI.

bernhardreiter commented 4 years ago

@aaronkaplan thanks for commenting!

Can you be more specific about which parts you do not agree? So far I understand that the choice of the python3 microframework maybe suboptimal.

Choice of microframework hug versus fastAPI

We'll take a look at FastAPI.

Explanation of the current suggestion hug: We did no new reevaluation of the progress of Python JSON Https service microframeworks, because a) It is not the main part of the work and not a strong dependency b) There is some synergy, which makes the work less expensive and both easier to operate when using hug for both fody-backend and intelmq-manager (backend). Otherwise we look at replacing fody-backend microframework as well. c) there are no special highspeed requirements of the intelmq-manager backend, the performance problem is more with how intelmq itself is able to come up with the necessary infos.

Small hint: fastapi itself lists hug as a major influence and good step forward, so it is not bad in the eyes of the fastapi developers. :) https://github.com/certtools/intelmq-manager/issues/80#issuecomment-594776863 (The question will be, are the new methods, specially some of the schema works in the past was not always fitting micro approaches, especially not as we are not trying to do an api in the string sence, but an internal data exchange.)

bernhardreiter commented 4 years ago

In https://github.com/certtools/intelmq/issues/708#issuecomment-595779283 there is a look about the strength and weaknesses of FastAPI versus hug as general web framework choice.

In conclusion FastAPI has the potential to surpass hug, because it takes more modern approaches. Currently it is not as mature and has some unneeded complexity for an internal interface (which is the main topic in this issue).

So there is some more work in using FastAPI, here for the deployment:

Deployment

FastAPI is newer, it needs an ASGI server and those again need an TLS capable HTTP server/proxy. Some of the dependencies are not considered production stable (yet).

To check the packaging level in Debian (as example):

For hug there are packages (though python-hug is not yet in Debian Stable):

Conclusion

The microframework choice is not the main focus of this issue. Both FastAPI and hug would work. FastAPI has more future potential, but for the time being causes more work on people that would like to deploy it.

We from Intevation would be fine to do either variant, so it is up to you @aaronkaplan, @wagner-certat and our customers to decide what you prefer.

pettai commented 4 years ago

I don't quite agree and I am also not sure if it fits with future architectures.

@aaronkaplan could you please be more specific on why hug et al isn't fit?

bernhardreiter commented 4 years ago

Sunet contracted us (from Intevation) do resolve this issue, timeframe is within two month.

ghost commented 4 years ago

The main problem I see is that we have multiple Python-based web applications in IntelMQ. Yet it's fody and and webinput, now the manager gets converted too. That will make the set-up procedure much harder , it's already a pain with only fody and webinput. Maybe the manager could act as an proxy/relay here to keep/make it manageable.

bernhardreiter commented 4 years ago

From what I've read and the existance of the issue itself, we agree that moving away from php is a good thing. :)

Proliferation of python3 microframeworks is a possible negative aspect. However gaining experience is a plus.

If we move to hug, it is one technology less and we will find something to see if we keep the urls between the manager and fody and other apis separated.

Moving to FastAPI would give us more experience with a potentially raising microserving technology, so we can make a good recommendation for https://github.com/certtools/intelmq/issues/708 one day. (Here we also find something to keep the URL parts separated.)

@aaronkaplan What is your opinion now, that you've read the various cons and pros?

bernhard-herzog commented 4 years ago

I'm going to start working on this. In order to keep the effort focused the broad strategy is to limit the change to replacing PHP with Python and keep everything else the same as much as possible.

I'm going to to use hug as framework for reasons already mentioned in other comments in this issues: we (Intevation) already use it in fody and it's easy to install on at least one platform we care about (Ubuntu). I'm going to keep most of the code independent of the framework so that it should be reasonably easy to use a different framework at some later point.

aaronkaplan commented 4 years ago

Hi @bernhard-herzog , @bernhardreiter ,

so... I'd like to schedule a quick call with you regarding the IntelMQ 3.0 architecture. First of all, to get your feedback and secondly on how you can integrate that with hug.

ok?

In principle, for the next version I don't care which RESTful API framework you want to use. The only thing is that I would like to make sure we support the same API endpoints as future IntelMQ bots. So, since this is a longer discussion , I would recommend to sync via a jitsi (or whatever) call.

Could you please mail me a few suggestions when you would have time (I can only do afternoons Mo-Thursday, FR the whole day).

Thanks + Best, a.

bernhard-herzog commented 4 years ago

I've now actually started on this with a bit of prototyping of a solution. So far I've got a simple hug-based API that I use as a replacement for controller.php. The HTML for the management interface is still served by PHP but all API requests that would go to controller.php now go to my API prototype. That prototype can only handle a few simple queries, such as ?scope=check and ?scope=queues but those are handled all the way via sudo to intelmqctl and back to the JS in the browser.

bernhardreiter commented 4 years ago

About the IntelMQ 3 architecture and the small mumble conf: We've talked about https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md and it does not affect this point, as the timeline is much longer and it is all in the idea and feedback collection phase. So all green light for getting this done quickly and with hug3. :)

@bernhard-herzog plans to have our prototype in a branch within the next days (when get around doing it).

aaronkaplan commented 4 years ago

On 21.04.2020, at 12:49, Bernhard E. Reiter notifications@github.com wrote:

About the IntelMQ 3 architecture and the small mumble conf: We've talked about https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architecture-3.0.md and it does not affect this point, as the timeline is much longer and it is all in the idea and feedback collection phase. So all green light for getting this done quickly and with hug3. :)

@bernhard-herzog plans to have our prototype in a branch within the next days (when get around doing it).

confirmed.

bernhard-herzog commented 4 years ago

I've just pushed the first part of my implementation to https://github.com/Intevation/intelmq-manager/tree/python-backend

bernhardreiter commented 4 years ago

technical considerations

Static parts for HTML, javascript, css for the single page web application could also be served outside of hug3 by a webserver, and only optionally like https://github.com/hugapi/hug/blob/develop/examples/static_serve.py . So preparation of the spa could happen as pre-procession. However I'm not sure if we get this done as part of this issue.

bernhard-herzog commented 4 years ago

Python-backend is feature complete

I've finished the port of the PHP backend and can now run IntelMQ-Manager without PHP.

The main thing left to do is installation and setup. This includes the usual python packaging things like setup.py but also corresponding documentation. I plan to do this this week, at least enough of it so that other people can more easily test this.

One thing that is missing is a timeout for the intelmqctl subprocesses. This is not easy to implement due to sudo being involved, and doesn't seem to work in the PHP version either (#195).

ghost commented 4 years ago

Wow, that is great news!

Could the timeout maybe easier be implemented in intelmqctl itself?

bernhard-herzog commented 4 years ago

Could the timeout maybe easier be implemented in intelmqctl itself?

I've replied to this in https://github.com/certtools/intelmq-manager/issues/195#issuecomment-625171241

bernhard-herzog commented 4 years ago

I've updated docs/INSTALL-DEV in our python-backend branch so that other developers can easily try it out.

bernhard-herzog commented 4 years ago

Some of next steps:

Feed-back

Has anybody tried it? Did it work? What do we need to change?

Code organisation

The Python code is in the intelmq_manager subdirectory. The PHP code in intelmq-manager has been removed, but the static files, like style sheets, images, JavaScript files, etc. is still there. This at least looks a little strange (only one character difference in the names, the '_' was necessary so that the directory name could be a package name).

Possible solutions:

Documentation

In particular, the installation instructions obviously have to be updated. The current instructions cover quite a range of systems. Do we still support all of them, particularly the older ones?

I've developed the python port on Ubuntu 18.04LTS, so the new dependencies, hug and mako, don't have to be particularly new (the minimum versions given in the setup.py file are those of 18.04). However, I don't know whether older versions of hug would also work. Older versions of mako should be fine as the templates only use features that have been in mako for a very long time

ghost commented 4 years ago

Has anybody tried it? Did it work? What do we need to change?

Works for me! I tried it on our development instance and haven't seen any bug yet :)

Possible solutions:

* rename `intelmq-manager` to `static`

* rename `intelmq-manager` to `intelmq_manager/static`

Do you prefer one over the other? I'm fine with both variants.

In particular, the installation instructions obviously have to be updated. The current instructions cover quite a range of systems. Do we still support all of them, particularly the older ones?

No, as 18.04 works, that's fine. For example we do not need to support Debian 8 and Ubuntu <= 16.04. I can add instructions for Debian, CentOS and openSUSE myself.

I've developed the python port on Ubuntu 18.04LTS, so the new dependencies, hug and mako, don't have to be particularly new (the minimum versions given in the setup.py file are those of 18.04).

:+1:

bernhardreiter commented 4 years ago

Personally my tendency is renaming to 'static' to separate the python code module more clearly.

bernhard-herzog commented 4 years ago

Has anybody tried it? Did it work? What do we need to change?

Works for me! I tried it on our development instance and haven't seen any bug yet :)

Great!

Possible solutions:

* rename `intelmq-manager` to `static`

* rename `intelmq-manager` to `intelmq_manager/static`

Do you prefer one over the other? I'm fine with both variants.

After looking into this a bit, I went with intelmq_manager/static because that fits Python's setuptools/distutils better. Putting data files into the Python package is the way these tools prefer.

bernhardreiter commented 4 years ago

My test was successful, I did an overview test, covering:

hints for improvement and packaging

positions.conf

This needs a writable manager/positions.conf file, even if it just an empty one and the documentation 

# if you use the intelmq manager (adapt the webservers' group if needed):
chown intelmq.www-data /opt/intelmq/etc/*.conf

from intelmq installation instructions https://github.com/certtools/intelmq/blob/develop/docs/Developers-Guide.md#development-environment may not be enough to ensure this, so packageing might. Maybe a solution is to have the manager be able to create the file if it does not exist (which we can put in a new issue).

bernhard-herzog commented 4 years ago

The actual porting is done. What's missing is documentation and packaging updates. It seems to me that that might be easier after this is merged, because then others can more easily contribute. I've therefore created a pull request now.

bernhardreiter commented 3 years ago

If the plan is to release IntelMQ Manager 2.3.0 with the new python backend module (https://github.com/certtools/intelmq-api) and because the documentation and packaging there is done (according to the message). It is only the release that is missing, isn't it?

(So basically we could close here, because Issues are closed when they are in the code and not when they are release?)

ghost commented 3 years ago

Yes, we can close here. Before doing the release just we need more testing

bernhardreiter commented 3 years ago

What kind of testing would be needed? (Just asking, if it is known, someone could estimate if they could help.)

ghost commented 3 years ago

What kind of testing would be needed? (Just asking, if it is known, someone could estimate if they could help.)

Just using it in practice to check if there are some glitches. We need to do that internally and I intend to write a mail to the mailinglists as well to announce this major change (and ask for testing it & feedback).