Migrate Web Interface to Django

[jaegeral]

as mentioned in some other issues that should be something that should be done.

@kevthehermit did already some work, so if anyone wants to pick it up.

https://github.com/kevthehermit/ViperV2/tree/master/web

[kevthehermit]

There are some issues with the way sqlachemy interacts with Django and some of the tables.

It means that there are occasions where django tries to use a db object that sqlalchemy has already purged from its session. This results in 500 errors

I never found a nice way of handling this that didn't make some major changes to the DB class that i wasn't comfortable with.

I am using django heavily in other projects and so am unable to learn a new framework like flask. Im aware it has outgrown bottle and needs a lot of love.

[botherder]

Is anyone willing to invest some effort in getting this thing going? It has been stalled for a while. I am not able to invest the time at the moment.

[razuz]

@kevthehermit is this random or can I reproduce it somehow easily ?

[botherder]

I'm top posting this, cause we really need to replace the viper-web, which is very broken right now.

[frennkie]

I think I might be able to help a little with this. I'm completely new to Viper but I've been doing quite a lot with Python and also just did a small Django PoC recently.

I copied over the code from kevthehermit/ViperV2 (which I think it's already a very good base!) into a feature branch in my viper fork ((https://github.com/frennkie/viper/tree/django_web) and started looking into it.

I have some questions:

Python versions - which ones are and will be supported?

• I've already seen that @Rafiot is working on Python 3 support -> which is great!
• Is anything older than Python 2.7 "officially" supported (I hope not ;-) )

Code duplication

Looking through the API, Web and CLI code I got the impression that there is code (e.g. creating a list of projects) that exists multiple times at different locations (which is usually bad). So the question is what the design for the future should be.

[frennkie]

@razuz @kevthehermit I also encountered the 500 errors. They are reproducible on the main_page for the list of tags associated to a sample. As far as I understand it the issue is related to lazy loading of the tags and when I change session.query(Malware).all() to session.query(Malware).options(subqueryload(Malware.tag)).all() (here) it works.

[Rafiot]

I'm currently not trying to support (or even test) anything below 3.4 on new developments (python 2.7 must die), and consider python 2.7 support in viper on its way out. So yeah, anything older than 2.7 is dead.

On the web interface side, I don't use it, so I don't plan to invest time on it but if you're into that, please go for it, you have my full support :)

[frennkie]

I would be happy if I/we could entirely focus on 3.4 (and newer). But the question is what do Viper users expect and can/will they do the switch from Py2 to Py3.

If we think it's ok to release a 1.3 or 1.4 (or maybe 2.0) without any Python2 support.. then let's work on that...

[Rafiot]

If you start a website that is pure python3 and uses the modules (python2 and 3), it will, I think, be fine :)

@botherder what do you think?

[frennkie]

I have spent some time working on bringing the implementing of both the Web interface and the API in Django forward: https://github.com/frennkie/viper/tree/django_web_wip

Before I spent more time on this I would like to get some feedback on the "design choices" I have made:

1. API is mandatory If we implement the Web interface in Django (which is the title of this issue) then having a (REST) API is not optional (e.g. turn on when needed) but mandatory. RESTful access to the database is very useful for a Web interface (e.g. dynamically list, update, delete tags for a Sample).

2. API framework: Django REST framework + drf-nested-routers Django DRF is a very popular framework that enables a ton of functionality without having to write too much code yourself.

The first screenshot below is from my "Viper API v3" (with Django DRF) and the second "shows Swagger" (a useful API documentation tool) which would not be required for running Viper...

Update (2017-04-19): Extended API

[ph0sec]

Hi guys, I was going over above comments and decided to check with you all. Why are you proposing to go for Django? I'm not very big expert but IMHO it's pretty big framework for such project like viper. Why not to go with flask which is pretty versatile framework to work with. I've worked with it for the past year for the similar project in terms of design.

In addition if I may suggest - you can point other place to discuss it.

Viper can be slightly redesigned in the following way:

the core is based on RESTful api server (flask for example). All the heavy lifting and management is done here. viper-cli with the help of requests (as an example) will communicate with the core using the RESTful api. viper-web based on client side js and ajax to communicate with the core. The web ui could be based on whatever contemporary technology we chose (react, bootstrap , etc.).

So with such design everything will be concentrated in the core while ui could be what ever you want with much less duplication and etc.

These of cause could be further discussed

Good day.

[frennkie]

I think the question in general is very valid.

I agree that Django is a framework with a much larger scope and feature set than what is required for Viper. Additionally Django is best used with it's native ORM which is similar to SQLAlchemy - but not the same. When implementing the complete Viper DB API in Django DRF (which btw is completed for read access) I noticed that quite a bit of the DRY-ness of Django is lost to this fact.

But still I personally like working with Django. It has a huge user base which means there are lots of people familiar with it and it's heavily tested. Also the project is very well managed and dependable -Django 1.11 just came out and will be supported three years (LTS). Django also comes "batteries included"- which could be useful e.g. for user authentication and other things (imaging a centrally hosted Viper instance which several people access via the Web/API interface).

I totally agree that no "heavy lifting" should be done in the web site code.. those things belongs either in core or possibly a new API server.

In the end I think this is not so much a technical decision ("is Django better or Flask") but more a question of strategy and the future roadmap for Viper.

[Rafiot]

My 2 cents: I don't plan to use viper's web interface (I use MISP if I need a web interface). So the main thing I really care about regarding the web interface is to have people(s) maintaining it.

The technology behind it matters much less to me.

[botherder]

@frennkie I'm happy with Django and if you want to contribute your work, I would be very happy to take it in.

[frennkie]

@botherder great to hear that..

I'm pretty happy already with the new API implementation.. but on the new Web page I just encountered an issue with the Javascript calls in Safari (Chrome and Firefox are working).. looking into it now.

[frennkie]

Fixed that Safari issue (was pretty simple..)... The port to Django is surely not yet perfect.. but I'm willing to support it further (although it would be great if others would join in - at least for testing).

[acd62081]

@frennkie Is the django web interface project dead? I can't find it on your repository any longer. Just curious. I thought it was much nicer than the flask version. I have been using it and had very few issues for the basic things that I do. The issues that did come up were fairly simple fixes. BTW I am not very efficient with django or python but know enough to get into trouble.

[frennkie]

Nope.. definitely not dead.. more 'resting'.. ;-) https://github.com/frennkie/viper/tree/django

Any help with finding bugs and improvement suggestions are highly welcome!

[acd62081]

Great, Thanks! I will submit anything I find...

I noticed that in the manage.py file the web.web.settings needed to be changed to viper.web.settings in order to use the manage.py functions, like create a superuser.

[frennkie]

I noticed that in the manage.py file the web.web.settings needed to be changed to viper.web.settings in order to use the manage.py functions, like create a superuser.

Fixed, Thanks!