Webportal development

[severinsimko]

There are 2 main Sections: Cluster and Applications

Cluster:

1. Start Cluster

• If the cluster is running -> Alert that the cluster is already running

• If the cluster is NOT running -> after clicking on the "start cluster" button, the progress elements will change the color to green, once the execution of the script is done then the elements will change the color for each node, red - node is not running, green - node is running

• [x] Need to change the output of the REST API script for starting the cluster, it needs to return status for each node - the output will be then parsed in the frontend code

2. Stop Cluster

• If the cluster is already not running -> Alert that the cluster is already not running

• If the cluster is running -> execute the script to stop the cluster and according to the given information change the color of progress elements

Applications

Every application is started on a particular port Protocols Statistics on 4051 Traffic Profiles on 4050 At the moment, this particular port is only set in the REST API script

• [x] Port needs to be given as a parameter from the frontend, it will be a config. file with applications and their ports - the frontend code will then read this config from a file

• [x] The algorithm for getting the statistical information about the applications NEEDS to be more effective

• [x] I need to create global functions, for functions that are repeatedly used.

DB

The frontend needs to periodically execute and get the statistical information about applications and store them in the DB.

On which server is the DB running? Don't know yet how to test this on my computer. @tomjirsa

• [x] In models/db.py need to add the table structure for statistical info about apps

What to do with the data in DB when the application is stopped? Do we need to delete all of them ? @tomjirsa

Application overview

• [x] Once the DB is running, a graph of the application performance should be implemented.

[tomjirsa]

On which server is the DB running? Don't know yet how to test this on my computer.

The DB is running on the same server as web (is part of web2py framework). Use database of the web application stream4flow.sqlite

What to do with the data in DB when the application is stopped? Do we need to delete all of them?

Yes, after an application is stopped, the data in DB should be deleted.

[tomjirsa]

For overview - please check, when the functionality is ready:

Cluster:

• [x] Start Cluster
• [x] Stop Cluster

Applications

• [ ] Start Application
• [x] Stop Application
• [ ] Application Status

[tomjirsa]

Adding @cermmik for following the issue and discussing the subject.

[severinsimko]

Progress bars for Start and Stop cluster implemented - the "cluster" section is done, only design changes are needed.

Design changes planned after the implementation

[severinsimko]

Nestretli ste sa pocas vyvoja s podobnou chybou?

[cermmik]

Unfortunately, no. Have you tried to download a new database file?

[severinsimko]

I did, it solved the problem but, on the other hand, it looks like I'm not able to define the new table in the DB.

I added db.define_table('dbtest', Field('application_id', 'string', required=True, notnull=True), Field('application_ave', 'integer', required=False, notnull=False) ) in the /models/db.py

And then, I tried to insert some values into this table. and it says that "no such table exists". Multiple restarts of the application didn't have any effect on it.

[cermmik]

It is necessary to update the file directly. Use sqlite3 (Linux) or SQLite Adminsitrator (Windows).

[severinsimko]

Thank you, problem solved!

[severinsimko]

Backend scripts modified that it runs everything under the "spark" user instead of "root"

[severinsimko]

Graph/Detail for a particular application is implemented. Kill application + database data deletion implemented. Running multiple application with the same name, but with different #cores and #memory is implemented.

[severinsimko]

I'm going to work on following tasks now:

1. Adding and running newly created applications
2. Need to reimplement the cluster stop/start output
3. I would like to add more configuration and monitoring options to the web portal - SNMP status, REST status

4. Still having performance issues - need to fix

   Tasks:

• [x] Newly added applications

• [x] Cluster stop/start output

• [x] More monitoring options

• [ ] Performance issues - fixes

[tomjirsa]

Good job.

3, I would like to add more configuration and monitoring options to the web portal - SNMP status, REST status

For the overall monitoring, we plan to use Graphana and Graphite, or some other monitoring tool. This task is going to be assigned to to @simkosvorad .

[severinsimko]

The option to run random application was implemented.

Summary: When you have a new application, there are only 2 things needed to do in order to be able to run the application through the web interface. First, you have to put the application on MASTER server ( on whatever path you want). Secondly, you have to add the FULL PATH through the web interface. When you do this, you are able to run this application.

You also have the option to delete the application if you want. So you have full control over the applications that you can or you want to run.

I added new DB table: application_available(id, application_name), application_name is in this case FULL PATH to the application.

In the web interface it looks like this:

The design is still missing, but it will be fixed after tests.

[severinsimko]

I can suggest some attributes/services/ports that we need to monitor and that are related to the web interface. @tomjirsa @simkosvorad

[severinsimko]

what is coming next:

• [x] Sliding window for application detail graph

• [ ] Progress bar when the application loads the data from Spark

• [x] Parameters/Atributes about ZK for application run

• [x] Script for automatic fetching and filling the DB with the application data ( each 30 sec) located in crontab

[severinsimko]

Sliding window/ application detail graph set for last 2 hours.

I added zookeeper option for application run.

The sqllite DB is now installed on the web server .14.55 in /opt/dbinsert + I dumped the db on this server. Now, I'm going to work on the script that will fill the db - the function for filling the db with data will be different from the one I use now - that's why it takes more time to implement it

[tomjirsa]

Thanks for info. Please sync with @cermmik the date of meeting for branch merge in advance.

[severinsimko]

I created the script that will fill the DB with data, it's located on web server in : /opt/dbinsert/, the script is called insert.py.

I added the record to crontab on this server, this record is still commented out, it's ready for final integration

[tomjirsa]

I have tested portal according to the manual (I have to install appconfig python module) and following error occurs when loading home:

Traceback (most recent call last):
  File "/var/www/web2py/gluon/restricted.py", line 227, in restricted
    exec ccode in environment
  File "/var/www/web2py/applications/Stream4Flow/models/db.py", line 18, in <module>
    myconf = AppConfig(reload=True)
TypeError: __init__() got an unexpected keyword argument 'reload'

The current deployment is on server from cesnet

[severinsimko]

Actually, this line "myconf = AppConfig(reload=True)" was in this configuration (db.py) file even before. Please try to ask @cermmik whether this option is required and what is the purpose for this option.

Or just try to use: myconf = AppConfig(), instead of what's there right now.

[cermmik]

In ./models/db.py is loaded following code with a wrong module:

# -------------------------------------------------------------------------
# app configuration made easy. Look inside private/appconfig.ini
# -------------------------------------------------------------------------
from appconfig import AppConfig

The original file uses following module that works well:

# -------------------------------------------------------------------------
# app configuration made easy. Look inside private/appconfig.ini
# -------------------------------------------------------------------------
from gluon.contrib.appconfig import AppConfig

Is there any reason for the first solution?

[cermmik]

There is no free capacity to finish this task.