Call/message usage reporting service for community edition

[ivelin]

Related to #283

Related IRC discussion:

'' gvag: You mean the service that will collect the stats? [10:25am] gvag: I think we have to go with Graylog for that, or a similar application [10:26am] ivelin: yes, the cloud service [10:27am] ivelin: Graylog is the tool, but I’m also thinking about the live service [10:27am] gvag: I think Graylog, which is based on ElasticSearch but provides many nice features, is perfect for what we need [10:27am] gvag: Can you elaborate on what you mean live service? [10:28am] ivelin: something like stats.restcomm.com where we show a worldwide map of monthyl, weekly calls/messages served by restcomm [10:28am] ivelin: anonymous global usage data [10:30am] charles-r: ivelin, wouldn't some people object to us collecting data about their running Restcomm instance? [10:31am] gvag: I am not sure about this service and what will take to build it. We can discuss it with Alex or Orestis for them to build something on AngularJS. Because I guess we don't want to have Graylog in public, unless if we can customize it completely and we provide our own logo etc [10:31am] ivelin: charles-r , yes its optional [10:32am] abhayani: we have more challenges here. Most of the enterprise doesn't like direct internet connection to their servers [10:32am] abhayani: so may be an alternative could be to periodically print in logs [10:33am] abhayani: and as and when net is available push only these lines from logs to central server [10:33am] abhayani: just thinking about various possibilities [10:33am] abhayani: as we want to have same for other legacy products too [10:35am] charles-r: We could start by showing number of downloads and from how many countries in the world. [10:35am] abhayani: gvag, I am thinking if this should be separate project which can reused by any product [10:35am] gvag: abhayani, the monitoring services provided by Restcomm is a feature for enterprise monitoring, so customers can run their own elasticsearch or graylog or whatever monitoring server, to collect stats from Restcomm. On the side, we will collect anonymous call data for a statistics [10:36am] gvag: abhayani, what do you have in mind for that? The monitoring service, checks for Restcomm live calls by receiving call events etc, this is something very specific to Restcomm. [10:37am] gvag: These stats are exposed to any monitoring tool by the REST API, very Restcomm specific also [10:37am] gvag: I don't see any other way that can be used in other projects [10:38am] ivelin: the problem with downloads is that its hard to monitor nowadays [10:38am] ivelin: between torrents and other sources, its hard to tell [10:39am] ivelin: but call/message stats is something that I think will be more representative [10:39am] ivelin: despite all the limitations you bring up. [10:39am] ivelin: yes, some machines are behind VPN without any internat access [10:39am] ivelin: but most are online [10:39am] ivelin: we aren’t looking for a perfectly accurate number [10:39am] ivelin: just a ballpark representation [10:40am] ivelin: Do we handle tens or hundreds of millions of messages / calls per day ? [10:41am] ivelin: let me open a new issue for 7.4 [10:41am] abhayani: for legacy we have like 100's of millions of SMS per day [10:42am] ivelin: yes, so then overall do we handle 100’s of millions or billions? ''

[rlimonta]

@deruelle Hello Jean, I'm sorry for the delay, I had a little contretemps in the last days. I need just finish the dashboards charts, and I'll finish today.

[rlimonta]

@deruelle I finished the implementation of the web module. Today I will publish into the OpenShift.

I tried to commit the cliente module into commons folder, but I'm not allowed to create sub-folders. You could create a folder called statistics?

I'm sending some screenshots from web module.

Best regards.

[deruelle]

this look great @rlimonta ! Can you do a pull request on the commons project ? That's how we accept new code usually and allow us to do peer reviews as described in the Open Source Playbook.

Is the data shown in the graph available through a REST API ? If multiple VMs are running with the commons stats module activated I assume the server side is concatenating results from all VM over time as well right ?

[rlimonta]

@deruelle I committed the statistics-service module into repository successfully. About the commons module, I can't make a pull request. I need select one source branch first.

About your questions: "Is the data shown in the graph available through a REST API ?" Yes, it's totally possible access through a REST API, I can implement the @GET methods.

"If multiple VMs are running with the commons stats module activated I assume the server side is concatenating results from all VM over time as well right ?" The system concatenating results from all VMs and grouping by timestamp and key.

Eg. VM1 sent: { "count" : 4560, "key" : "sip-requests", "timestamp" : NumberLong(1467920534273) } VM2 sent: { "count" : 5350, "key" : "sip-requests", "timestamp" : NumberLong(1467920534273) }

The chart will display count: 9910 key: sip-request at timestamp: 1467920534273.

One important thing is about the key. The Key is dynamic, and we have possibility to create new keys automatically at the client side.

Eg. Client Side Counter counter = metrics.counter("sip-request");

[deruelle]

@rlimonta did you follow the procedure at https://docs.google.com/document/d/1RZz2nd2ivCK_rg1vKX9ansgNF6NpK_PZl81GxZ2MSnM/edit#heading=h.mcjitemt6ng1 for creating your own fork, committing to it and then doing a Pull Request from it ?

Yes, it's totally possible access through a REST API, I can implement the @GET methods.

Great, let's do it

Great on concatenation.

Did you write some tutorial on how to deploy the server side to OpenShift by example ?

[rlimonta]

@deruelle Thank you for your response, now I got it :)

About the OpenShift, we can do in two ways:

01) I can create a environment associated with my account.

02) You can create an environment associated with a project account. We need Wildfly 10 and MongoDB.

One important thing is about the hostname. The client module sends messages to http://stats.restcomm.com:8080/statistics/rest/.

This address is in a properties file (config.properties) and can be changed if necessary.

We need to define the service hostname.

[deruelle]

@rlimonta let me try to create my own environment on OpenShift to test the documentation. Can you create a tutorial in asciidoc format push it to the statistics-service repo ?

[rlimonta]

@deruelle Yes, I will work on documentation and @GET methods.

[deruelle]

Thanks @rlimonta. Looking forward to set it up and start testing

[zahidMed]

@deruelle I think it's too late for proposal. But I think that stats should be devided into differents layers (ObjectType) each ObjectType contains counters according to the nature of this layer (number of success requests, nbr of failure requests, latency ...). kpi4j was developed for that purpose and can be used in all the modules. the ObjectType activation is done through the config file and I can adapt it to your need. https://github.com/zahidMed/kpi4j

[rlimonta]

@deruelle I finished the implementation of @GET methods and improved the dashboard graphics. I created a single chart by metric.

Now I need to finish the Installation Guide. I have all screenshots and need to finish the doc.

[deruelle]

@rlimonta looking better by the day. Looking forward to the doc to set it up !

[rlimonta]

@deruelle In the last three days I had problems accessing OpenShift environment.

I do not know if this is true for all plans or only for free instances.

What do you think of using another alternative like Doker, AWS, etc ?

I received this message:

[deruelle]

It seems to work on my side. But I'm fine to have a Docker guide to start with.

[rlimonta]

Ok.

[deruelle]

@rlimonta any news on this yet ?

[deruelle]

@rlimonta were you able to wrap this up by any chance ?

[rlimonta]

@deruelle I apologize for the delay, but I had to travel to work in the last two weeks. I need only finalize the document. I'll work on it and send you until the weekend. The OpenShift environment is running fine now!

[rlimonta]

@deruelle I wrote the document README.md explaining the deployment process into OpenShift environment. If you have any comments or suggestions, please feel free to let me know.

[deruelle]

Thanks @rlimonta. I just returned from vacations. Let me check that out.

[rlimonta]

@deruelle Ok Jean, feel free to any suggestion.

[deruelle]

Thanks @rlimonta I was able to get it up and running and push some test data to it but I see that when I try to change the date, the end date doesn't seem to be taken into account or something else

Also I tried the GET API on http://statistics.restcomm.com/rest/counter/1471167956/1471945556/sip-invites but it didn't return anything

Shall we protect the GET API under the same username and pwd as the UI by the way ?

[deruelle]

Also restarting the OpenShift application led to loosing all the information. It seems counters are back to 0.

[deruelle]

@rlimonta

Also I tried the GET API on http://statistics.restcomm.com/rest/counter/1471167956/1471945556/sip-invites but it didn't return anything

actually I just noticed the timestamp is in seconds so using http://statistics.restcomm.com/rest/counter/1471167956000/1471945556000/sip-invites made it work.

Is there a Rest API for having a concatenated aggregated number of the counters over the period ?

[rlimonta]

@deruelle Jean, I made a test on my environment and is working properly.

I have one suggestion for you simmulate quickly.

01) Change your Remote Server on StatsCounterTest class, like bellow:

I recovered the indicators through the dashboard and rest api:

I accessed my environment via ssh, and the data are saved on MongoDB.

I'm sending the address of my test environment: http://rlimonta-restcommstats.rhcloud.com/ user: restcomm pass: restcomm123

If you want, I can review your environment with you.

[rlimonta]

@deruelle Jean, about your question:

"Is there a Rest API for having a concatenated aggregated number of the counters over the period ?"

The answer is no, but I can implement that. It's simple...

Best regards.

[deruelle]

Is there a Rest API for having a concatenated aggregated number of the counters over the period ?

https://github.com/RestComm/statistics-service/issues/1 done ;)

[rlimonta]

@deruelle :)

[deruelle]

@rlimonta on the dashboard, it works for the current day but if I try to see for a longer period of time the dashboard doesn't seem to display well for some reason. Did you try displaying values for counters inserted like a week before ?

[rlimonta]

@deruelle This may be happening because of the default grouping in the chart is HH:mm. When you increase the period, it generates a lot of points to zero.

I will improve to support (month, day, hour, minute).

[deruelle]

@rlimonta also the UI seems broken under Firefox. Chrome doesn't display well as it does in your case either on Linux.

[rlimonta]

@deruelle I'll check it out today.

[rlimonta]

@deruelle I implemented the following items: 01) Firefox Issue: Fixed 02) Dashbord Layout Improvement: Fixed 03) Issue #1: Closed.

[deruelle]

Thanks @rlimonta ! Confirmed it works fine now !