Dashboard stops working properly after 50 000 hits

[kyyberi]

Description

I generated traffic to avoindata.net API with some scripts (10 pcs running in parallel generating >some thousand hits per hour). In the morning I noticed that charts are not shown, I have no idea of the real call amount (since all I see is 50 000 hits, which is not true).

[kyyberi]

In addition in the charts I only see Fri 14 even though I demanded to see 11.4.2017-14.4.2017. Expected: show the requested timeframe in the chart.

[kyyberi]

I took offline a few scripts and thus dropped the traffic amount. After that I was able to see chart with hour granularity.

This 50k limit seems a bit ridiculous. For example one of our prospect customers has 14 000 000 API calls / month. We must be able to show charts for high traffic APIs. Find a way.

[brylie]

We must be able to show charts for high traffic APIs. Find a way.

After discussing dashboard performance in issue #1340, we agreed to use Elasticsearch directly for dashboard data aggregation. @frenchbread ran several experiments with Elasticsearch aggregation which gave promising results.

The follow-up work means re-writing our dashboard code, including each chart and related data aggregation. We will continue the work, but have to see where it fits in the project roadmap/priorities (@bajiat, @ccsr), which currently includes Epic level work such as:

• REST API design/implementation
• Adding chatbots to APInf
• Developer experience onboarding process
• User notifications
• API Monitoring
• APInf as a global catalog harvester
• Rethink (i.e. rebuild) API Dashboard

[frenchbread]

This one: https://github.com/apinf/emq-analytics-dashboard/pull/7

It's for EMQ dashboard though, but we can use it as a reference. Moving towards using ES aggregated data on APInf dashboard means getting rid of data-table at some point.

[philippeluickx]

So basically we need to have a prioritization discussion. This has been on the table for way to long. I'd rather have no analytics than the current implementation. I consider this feature broken.

@bajiat @kyyberi would this get top priority over the other tasks? I've been +1 on this for almost a year now.

[brylie]

I'd rather have no analytics than the current implementation. I consider this feature broken.

That is a really strong statement with very little foundation. It doesn't really suggest any helpful way forward. Lets keep the conversation and feedback constructive.

[philippeluickx]

I am being constructive here and clearly own my statement. For me this feature is not done. It allows me to more easily prioritize it when looking at it this way.

As to the investigations done so far, I am very happy to see we are making progress to a workable solution. I would like to understand how come we did not push to the end and managed to close this issue? It's been on the radar for months.

[brylie]

I am being constructive here

By constructive, I mean 'suggest a way forward' rather than simply stating 'the current solution is inadequate and might be better to throw away'.

Some ways to give constructive feedback:

• Be focused. No hinting. No generalizations. Be specific. Use concrete examples
• Don't ask too much. Set realistic goals.
• Be a guide. Supply feedback and suggest a course of action

Inc. Magazine How to Give Constructive Feedback

I would like to understand how come we did not push to the end and managed to close this issue?

We have some process and prioritization challenges to overcome. For the past year we have been scrambling to keep up with incoming, Epic level, requirements from several sources. This is still ongoing, and perhaps growing stronger, as can be evidenced from our Roadmap above.

[marla-singer]

@preriasusi This bug is resolved in a new Dashboard and can be closed