search

SeldonIO / seldon-server

Machine Learning Platform and Recommendation Engine built on Kubernetes
https://www.seldon.io/
Apache License 2.0
1.47k stars 299 forks source link

Seldon dashboard with hundreds of REST requests #29

Open beeva-enriqueotero opened 8 years ago

beeva-enriqueotero commented 8 years ago

Seldon grafana dashboard shows every API request for a client. With Recommendation REST API each user generates a different entry in charts. So after hundreds of users the chart legend is huge and the web interface hangs up.

Regards, and thank you Enrique

ukclivecox commented 8 years ago

Hi,

The dashboard for recommendations should aggregate by request type as shown in http://docs.seldon.io/img/grafana.png Do you have the version of Seldon you are running and perhaps some example or screenshot to help us debug this further?

beeva-enriqueotero commented 8 years ago

Hi,

The dashboard for recommendations aggregates by request type based on url. The problem with REST API is that /users/1/recommendations and /users/10/recommendations, for instance, generate different request types.

I tested this behaviour with Seldon Server 1.3.11

ukclivecox commented 8 years ago

Yes, this will need a fix to the kafka-streams job that aggregates the stats and sends them to influxdb. We will need to add to it knowledge of the REST endpoint URL structures so the user can be removed from the request. An alternative would be to change the influxdb (via grafana) SELECT statements if possible, but I'm not sure of the flexibility of the influxdb query langauge to accomplish this.

beeva-enriqueotero commented 8 years ago

I don't know much about influxdb. But I don't find in its documentation any string function. So it doesn't seem easy to be done only by changing SELECT queries.

seldondev commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. /lifecycle stale