Improve the option: statistics - (to fix web)

[ediyes]

Hey @Rub21 cuándo van a mejorar las opciones del to fix de la web, específicamente la opción Statistics la información que nuestra es incorrecto, por ejemplo en la tarea del crossingminor en el slack con 532 tareas pero al corregir pasa más de 600 tareas. :thinking:

En otros caso por ejemplo en la tarea de unconnectedmajorhighways no permite ver las estadísticas. @Rub21 @ajithranka Necesitamos obtener la información de las estadíticas para poder comentar cada semana en los master ticket de cada tarea, por favor sería bueno que retomen y mejoren el to fix web.

Otro error es cuando las tareas estan completadas no pasan a la lista de Completed Tasks muestra como si estuviera incompleta pero ya no descarga ni un error más para corregir.

[ajithranka]

I've hidden the stats graph from to-fix until we fix it. I'm checking now why the numbers seem off.

@Rub21 @ediyes

[geohacker]

Great observations @ediyes! @Rub21 @ajithranka - how does this sound?

• Can we rename type of the project during every update to version. Rather than creating an arbitrary ID for each update of the project, can we use version numbers like v1, v2?
• For every activity of a project, can we store the action, project, version, timestamp, user.
• Can we then build an end point for stats that can:
  1. /project_name/stats?version=v1 - will give activity for v1 tasks of that project. grouped by users/time.
  2. /project_name/stats?from=timestamp&to=timestamp - will give all activities for the current version.
  3. /project_name/stats?version=v1&from=timestamp&to=timestamp - will give all activities for v1 of the project between that time frame

[Rub21]

@geohacker . I think you mean task instead of project .

Can we rename type of the project during every update to version. Rather than creating an arbitrary ID for each update of the project, can we use version numbers like v1, v2?

sounds fine!!!

For every activity of a project, can we store the action, project, version, timestamp, user.

currently, we are storing

{
time: 1486638122,
key: "c89eadcca79b235ccfe1c0da5c393c608f487710",
action: "edit",
editor: "id",
user: "Amisha Singla"
},

do we need to store project, version ? i am not sure how valuable will be that.

• Can we then build an end point for stats that can:
  1. /project_name/stats?version=v1 - will give activity for v1 tasks of that project. grouped by users/time.
  2. /project_name/stats?from=timestamp&to=timestamp - will give all activities for the current version.
  3. /project_name/stats?version=v1&from=timestamp&to=timestamp - will give all activities for v1 of the project between that time frame

This would be a bit slow because to count, we need to request all the activity and then count in the backend, there is no way to make complex queries in ES, Also to request all the activity is done chunk by chunk and it takes time.

In the past, I was working that way, but the statistics sometimes did not load, and it was slow. For that reason I've changed the format to count by day, we do not need to make complex queries to account only make a request to a type to get this data. Now it works well.

I do not know why it separates the statistic by version, we already have the accounting for each update, example.

All the activity and stats are handled in differents type. but they are storing all the log in each update.

[geohacker]

do we need to store project, version ? i am not sure how valuable will be that.

Yes I meant task. @Rub21 what is the key here? We will have to store the task and the version for each activity. If we aren't, then how do we get stats for each update?

This would be a bit slow because to count, we need to request all the activity and then count in the backend

@Rub21 can you say more? To me this is just a slight change from how it working right now. We can do all this with a single elastic search search query.

[Rub21]

@geohacker

what is the key here? key it is the has of the feature in a featureCollection. We will have to store the task and the version for each activity. If we aren't, then how do we get stats for each update?

don't get you here?

@Rub21 can you say more? To me this is just a slight change from how it working right now. We can do all this with a single elastic search search query.

Can you try it?, I've created the stats from the activity, but it worked slow, to count the items in a type in not like a table, you can request all items at once in a type which has a few items, but when the type contain a lot of items it is quite hard, and you should do many requests, even when you want count according to some parameters you should do more that one request.

Can we rename type of the project during every update to version. Rather than creating an arbitrary ID for each update of the project, can we use version numbers like v1, v2?

this is done, can you check here https://build-to-fix-production.mapbox.com/tasks

[ajithranka]

Update: The stats endpoint now shows day-wise number of edits made by each user. This needs to be aggregated on the frontend to display user stats within a date range.

https://build-to-fix-production.mapbox.com/tasks/strangelayer/track_stats/from:2016-02-15/to:2017-02-15

...

"stats": [{
  "start": 1484744400,
  "edit": 7,
  "skip": 0,
  "fixed": 0,
  "noterror": 1,
  "samely": {
    "edit": 4,
    "skip": 0,
    "fixed": 0,
    "noterror": 0,
    "id": 4
  },
  "RichRico": {
    "edit": 1,
    "skip": 0,
    "fixed": 0,
    "noterror": 0,
    "josm": 1
  },
  "piligab": {
    "edit": 1,
    "skip": 0,
    "fixed": 0,
    "noterror": 1,
    "josm": 2
  }]

...

This change is pushed to production and the stats page on to-fix is not currently functional.

[geohacker]

@Rub21 thanks for renaming the type to versions. I think per chat with @ajithranka you all have agreed that the day wise stats is enough for metrics right now. Let's try this, and revisit here.

Thank you.