Reps Analytics not complete

[MichaelKohler]

I've noticed that the Reps Analytics (Events and Activities) are not complete for the past few months. This might be due to a recent Portal deployment, happy to help out if so.

There seem to be spikes where it should be mostly continous (with some fluctuation of course):

Activities

There don't seem to be any reports between July 20th and July 13th:

However there are activities in the data source at https://reps.mozilla.org/reports/:

Events

The same goes for events:

However there are activities in the data source at https://reps.mozilla.org/events/#/period/past/:

As seen in the initial screenshots, these timeframes are not the only one affected, I've just analyzed this one.

Would be great if this could be fixed. As said, I'm happy to help debug this if needed. Kudos to @freaktechnik for making me aware of this.

[Mte90]

Very strange, we didn't changed the rest api since a while so maybe it is a timeout? We need an error log from bitergia to see the problem

[mafesan]

Hi @MichaelKohler and @Mte90,

I'm Miguel-Angel, from Bitergia support team. Thank you for reporting this issue and sorry about the delay with it. Our Ops team has re-generated the database indexes and the data looks good now. However, our team is still investigating what caused this error. We'll keep you updated as soon as we have more information about it.

Best, Miguel-Angel

[MichaelKohler]

Definitely looks better now! The bar charts seem ok (didn't do a full check). However, the "Last events" widget still is missing some data - which is present in the bar chart.

[mafesan]

@MichaelKohler Our team has reviewed this again, we think there is a difference in the data because each of these events has a starting and an ending date: in the dashboard appears the starting date of each event (the list in your screen-shot is sorted by this field), while in https://reps.mozilla.org/events/#/period/past/ the list is sorted using the ending date.

For instance, if you look at the Moz Stall @ Angel Hack event in your website,

You will see that event both in the bar-chart and the Last Events widget marked with the starting date (Jun 30th):

(You can check this event in https://analytics.mozilla.community/goto/68136e73c00b0a1e870ec97a692fad1a, which appears on the bar-chart too).

What do you think about this? Thank you!

[MichaelKohler]

OK, so far so good. I still see for example this gap though (and I've sorted by start date):

There is no event listed between June 22nd and June 29th.

For example:

Let's take the "Security Analytics Workshop" entry. This has a starting and end date of June 23rd. I can also see it when filtering in Kibana: https://analytics.mozilla.community/goto/5466e58dc75c30e213f17f72414fd86e

However, without any filter, I don't see it showing up in "Last events" (see first screenshot). There is something wrong, I can't pinpoint it though.

[mafesan]

@MichaelKohler You are absolutely right, thank you for providing us that example. Due to how the Last Events widget is built, we set a limit to display only 30 results in it, in order to increase the performance of the dashboard. Instead of displaying the last 30 events, it seems Kibana has skipped some of the results randomly (only in June there were 89 events), so we have opted for increasing this limit to 100 and now you should see this data in the widget.

Please, can you check it again? Thank you and sorry for the inconveniences.

(See https://analytics.mozilla.community/edit/goto/10fb8ba0c0b3d3005b35ec259fdb0fae)

[MichaelKohler]

Ah, got it. This looks way better now. However, I think the Order does not work yet. I have to manually sort by start date to get them in descending order:

[canasdiaz]

Having a look at this table and the metrics it has, the displayed date is the max start_date while the sorting is using start_date.

This is an UI issue, we are working on it. Thanks for the feedback @MichaelKohler

The table by default is sorting by start_date but it is not working properly when items with same title, country and representative are found (and with different start_date). In those cases only one of these items is correctly sorted while the other appear next to it.

How to reproduce the error:

• go to https://analytics.mozilla.community/edit/goto/7b3381ab5f5503c8d35abbd623fe69a1
• make sure the UI is not sorting the tables, the "Start date" should show the two arrows (red circle)
• find events with same title, country and representative (blue rectangle)

[canasdiaz]

The issue we have with the table is related to the way Kibana display the information. The visualization by default groups by 'event title' and sorts by 'start date' instead of using only start date to sort all the events. This is why we get this strange behavior by default (until you click on the start date). The most simple way of changing this behavior is to change the order of the columns so the first one is unique ('event title' is not unique).

Due to this widget is used by you, we would love to get your feedback about the proposed changes before applying them to the standard panel. Are u ok with the changes?

Old one: https://analytics.mozilla.community/edit/goto/7b3381ab5f5503c8d35abbd623fe69a1 Proposed one: https://analytics.mozilla.community/edit/goto/ad6ed370089e6e2e9ca0d00cf9195783

Thank u :)

[MichaelKohler]

I'd be fine with this.

[hmitsch]

@Mte90 are you also okay? Once we have @MichaelKohler and you agree to this, I'd request @sanacl to move ahead with the proposed change and apply them to the standard panel.

-Henrik

[Mte90]

I am fine too!

[canasdiaz]

This is done. I've also fixed the link on the right table, which was not properly displayed.