Bug in Reps Events (NOV 2017)

[hmitsch]

Reps portal

• Test condition URL: https://reps.mozilla.org/events/#/period/custom/end/2017-10-31/
• Actual result: 143 events found till 2017-10-31.

Community Analytics

• Test condition URL: https://analytics.mozilla.community:443/goto/c10c9173b4207ada4e71a84df46a5b55
• Actual result: 11 events

Additional information

According to an analysis by @kelimuttu, @mte90 and @MichaelKohler:

[..] the data in Reps tab is starting to digress from April. [..] There's definitely an issue on the data fetching.

[hmitsch]

@sanacl can you please assign this to the correct person at Bitergia?

[canasdiaz]

@hmitsch I'll be the proxy. We are currently working on it.

[canasdiaz]

We still owe you guys an update about this. Work in progress. As soon as we get an ETA we'll let you know

[canasdiaz]

Tested with latest release and it is fixed with that version. So, as soon as Bugzilla is finished (tomorrow morning the latest) we'll recover Reps using latest release.

@hmitsch ETA: Thursday 5PM CET

[canasdiaz]

@hmitsch data looks much better now with our latest release

2017:

• I've done a query over https://reps.mozilla.org/events/#/period/custom/end/2017-10-31/start/2017-01-01/ and got 901 events

October:

• 143: https://reps.mozilla.org/events/#/period/custom/end/2017-10-31/start/2017-10-01/

November (12th to 25th):

• 39: https://reps.mozilla.org/events/#/period/custom/end/2017-11-25/start/2017-11-12/

We have a couple of more events on the dashboard, maybe due to the interpretation of some dates. This is under study.

[Mte90]

I can confirm that now is working :-D

[canasdiaz]

I see little differences on the data, this is something that needs to be fixed.

Original data source returns 11 event in December (until today): https://reps.mozilla.org/events/#/period/custom/end/2017-12-08/start/2017-12-01/

Dashboard returns the same number of events but some of them differ: https://mozilla5.biterg.io/goto/cd788e3b7629ed9af236e5131e7ac699

I miss on the dashboard:

• Weekly Call-1 "Logout"- The Cyber Awareness Campaign
• Reps Council Weekly Meeting 2017-12-05

Event that appears on the dashboard but not on the original data source:

• Rust India Community Monthly Call

[Mte90]

Uhm maybe because they are missing the metrics after event. For the reps council call i have to remove it (we didn't do the meeting) so I didn't added them. So if is this the reason probably ignore them is the right solution because is an activity not yet finished.

[hmitsch]

@Mte90 thank you so much for helping on this!

@sanacl can you confirm @Mte90's assumption (events are missing metrics)?

How should we proceed with this?

[Mte90]

In the meantime the council weekly call event mentioned was removed ;-)

[canasdiaz]

Our development team is trying to isolate the bug. I'll keep you updated. This ticket is high priority.

[acs]

@hmitsch @Mte90 after reviewing the issue, it seems that the ReMo API is now returning the events from the newest (page=0) to the oldest (page=334): https://reps.mozilla.org/api/remo/v1/events

In our backend, the logic is the reverse, so the incremental update is broken. Our guest is that the APi has changed the behavior about the ordering of the events. We need to get the oldest first in order to have a robust incremental update of the data source.

Could you check if the API was changed? Could you check if it is possible to change the order in which the events are returned, so page=0 are the oldest ones?

Maybe is it already available? The Django REST Framework has support for this kind of stuff: http://www.django-rest-framework.org/api-guide/filtering/#filtering-against-query-parameters

Thanks and sorry about the delay for attending this ticket.

[MichaelKohler]

Could you check if the API was changed? Could you check if it is possible to change the order in which the events are returned, so page=0 are the oldest ones?

Yes it did, see https://github.com/mozilla/remo/pull/1407

[Mte90]

Maybe we can add a parameter to define the order as ASC or DESC to avoid any issue

[acs]

Yes, if you add a parameter to change the order, this fix for the ReMo backend is pretty easy (just to pass this option). And it is the only way to have a robust incremental collection of the data. Guys, could you confirm that this feature can be added? If so, we will wait until it is ready and then, we will modify the backend and deploy it.

Thanks!

[acs]

@Mte90 @MichaelKohler any news? Thanks!

[acs]

@Mte90 @MichaelKohler any news? Thanks!

[Mte90]

We discussed and we need to find a slot of time to fix that issue on the portal.

[acs]

@Mte90 thank you very much for the update. Is there any issue we can track to check when it has been fixed?

Cheers!

[MichaelKohler]

I'd say this one here? ;)

[acs]

Ok @Mte90 . I was not sure if you have also another ticket in your portal development issue system. So let's focus in this one :)

Cheers!

[canasdiaz]

Hi @Mte90 @MichaelKohler, this ping is just to make sure we are on the same page and the parameter has not been applied yet. Thanks!

[Mte90]

I made a patch for that on remo: https://github.com/mozilla/remo/pull/1440

Waiting for tests and approval :-D

[canasdiaz]

Great. Thanks for let us know @Mte90 Crossing fingers to get those tests passed :nerd_face:

[Mte90]

The Pr is on production! With https://reps.mozilla.org/api/remo/v1/events/?format=json&orderby=ASC you can get the events by the most oldest if you change on DESC you will get the most recent :-)

[canasdiaz]

@hmitsch @acs We've been doing some tests. Events can be collected from the older to the newer. Nevertheless, the other categories (activities and users) doesn't seem to be collected from the older to the newer using the parameter orderby=ASC. Is it a problem? I guess no, but just to make sure we don't need anything else from our Mozilla folks.

[acs]

@hmitsch @acs We've been doing some tests. Events can be collected from the older to the newer. Nevertheless, the other categories (activities and users) doesn't seem to be collected from the older to the newer using the parameter orderby=ASC. Is it a problem? I guess no, but just to make sure we don't need anything else from our Mozilla folks.

It is a problem because we are also collecting activities following the same approach than events. So we need the same feature for activities at least.

[MichaelKohler]

As far as I know the behaviour of the activities hasn't changed. :/ I'm on my phone right now, so I can't test that well. Is it broken as well using the same URL as you have been before?

[Mte90]

I added the code only for events because the ticket was referring on them. I think that we can add the same code also for the activities and users without problems. I will see today if I can do a PR :-)

[canasdiaz]

@Mte90 that would be great. Sorry for not giving you all the information in the first place.

[Mte90]

Another PR made waiting for the approval and push to production, https://github.com/mozilla/remo/pull/1445

[hmitsch]

Reps portal was deployed to production end of last week. Sorry for getting back so late.

[acs]

@hmitsch the remo backend has been fixed to work with the new API. The fix is waiting for review at: https://github.com/chaoss/grimoirelab-perceval-mozilla/pull/30. Once it is accepted, it will be included in the next release of the platform, which will be deployed to your dashboard and then, the remo data will be another time active.

Probably all of this should occur during the next week but we have two bank holidays next week, so probably the release will appear in two weeks. And after that, it needs to be deployed.

@sanacl could provide you a more accurate timeline if needed.

[canasdiaz]

Code is merged upstream, waiting for the next GrimoireLab release.

[canasdiaz]

New version deployed. Waiting for the analysis to be finished.

[canasdiaz]

Reps Events in October 2017:

• 147 according to the source https://reps.mozilla.org/events/#/period/custom/end/2017-10-31/start/2017-10-01/
• 147 according to the dashboard https://analytics.mozilla.community/goto/c908e84ce5716449f19f664b350d627b

Reps Activities:

• This is not working OK. This contributor should have activity in May https://analytics.mozilla.community/goto/d2cf844460a81c16ae0cf0cc98f15e63
• His activity in the data source https://reps.mozilla.org/u/michaelkohler/

[Mte90]

This is a fail of the Reps REST API or not?

[canasdiaz]

@Mte90 don't think so. We're still debugging it.

[canasdiaz]

This is finally fixed.

Reps Events in October 2017:

• 147 according to the source https://reps.mozilla.org/events/#/period/custom/end/2017-10-31/start/2017-10-01/
• 147 according to the dashboard https://analytics.mozilla.community/goto/c908e84ce5716449f19f664b350d627b

Reps Activities:

• 4 reports during May (up to May 9th) https://analytics.mozilla.community/goto/d2cf844460a81c16ae0cf0cc98f15e63
• His activity in the data source https://reps.mozilla.org/u/michaelkohler/ is coherent :heavy_check_mark: