Closed MichaelKohler closed 5 years ago
Hi @MichaelKohler,
Thank you for your feedback. Our team reviewed this data last week and indeed there was a gap with data from Sep 10th to Sep 14th.
We have generated a new raw index and the missing data is now available.
Sorry for the inconveniences!
Thanks for your work! I've had a look again and discovered some other dates where the import did not work. This seems to be an ongoing issue.
There are some gaps here, I've looked at two of them:
Sep 27th does not have any events in Kibana, there were 4 events on the Portal:
Sep 25th does not have any events in Kibana, there were 2 events on the Portal:
Please note that the gap on Sep 19th and on Sep 10th are correct!
Did you have a chance to look at the Activities as well? There are many gaps here and I don't see - on first sight - any improvement there. Just to make sure, what is the field from the API that you take as date? Not that it matters, as most gaps of those should not exist no matter which field it is.
For example there is a big gap between Sep 23rd and Sep 28th with absolutely no reports. Given the fact there were events, there are at least those automatically generated reports available on the Portal. The same goes for the gap between Sep 16th and Sep 21st and possibly others before that.
It's possible that some days didn't have any report, but definitely not that many, as said above, every event generates an automatic report as well. So if there was no event on one day and nobody filed a manual report, then it could be empty. But not otherwise.
Once these issues are fixed, I can also verify the total amount per day, as I've not looked into that yet, but I assume for now that if something is imported, the total is correct.
You can check out https://reps.mozilla.org/reports as graphical interface to the API for the reports and https://reps.mozilla.org/events/#/period/past/ for the events.
Thanks so much for having another look into this! Michael
I am joining only because I am curious about why this is happening :-)
Thank you @MichaelKohler for your detailed comment. We are having a deeper look into this issue, we will get back to you as soon as we can provide more information about it. Please stay tuned!
Best, Miguel-Angel
Hi,
We are currently working to upgrade your dashboard to the newest version of the platform, so we are waiting until the upgrade is ready to test this issue again with our latest release. Sorry for the inconveniences!
Best, Miguel-Angel
Hey @mafesan , the upgrade is done so we are not using software from the past. Let's push this task as we are now able to apply any fix needed in our software.
Thank you for the heads-up @sanacl! Our team is reviewing this issue again after the upgrade and we will post a comment as soon as we get more information. Please stay tuned! (cc @MichaelKohler, @Mte90).
Best, Miguel-Angel
Sorry for the delay on this issue 🐢. We have been reviewing this issue and after several checks we noticed the problem is on the incremental retreival for Remo backend (when we collect the data from scratch, we get the complete set of elemements). Our Dev team is working to solve this as soon as poosible, we will post a comment with any news. Sorry for the inconveniences!
Best, Miguel-Angel
This is still a WIP for our developers. I guess it won't be ready by this Friday :(
We were working on this issue but our Dev team could not reproduce this bug :bug:, so we have taken the following actions:
Best, Miguel-Angel
Instead of pursuing the incremental data update approach. The Reps data is now fully refreshed every day.
It seems that we are missing some Reps data.
Events
According to the "Last events" table, there were no events between Sep 9th and 14th. However, on the Reps Portal, you can see that there definitely were events:
Activities
The same goes for Activity Reports. You can take my reports as an example. When I choose "Last 30 days" the most recent recent report from me in the "Last activities" table is from August 28th. I've had reports since then:
For both cases it doesn't seem that the imports stopped as there are very recent entries, but rather that there are entries missing inbetween (gaps). Could you have a look at this?
Thanks!