Twitter data is not updated

[canasdiaz]

@hmitsch just reported Twitter data is broken since March 28th. Highest priority.

[canasdiaz]

Hi @hmitsch , we have two issues here:

• enrichment is was not working (should be solved during next working hours)
• we've lost data from March 28th to April 6th, it is not possible to collect that data using the Twitter API

The Twitter data collection was broken until this Monday, that day the collection was restarted and the tweets were restored from the past 7 days (this is all the API can offer). The issue with this is we've lost data for the period of time between March 28th and April 6th. That means that when the enrichment finishes you'll see an awful gap in your data. The recovery process we have for avoid losing Twitter data did not worked, the guys fixed it yesterday.

This is pretty embarrassing so we've decided to refund part of the subscription fee. Let's talk privately about this.

This ticket will remain open until enrichment shows the Twitter data from April 6th until today.

[hmitsch]

Hi @sanacl thanks a lot for the details and the quick response. Don't worry about the lost data. From a Mozilla-internal perspective, this function is still in beta.

[canasdiaz]

Data is updated with the awful gap from March 28th to April 6th.

In order to avoid this in the future:

• Jenkins check has been fixed to watch every hour the Twitter indexes
• backups script deployed to get the data from the API in case logstash fails

We are done here.