Alerts sometimes appear to be several weeks late

[sebbacon]

Alerts are only sent out to users when a request state changes. This means that a request that got a reply several weeks ago will only turn into an email alert when someone marks it as "successful" (or whatever).

For alerts which ask for all updates from a particular body or a particular request, we should change the logic to alert on a new incoming request, rather than a state change.

Notes from Francis:

For queries which include the status, it obviously can't do that.

Right now the alert engine uses a search query internally. Very few alerts though are free text so there are two options:

a) For certain kinds of alerts (like on a particular public body) special case it to use the created at rather than described at. Continue to use described at for all free text search alerts.

b) Do as in a), but also somehow parse the free text alert. Maybe via a tree you get back from Xapian. Use that to see if it is referring to the status of the request, in which case use described at rather than created at.

Might need care as to how the rest of the alerts are structured in how they decide what to alert on and used described vs. created.

This fits in a bit with how you want to do a permission system as well.

In the current code, I was deliberately driving lots of things through search, which is weird to people who love relational database, but to my mind is more general (you always need full text search in the end) and leaves less complex code. My plan was to do permissions via the search as well.

However, for other reasons, you might want to do more things via database queries. But keep in mind that it might make the code more complex when you add search features or permissions.

[crowbot]

Another comment from a user who was alerted on the same incoming message twice - because it had changed state.

[hsenag]

Just to note WhatDoTheyKnow gets one support email every few months about this.

[RichardTaylor]

I agree with the above comments which suggest an alert message to a user tracking a request or search term should be prompted by an incoming message, and not a change in classification.

Also the email which goes out should make clear why the alert is being issued; the below example email to the WhatDoTheyKnow support address quotes an alert message which doesn't say that it was a classification on the 19th of March which prompted it, in fact it suggests there's a new update on teh thread, which there isn't from the point of view of the reader:

Hi there, this was 2 months ago, is this the speed of update notifications? Kind regards

----Original Message---- From: [removed] Date: 19/03/2018 2:25 To: [removed] Subj: Your WhatDoTheyKnow email alert

New updates for the request 'Delegation of Police Crime Commissioner Duties and published reports / recommendations as under the Police reform and Responsibility Act 2011'

Norfolk Police and Crime Commissioner sent a response to Wayne Leigthon (19 January 2018) "Dear Mr Leighton Please find attached a response from Mark Stokes. Kind regards ===================================================================..." https://www.whatdotheyknow.com/request/delegation_of_police_crime_commi?nocache=incoming-1098717#incoming-1098717

[RichardTaylor]

Just to offer another example showing the need to explain why an alert has been sent:

On Thu, Aug 2, 2018, 6:19 PM WhatDoTheyKnow track@[domain] wrote:

New updates for the request '[request subject]'

Home Office sent a response to [user name] (25 April 2016) "Dear Sir or Madam FOI [number] Please find attached a reply to your Freedom of Information request that you sent on 24 March. Regards..." [url]

Another related idea: when old responses, say those received over 2-3 months ago, are classified - don't send alerts on classification

[garethrees]

Another comment from a user who was alerted on the same incoming message twice - because it had changed state.

I just had one of these.

New updates for the request 'FixMyStreet questionnaire results'
===============================================================

Oxfordshire County Council sent a response to Helen Highwater (24 September 2018)

It looks broken, because we're saying "sent a response" and the date is last year.

What actually happened was that the user changed the classification.

[garethrees]

I just had another of these, and something looks fishy.

New updates for the request 'FixMyStreet Pro'
=============================================

Lincolnshire County Council sent a response to Abdul Hai (15 January 2019)
"Good morning   Please see the attached response to your FOI request from 25th August 2018. Sincere apologies for the delay.   Kind regards  ..."
https://www.whatdotheyknow.com/request/fixmystreet_pro?nocache=incoming-1293348#incoming-1293348

I'd need to really read the code around described_state and calculated_state to remember how all this works, but it looks like the request was classified as successful when the response came in on 2019-01-15 (1). Then, on 2019-06-18, the request was re-classified as successful from awaiting_response (2).

I'd already received an alert from the initial classification on 2019-01-15.

---

Looking back at https://github.com/mysociety/alaveteli/issues/27#issuecomment-489025650, we see the same pattern – this time with not_held:

---

The several edit events are caused by https://github.com/mysociety/alaveteli/issues/4776, which I didn't think was related, but I wonder if the addition of creating an InfoRequestEvent in https://github.com/mysociety/alaveteli/commit/0f7994989a7608b6f1eae101e73160994fd13747 means that we lose the "calculated state" from the response event?

[MattK1234]

Not 100% sure if this is the correct issue to add this to, but...

A user today contacted WDTK support saying:

I have an alert for "aberdeen" set up and today got an item in the digest for an FOI request dated in 2018 https://www.whatdotheyknow.com/request/you_have_and_office_in_the_marsc?nocache=incoming-1268033#incoming-1268033 No idea what happened there, perhaps there was an update that was then deleted but thought I'd flag it you anyway.

This request was categorised by me earlier in the day.

[RichardTaylor]

A user has today suggested that alerts should be sent to those following a request when new correspondence arrives on the thread.

That is how I would expect the alert / track / follow to work too.

[mdeuk]

A user has today suggested that alerts should be sent to those following a request when new correspondence arrives on the thread.

That is how I would expect the alert / track / follow to work too.

To my knowledge, we do exactly what is being suggested here; however, after a substantive event happens (e.g. after a response is classified), we appear to generate the same tracking message again.

Case in point:

I have a track setup for request 624904 on WDTK (tid:152807). A track message received on 8th January contained:

New updates for the request 'RSAS accreditation
================================================

Bob Packard sent a follow up message to Police Crime Prevention Initiatives Ltd ( 8 January 2020)
"I note that I have not received a response to my Freedom of Information request dated 3 December 2019, nor, have you had the courtesy to acknowledge..."
https://www.whatdotheyknow.com/request/rsas_accreditation?nocache=outgoing-979977#outgoing-979977

Police Crime Prevention Initiatives Ltd sent a response to Bob Packard ( 8 January 2020)
"Thank you for your request for information, and apologies for the delay in replying to your request. Please find attached our response to your request...."
https://www.whatdotheyknow.com/request/rsas_accreditation?nocache=incoming-1497226#incoming-1497226

However, when the response was eventually classified on the evening of the 10th of June (e.g. six months later!) we generated the same tracking message again:

New updates for the request 'RSAS accreditation'
================================================

Police Crime Prevention Initiatives Ltd sent a response to Bob Packard ( 8 January 2020)
"Thank you for your request for information, and apologies for the delay in replying to your request. Please find attached our response to your request...."
https://www.whatdotheyknow.com/request/rsas_accreditation?nocache=incoming-1497226#incoming-1497226

Are we generating this second tracking message because Alaveteli is detecting the change of classification (or event being generated) as a reason to which it should send a track?

If so, I ponder if we should either include the reason for sending it (e.g. the request has been classified, which might prompt a casual reader to look at it in more detail), or stop Alaveteli from generating tracks based on classification events.

[HelenWDTK]

+1 A user has got in touch to let us know about a dupe alert that they received. This was caused by the request being classified a month after the first alert.