search

ElixirTeSS / TeSS_scrapers

TeSS HTML page scrapers in Ruby looking for training resources and events metadata.
Other
9 stars 9 forks source link

NBIS scraper: cancelled events not removed from TeSS #73

Open nmb opened 5 years ago

nmb commented 5 years ago

The NBIS scraper collects information from a google calendar. However, if we cancel an event and remove it from the calendar, the event is not removed from TeSS.

Example of a cancelled event.

njall commented 5 years ago

Hi @nmb

The problem is the absence of an event does not necessarily mean it has been cancelled.

  1. Sometimes the scraper runs into errors
  2. Sometimes the target website is temporarily inaccessible
  3. Sometimes the event has finished successfully and then is removed from the target website. We like to keep an archive of past events because this data could be useful for future scheduling, or for someone to verify dates of an event they've attended.

Removing the event seems like a bad idea because people may have added extra data to TeSS about it.

At the moment we add a flag to the event in TeSS if the event has been inaccessible for 3 consecutive days. This can only be seen by Curators (of which we currently have none). The idea is they will then delete the event or relabel it as cancelled.

We currently let one user 'own' a content provider so they can add/edit/delete freely. I could assign you this role for NBIS, which would allow you to delete cancelled events.

It would be good to give greater control to content providers in the future - e.g. allow multiple accounts shared control over a content provider, send e-mail notifications when a resource has been accessible for 3 days asking them to remove/flag it as cancelled.

Any thoughts on what you would like to see to handle this type of scenario?

nmb commented 5 years ago

Sounds like a good plan!

Better to set our training coordinator @jlindvall as the user that can edit NBIS content.