opencivicdata / scrapers-us-municipal

Scrapers for US municipal governments.
MIT License
101 stars 67 forks source link

Metro: Skip minutes inference if there is more than one minutes file #350

Closed hancush closed 3 years ago

hancush commented 3 years ago

Description

This PR handles the case where there is more than one file approving minutes of a particular meeting. This is a stopgap to unbreak event scrapes while I work with Metro to determine the right thing to do in this instance.

The breaking scenario is that minutes for a particular meeting were scheduled to be approved, but the approval was carried over to the next meeting. Then, a second file for the approval was created, rather than reusing the existing file. Interestingly, this file was withdrawn, so neither file actually represents the approval of minutes.

As I said, I'm working with Metro to determine what to do in this case, but this only affects one event, and I don't think the scrapes should stay broken in the interim.

Testing instructions

Done locally.

$ docker-compose run --rm scrapers pupa update lametro events --rpm=0
...
08:25:27 WARNING legistar: Found more than one minutes file for the LA SAFE meeting of June 25, 2020.
...
lametro (scrape, import)
  events: {}
events scrape:
  duration:  0:10:45.155556
  objects:
    event: 487
jurisdiction scrape:
  duration:  0:00:00.163220
  objects:
    jurisdiction: 1
    organization: 3
    post: 18
import:
  event: 487 new 0 updated 0 noop
  jurisdiction: 0 new 0 updated 1 noop
  organization: 0 new 0 updated 3 noop
  post: 0 new 0 updated 18 noop