search

Metro-Records / la-metro-councilmatic

:metro: An instance of councilmatic for LA Metro
MIT License
6 stars 2 forks source link

Create a scraper/data dashboard #257

Closed reginafcompton closed 4 years ago

reginafcompton commented 6 years ago

Metro admin should be able to monitor the amount of data in Councilmatic and its relation to that of Legistar. The below provides a list of possibilities. (Collated from suggestions given over a period of several months.)

Data report: Possible formats

(1) An auto-generated weekly report sent to the Metro team via email.

(2) An admin dashboard, where Metro staff can generate a "data integrity" report, whenever desired. (N.b., Metro staff already have login credentials for Councilmatic, i.e., for uploading agenda PDFs).

Wish list

shrayshray commented 6 years ago

@reginafcompton, this looks good to me. @camachoo anything to add?

hancush commented 4 years ago

We're building a data dashboard! We discussed the things this dashboard should display at our 5/6 priorities meeting. Here's a text summary: https://docs.google.com/document/d/1vVcAHISA3_KSEB13CW74Lsplc9-6efe9BvacDRtbB_M/edit?usp=sharing

@shrayshray @camachoo Let me know if you think of anything else.

We also talked about being able to request that specific bills/events are scraped. This is a great long term goal for dashboard (I'll open a separate issue), but it won't be in the first version. We can, however, prioritize recipes for specific scrapes that will allow us to respond more quickly to these requests (i.e., run them with a button click, instead of writing and running them ad hoc), as part of the first iteration.

Also opening a separate issue for exposing useful metadata to admins on report and event detail pages.

hancush commented 4 years ago

P.s., the work for the dashboard is over in a private repository: https://github.com/datamade/la-metro-dashboard. I'll use this issue as a stand-in for that work in our project board.

hancush commented 4 years ago

The first draft of the dashboard is ready for input: https://la-metro-dashboard.datamade.us/dashboard/. Sent an email with more specific prompts – let's continue discussion there!

hancush commented 4 years ago

Really thrilled to report that the dashboard is in a stable place! FYI, @shrayshray, I'm going to be out of pocket 8/13 through 8/17, but I will be reaching out to you all next week to coordinate test data entry and put the dashboard through the paces before we deploy to production and sunset the old scrapers. 🚀

shrayshray commented 4 years ago

Sounds great, thanks @hancush !

hancush commented 4 years ago

Hey, @shrayshray – Thanks for your patience here. I'm putting together the production deployment this week. Are you available Thursday or Friday to coordinate on some test data entry, specifically adding some fake agendas / other sorts of data entry that happens during support windows? Let me know!

shrayshray commented 4 years ago

@hancush - yes! How about Thursday 8/27 at 11am or 12pm PDT?

hancush commented 4 years ago

12 p.m. Pacific / 2 p.m. Central would be perfect! I'll configure the support scrapes to run during that time period, and update this thread to confirm they're ready for you.

shrayshray commented 4 years ago

@hancush I apologize - would it be possible to do this Monday 8/31 instead? My mistake for not having it on the calendar, but the Regular Board meeting will still be running at noon on Thursday, and after it I'm booked until 5pm PDT.

hancush commented 4 years ago

No worries, @shrayshray! Monday would be great. Same time (12 PDT / 2 CDT)?

shrayshray commented 4 years ago

@hancush Yes, 12 PDT / 2 CDT Monday is good. Should I set up a GoToMeeting or do you just want to use email?

hancush commented 4 years ago

Email should be fine, @shrayshray!