CouncilDataProject / cookiecutter-cdp-deployment

Cookiecutter template for creating new CDP instances.
Mozilla Public License 2.0
26 stars 9 forks source link

[Instance]: Alameda #78

Closed phildini closed 2 years ago

phildini commented 2 years ago

Municipality Name

The City of Alameda

Governing Body Type

city council

Maintainer GitHub Name

phildini

Legistar Client Id

alameda

Municipality Timezone

America/Los_Angeles

Municipality Slug

alameda

Firestore Region

No response

Code of Conduct

edit by @JacksonMaxfield, testing if this instance can be deployed with new LegistarScraper setup

github-actions[bot] commented 2 years ago

Hello! 👋 Thanks for initiating the process to configure a new CDP instance.

I am a bot that will validate the information provided in your form. If any check fails, please update your issue by opening the '⋯' dropdown in the top-right-corner of your GitHub Issue and selecting 'Edit'. I will automatically rerun the checks after you update the issue to validate the changes.

A member from the CDP team will respond as soon as possible!

Form Validation Results

✅ Governing body type is an accepted value. ✅ @phildini has been marked as the instance maintainer. ✅ CouncilDataProject/alameda is available. ✅ The municipality's Legistar instance contains the minimum required CDP event ingestion data.

Retrieved Data
```json { "body": { "name": "Transportation Commission", "is_active": true, "start_datetime": null, "description": null, "end_datetime": null, "external_source_id": "177" }, "sessions": [ { "session_datetime": 1653528600.0, "video_uri": "http://archive-media.granicus.com:443/OnDemand/alameda/alameda_d1b734a9-6348-4b8f-aef9-d635a90dad41.mp4", "session_index": 0, "caption_uri": null, "external_source_id": null } ], "event_minutes_items": [ { "minutes_item": { "name": "ROLL CALL", "description": null, "external_source_id": "58400" }, "index": 2, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "AGENDA CHANGES", "description": null, "external_source_id": "58401" }, "index": 3, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2051", "description": "Staff Communications (A-D)", "external_source_id": "58403" }, "index": 4, "matter": { "name": "2022-2051", "matter_type": "Staff Communications", "title": "Staff Communications (A-D)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11828" }, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "ANNOUNCEMENTS/PUBLIC COMMENT", "description": null, "external_source_id": "58404" }, "index": 5, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "CONSENT CALENDAR", "description": null, "external_source_id": "58405" }, "index": 6, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2052", "description": "Approve Meeting Minutes \u2013 March 23, 2022 (Action Item)", "external_source_id": "58406" }, "index": 7, "matter": { "name": "2022-2052", "matter_type": "Consent Calendar Item", "title": "Approve Meeting Minutes \u2013 March 23, 2022 (Action Item)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11829" }, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2053", "description": "Accept Status Report on Transportation (Action Item)", "external_source_id": "58407" }, "index": 8, "matter": { "name": "2022-2053", "matter_type": "Consent Calendar Item", "title": "Accept Status Report on Transportation (Action Item)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11830" }, "supporting_files": [ { "name": "Status Report on Transportation: May 2022", "uri": "http://legistar1.granicus.com/alameda/attachments/98908056-a44c-4d1a-9361-3c32cdf5ad7f.pdf", "external_source_id": "23479" } ], "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2054", "description": "Accept Updated Equity Priority Analysis Approach (Action Item)", "external_source_id": "58408" }, "index": 9, "matter": { "name": "2022-2054", "matter_type": "Consent Calendar Item", "title": "Accept Updated Equity Priority Analysis Approach (Action Item)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11831" }, "supporting_files": [ { "name": "Exhibit 1 EquityPrioritization Presentation", "uri": "http://legistar1.granicus.com/alameda/attachments/456b98da-c823-4834-a004-7c119696768a.pdf", "external_source_id": "23480" }, { "name": "5C Exhibit 2 Equity Priority Areas Memo", "uri": "http://legistar1.granicus.com/alameda/attachments/0e60c7d6-4ede-4e87-88de-b644c5085459.pdf", "external_source_id": "23481" } ], "decision": null, "votes": null }, { "minutes_item": { "name": "REGULAR AGENDA", "description": null, "external_source_id": "58409" }, "index": 10, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2055", "description": "Recommend City Council Approval of the Grand Street Improvement Project Concept (Robert Vance, City Engineer and NCE/Fehr & Peers staff) (Action Item)", "external_source_id": "58410" }, "index": 11, "matter": { "name": "2022-2055", "matter_type": "Regular Agenda Item", "title": "Recommend City Council Approval of the Grand Street Improvement Project Concept (Robert Vance, City Engineer and NCE/Fehr & Peers staff) (Action Item)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11832" }, "supporting_files": [ { "name": "Street Configuration (Alternative)", "uri": "http://legistar1.granicus.com/alameda/attachments/dea9ea0c-1656-41bc-98ee-9b8622921a11.pdf", "external_source_id": "23482" }, { "name": "Proposed Street Configuration (Recommendation)", "uri": "http://legistar1.granicus.com/alameda/attachments/ae9c90d1-b390-4cd6-a473-5de2b1f9915a.pdf", "external_source_id": "23483" }, { "name": "2022 Survey - GrandStreet Questionnaire April 2022", "uri": "http://legistar1.granicus.com/alameda/attachments/617a6aa9-335c-4086-be01-649b71afdd5d.pdf", "external_source_id": "23484" }, { "name": "2022 Survey - GrandStreet Results April 2022", "uri": "http://legistar1.granicus.com/alameda/attachments/ddd025d1-f9d5-4b81-9d19-15fe9ae72de3.pdf", "external_source_id": "23485" }, { "name": "Grand St Community Workshop #2 Notes May 6 2022", "uri": "http://legistar1.granicus.com/alameda/attachments/6e8203c6-b896-4535-89c6-81dcf45208e0.pdf", "external_source_id": "23486" }, { "name": "Grand St Transportation Commission May 25, 2022 PowerPoint", "uri": "http://legistar1.granicus.com/alameda/attachments/86082e03-7eb7-49b0-9cc5-861a75c381d5.pdf", "external_source_id": "23487" }, { "name": "Correspondence \u2013 Batch 1", "uri": "http://legistar1.granicus.com/alameda/attachments/058ddaa1-415c-448d-80c2-25d7adc78e73.pdf", "external_source_id": "23510" }, { "name": "Correspondence - Batch 2", "uri": "http://legistar1.granicus.com/alameda/attachments/4f4529ec-1df9-4a20-a636-1ebfe983ccbc.pdf", "external_source_id": "23545" }, { "name": "Correspondence - Batch 3", "uri": "http://legistar1.granicus.com/alameda/attachments/e4677977-4ffe-47eb-8740-de30b63a84a4.pdf", "external_source_id": "23557" }, { "name": "Correspondence - Batch 4", "uri": "http://legistar1.granicus.com/alameda/attachments/8ff938ca-7746-4bb1-a62d-0efac23a1f35.pdf", "external_source_id": "23577" } ], "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2056", "description": "Discuss the Citywide Roundabout Analysis (Gail Payne, Senior Transportation Coordinator and Mike Alston, Kittelson & Associates, Inc.) (Discussion Item)", "external_source_id": "58411" }, "index": 12, "matter": { "name": "2022-2056", "matter_type": "Regular Agenda Item", "title": "Discuss the Citywide Roundabout Analysis (Gail Payne, Senior Transportation Coordinator and Mike Alston, Kittelson & Associates, Inc.) (Discussion Item)", "result_status": "In Progress", "sponsors": null, "external_source_id": "11833" }, "supporting_files": [ { "name": "Roundabout Education Presentation", "uri": "http://legistar1.granicus.com/alameda/attachments/24c337b0-4c53-4726-aa26-3f102a5203ee.pdf", "external_source_id": "23488" }, { "name": "Roundabout Sherman Encinal Central ROW memo", "uri": "http://legistar1.granicus.com/alameda/attachments/b2b2ba16-83b5-4656-b4a1-683f1e2fb959.pdf", "external_source_id": "23489" }, { "name": "Roundabout Screening and Feasibility Memo for High Injury", "uri": "http://legistar1.granicus.com/alameda/attachments/7a4bee44-47ed-4ca8-822b-da859883bfda.pdf", "external_source_id": "23490" }, { "name": "Alameda Citywide Roundabout Screening", "uri": "http://legistar1.granicus.com/alameda/attachments/8e76394d-5d7d-4e3e-a6f4-810b5ecfb5d7.pdf", "external_source_id": "23491" }, { "name": "Alameda Citywide Screening Map", "uri": "http://legistar1.granicus.com/alameda/attachments/e1af534c-4f9e-4d19-9321-c06c99d31b64.pdf", "external_source_id": "23492" }, { "name": "Alameda Citywide Roundabout Rankings", "uri": "http://legistar1.granicus.com/alameda/attachments/9b00ad19-7fb2-4ca3-9454-bd071121e429.pdf", "external_source_id": "23493" } ], "decision": null, "votes": null }, { "minutes_item": { "name": "2022-2057", "description": "Incorrect Staff Report Loaded: Removed 5/26/22", "external_source_id": "58412" }, "index": 13, "matter": { "name": "2022-2057", "matter_type": "Regular Agenda Item", "title": "Incorrect Staff Report Loaded: Removed 5/26/22", "result_status": "In Progress", "sponsors": null, "external_source_id": "11834" }, "supporting_files": [ { "name": "April 2022 Questionnaire", "uri": "http://legistar1.granicus.com/alameda/attachments/61e3e309-7ee3-43f2-bf6c-07f0243a34ee.pdf", "external_source_id": "23494" }, { "name": "Survey Summary", "uri": "http://legistar1.granicus.com/alameda/attachments/2816487a-ad70-4c05-90af-8e74e50ad8fa.pdf", "external_source_id": "23495" }, { "name": "Workshop Open House Summary", "uri": "http://legistar1.granicus.com/alameda/attachments/de1f08a8-f617-42ed-96f3-abadcecc83d1.pdf", "external_source_id": "23496" }, { "name": "Draft Concept", "uri": "http://legistar1.granicus.com/alameda/attachments/6bec5613-b55b-4241-999c-b9ae618044b6.pdf", "external_source_id": "23497" }, { "name": "Presentation", "uri": "http://legistar1.granicus.com/alameda/attachments/89fb1f48-3b4b-463f-b634-c9b4aae16c85.pdf", "external_source_id": "23574" } ], "decision": null, "votes": null }, { "minutes_item": { "name": "ANNOUNCEMENTS/PUBLIC COMMENTS", "description": null, "external_source_id": "58413" }, "index": 14, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "ADJOURNMENT", "description": null, "external_source_id": "58414" }, "index": 15, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "****NOTES****\nAccessibility\nAccessible seating for persons with disabilities (including those using wheelchairs) is available. Translators or sign language interpreters will be available on request. Equipment for the hearing impaired is available for public use. For assistance or to request an interpreter, please contact the Planning, Transportation and Building Department at (510) 747-6805 or the City Clerk\u2019s Office (510) 522-7538 (TDD number) at least 48 hours prior to the meeting. Minutes of the meeting may be made available in enlarged print. Please contact the Planning, Transportation and Building Department at (510) 747-6805 at least 48 hours prior to the meeting to request agenda materials in an alternative format, or any other reasonable accommodation that may be necessary to participate in and enjoy the benefits of the meeting.\nBicycle Parking\nBicycle racks are available outside the City Hall entrances on Oak Street and Santa Clara Avenue.\nDocuments\nDocuments related to this agenda are available for public inspection and copying at the Planning, Transportation and Building Department, City Hall Room 120, Alameda, CA 94501, and at the City Clerk\u2019s office, City Hall, Room 380 during normal business hours or on the City\u2019s web page at https://alameda.legistar.com/Calendar.aspx\nPublic Input Process\nShould you have questions regarding the above projects, please contact Gail Payne 510-747-6892 or email: gpayne@alamedaca.gov. If you have comments about these projects and are unable to attend the meeting, you may submit written comments to Gail Payne, Planning, Transportation and Building Department, City Hall Room 120, Alameda, CA 94501 or by email gpayne@alamedaca.gov. All written comments received before the meeting begins will be made part of the official public record and provided to the Transportation Commission members prior to its consideration of the project.\nSign up to receive agendas here: https://alameda.legistar.com/Calendar.aspx", "description": null, "external_source_id": "58415" }, "index": 16, "matter": null, "supporting_files": null, "decision": null, "votes": null }, { "minutes_item": { "name": "For Zoom meeting registration, please click:\nhttps://alamedaca-gov.zoom.us/webinar/register/WN_aHkm-xTSRtKFO4LzjIcaSg \nAfter registering, you will receive a confirmation email containing information about joining the webinar.\nFor Telephone Participants:\nZoom Phone Number: 669-900-9128\nZoom Meeting ID: 846 7123 5326", "description": null, "external_source_id": "58416" }, "index": 1, "matter": null, "supporting_files": null, "decision": null, "votes": null } ], "agenda_uri": "https://legistar1.granicus.com/alameda/meetings/2022/5/5482_A_Transportation_Commission_22-05-25_Meeting_Agenda.pdf", "minutes_uri": null, "static_thumbnail_uri": null, "hover_thumbnail_uri": null, "external_source_id": "5482" } ```

✅ All checks successful :tada:

All Cookiecutter Parameters

{
 "municipality": "The City of Alameda",
 "governing_body_type": "city council",
 "municipality_slug": "alameda",
 "python_municipality_slug": "alameda",
 "infrastructure_slug": "cdp-alameda-d3dabe54",
 "maintainer_or_org_full_name": "phildini",
 "hosting_github_username_or_org": "CouncilDataProject",
 "hosting_github_repo_name": "alameda",
 "hosting_github_url": "https://github.com/CouncilDataProject/alameda",
 "hosting_web_app_address": "https://councildataproject.github.io/alameda",
 "firestore_region": "us-central"
}

Steps for Internal CDP Team

To proceed with the deployment process, please do the following:

More details on the make commands can be found in cdp-backend.

This comment was written by a bot!

github-actions[bot] commented 2 years ago
Generated Infrastructure Slug

cdp-alameda-d3dabe54

This comment was written by a bot!

evamaxfield commented 2 years ago

Hey @phildini Thanks for opening this issue! Excited at the potential for deploying in Alameda!

I am attempting to rerun the bot just to see if we get the weird city_council bug again, but on inspection of the actual data, the original bot report of "we have everything we need except the video links" does seem accurate.

Here is the data we received from Legistar for the last month of meetings for example: alameda.md -- JSON blob as markdown because GitHub can't handle JSON upload

What that basically means is: we can easily pull information about the event dates and times, the minutes items, the body, and a few more items.

Interestingly, I don't think the Alameda Legistar stores voting records or councilmember information. Which is quite frustrating because even after a few minutes of searching I can't find any city council voting records for Alameda. What is with cities and not having council member voting records easily accessible. :exploding_head:

The big item that is missing is the "video link" but I see that Alameda follows the same pattern as our existing King County deployment which is basically: "Legistar data includes a link to the "event page" and the event page has a link to the "video page" and you can get the video from there.

Example:

Our King County scraper does basically that process and you can see that code here: https://github.com/CouncilDataProject/cdp-scrapers/blob/main/cdp_scrapers/instances/kingcounty.py#L83


Why am I telling you all of this:

If you want to deploy a CDP instance for Alameda and for us to cover it, you would need to basically fill in that function I linked for King County but for Alameda (honestly you could probably copy and past a lot of the code in the King County scraper file kingcounty.py as a good starting point). (cc @dphoria this is where we should try to generalize this workflow, if the returned EventInSiteURL is populated, we should try this automatically potentially?)

Once that is done, we can deploy a new instance of CDP for Alameda which would have:

but it would be missing:

All in all, Alameda should be decently easy to spin up and get running if you are interested in making the scraper and CDP would definitely be able to host and help get you setup.

Let me know what you think and thanks for even stopping by and making an issue!

phildini commented 2 years ago

Hey! Thanks for getting back to me.

So, hilariously, they keep the votes in the PDF of the minutes (see "Minutes" doc linked here) but they don't put them into Legistar, I bet because they don't know how

I'm happy to take a stab at making the scraper, where would I make a PR for that?

evamaxfield commented 2 years ago

I'm happy to take a stab at making the scraper, where would I make a PR for that?

Awesome! Feel free to make a PR to cdp-scrapers and place it in the "instances" sub-module.

So, hilariously, they keep the votes in the PDF of the minutes (see "Minutes" doc linked here) but they don't put them into Legistar, I bet because they don't know how

Oh interesting. I didn't even open those because it looked like they were posted very delayed. But that makes sense I guess. Still sad there isn't just a single "show me the voting records for this person" easily findable? (Again, I don't live in Alameda so maybe I just missed something) -- and I guess that's somewhat what CDP is trying to do anyway :joy:

The good note then is that because they do store them, if you wanted to get an Alameda scraper and CDP instance setup then we can do that now, and if in the future you wanted to get that info in as well then you would just need to update the scraper to pull and injest that info too.


Anyway, happy to help with that scraper over GitHub issue back and forth or over Slack: https://join.slack.com/t/councildataproject/shared_invite/zt-112g7qylg-I0B5_AvtLvER9gFwP0je9Q

evamaxfield commented 2 years ago

Note to self- this deployment request fixed a bug in the validation bot: https://github.com/CouncilDataProject/cookiecutter-cdp-deployment/commit/9a1b3c968d64e76bc4aa34bd4bd12499f37deb9e

evamaxfield commented 2 years ago

@phildini due to some recent work by @dphoria, we can now fully deploy this with ease. Are you still interested? I can fully deploy it later today.

phildini commented 2 years ago

I am definitely still interested.

evamaxfield commented 2 years ago

Will attempt to deploy later today then!

evamaxfield commented 2 years ago

/cdp-deploy

github-actions[bot] commented 2 years ago

Deployment Status

:tada: :tada: Repository Created :tada: :tada:

A new CouncilDataProject Instance Repository was created (CouncilDataProject/alameda), external collaborator added (@phildini), and cookiecutter files generated and pushed to repository.

The instance is setting itself up right now and the process will take around 10 minutes to complete. Once completed, a CDP maintainer will comment on this issue with your instance's website link. See the instance's GitHub Action job history for more details on the deployment setup progress.

Your CDP instance will be populated with data within 6 hours of website creation.

At any point in the future if you would like to destroy this instance, please just add a comment to this thread and a maintainer will help you.


Steps for Internal CDP Team

Final Setup
Deletion Steps (Future Reference)

More details on the make commands can be found in cdp-backend.

evamaxfield commented 2 years ago

Deployment Status - Complete

Website: https://councildataproject.org/alameda Initial Data Backfill (just the last couple of days): https://github.com/CouncilDataProject/alameda/runs/6632548845?check_suite_focus=true

Data backfill should finish processing in an hour or so. Feel free to backfill more data if you would like to.

evamaxfield commented 2 years ago

Highly recommend reading the: admin-docs for further setup / customization.

phildini commented 2 years ago

Beautiful, thank you! I'll review soon.

evamaxfield commented 2 years ago

Seems like you may want to add a couple of "ignore minutes item patterns" to the scraper init: https://github.com/CouncilDataProject/alameda/blob/main/python/cdp_alameda_backend/scraper.py#L18

LegistarScraper docstring: https://github.com/CouncilDataProject/cdp-scrapers/blob/main/cdp_scrapers/legistar_utils.py#L567

image

From: https://councildataproject.org/alameda/#/events/9ebe703133a0