search

Metro-Records / la-metro-councilmatic

:metro: An instance of councilmatic for LA Metro
MIT License
6 stars 2 forks source link

Board Reports: Confirm revision to board correspondence display logic works as expected on live data #739

Open shrayshray opened 3 years ago

shrayshray commented 3 years ago

An update to the logic established in #477 for when to display Board Correspondence items - previously, the logic required these 3 fields to be set as follows: MatterTypeName: Board Correspondence, MatterStatusName: Agenda Ready, MatterRestrictViewViaWeb: False

MatterStatusName should be updated to MatterStatusName: Sent

@hancush can this please be prioritized for immediately after reCAPTCHA is complete?

hancush commented 3 years ago

Hi, @shrayshray, this is the current logic we use to determine whether to show a bill.

https://github.com/datamade/la-metro-councilmatic/blob/61f7b11e7c046e424af30ec61a77509655350354/lametro/models.py#L47-L78

tl;dr - We don't currently filter on MatterStatusName. We show all board correspondence that is not registered as private (MatterRestrictViewViaWeb is True, or MatterStatusName is "Draft" or "TO BE REMOVED").

We don't capture MatterStatusName in the scrape, so we don't have that information readily on hand to apply the filter. Before we make changes, is it possible to set MatterRestrictViewViaWeb to True for board correspondence you are not ready to show on the board agendas site? Let me know either way and we can go from there!

camachoo commented 3 years ago

@hancush

We don't currently filter on MatterStatusName. We show all board correspondence that is not registered as private (MatterRestrictViewViaWeb is True, or MatterStatusName is "Draft" or "TO BE REMOVED").

Two things to address here. 1) MatterStatusName will not show "TO BE REMOVED". "TO BE REMOVED" is located in MatterBodyName. 2) Maybe I'm looking at this wrong, but isn't checking MatterStatusName for "Draft" the same as filtering?

is it possible to set MatterRestrictViewViaWeb to True for board correspondence you are not ready to show on the board agendas site?

Yes, it is.

hancush commented 3 years ago

"TO BE REMOVED" is located in MatterBodyName.

Whoops, @camachoo, you're totally right. I misread – we check whether MatterBodyName equals "TO BE REMOVED" as a condition for marking a bill as private.

isn't checking MatterStatusName for "Draft" the same as filtering?

You can think of it as an indirect filter, sure. If the status is "Draft", the body name is "TO BE REMOVED" or MatterRestrictViewViaWeb is True, then we mark the bill as private in our database during the scrape. Then, the app filters out private bills when determining what to show on the board agendas site.

If MatterRestrictViewViaWeb is (or can be) a reliable indicator of when to show board correspondence (i.e., it is always True if the board correspondence should not be shown), then the existing logic around private bills should handle this condition. To confirm, @camachoo, is that the case?

camachoo commented 3 years ago

@hancush, My hesitation in only looking if MatterRestrictViewViaWeb is False as an acceptable solution for posting to the web is that this file type doesn't follow the normal agenda process. I feel as though there needs to be an additional layer of protection like using MatterStatusName as an extra check. It also makes posting of this type of report intentional. How difficult would it be to add this extra layer of protection?

hancush commented 3 years ago

@camachoo Got it. It shouldn't be too tough. Since we aren't currently importing MatterStatusName, I'd suggest making the change in the scraper. Namely, I'd add a condition to mark a bill as private if:

Does that seem right to you?

shrayshray commented 3 years ago

@hancush that seems right to me!

hancush commented 3 years ago

Great, thanks, @shrayshray! I'll draft something in the morning and let you know when it's staged.

hancush commented 3 years ago

@camachoo @shrayshray Ok, I've staged this change in scrapers that populate the staging instance. When would you two like to do test data entry?

camachoo commented 3 years ago

Let's test Friday morning. I'll create three test files to make sure only the expected one single report shows.

Omar

From: hannah cushman garland @.> Sent: Thursday, June 24, 2021 9:00:26 AM To: datamade/la-metro-councilmatic @.> Cc: Camacho, Omar @.>; Mention @.> Subject: Re: [datamade/la-metro-councilmatic] Update to logic for displaying Board Correspondence (#739)

@camachoohttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcamachoo&data=04%7C01%7Ccamachoo%40metro.net%7C12c45e45c39646a55db808d937292e9b%7Cab57129bdbfd4cacaa77fc74c40364af%7C0%7C0%7C637601472289586030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g7KeaHRT485bybX7Bm%2FieScFATfNp91Yq%2BKAlb7kaWo%3D&reserved=0 @shrayshrayhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshrayshray&data=04%7C01%7Ccamachoo%40metro.net%7C12c45e45c39646a55db808d937292e9b%7Cab57129bdbfd4cacaa77fc74c40364af%7C0%7C0%7C637601472289586030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NXD0FIX9Nd%2F4borzWNO3PiVS8kuAZ8aXxRxLZ%2FhbYbc%3D&reserved=0 Ok, I've staged this change in scrapers that populate the staging instance. When would you two like to do test data entry?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdatamade%2Fla-metro-councilmatic%2Fissues%2F739%23issuecomment-867760002&data=04%7C01%7Ccamachoo%40metro.net%7C12c45e45c39646a55db808d937292e9b%7Cab57129bdbfd4cacaa77fc74c40364af%7C0%7C0%7C637601472289595987%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aETCvsXSHL57dIc9sqthRmOQImKDgBOU40PGRQDDUik%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAG3PL6XUFAB7TPTN4NCUEHDTUNJBVANCNFSM47CPFTYQ&data=04%7C01%7Ccamachoo%40metro.net%7C12c45e45c39646a55db808d937292e9b%7Cab57129bdbfd4cacaa77fc74c40364af%7C0%7C0%7C637601472289595987%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lTnSD%2BYBrpQelCmNLj2qt%2FhOUwnZWtoEasIQizf2A9Q%3D&reserved=0.

shrayshray commented 3 years ago

@camachoo aye, aye!

camachoo commented 3 years ago

@hancush , @shrayshray, Last night I setup some files TMP-0101, TMP-0119, and TMP-0121 for this test.

Situations: TMP-0101 - Illustrates an accidental marking of MatterRestrictViewViaWeb is False. TMP-0119 - Illustrates an intentional release of a Board Correspondence with a workflow approval that results in MatterStatusName is Sent and MatterRestrictViewViaWeb is manually set to False. TMP-0121 - Illustrates an incomplete release of a Board Correspondence with a workflow approval that results in MatterStatusName is Sent but MatterRestrictViewViaWeb is still True.

So, only TMP-0119 should make it to the web.

hancush commented 3 years ago

Looks like we've got 0119: https://lametro-upgrade.datamade.us/board-report/tmp-0119/

But not the others:

🎊

hancush commented 3 years ago

The change in scrape logic was deployed.

hancush commented 3 years ago

Board correspondence has not yet been added. I'll open a separate issue as a reminder to touch base on this logic once it is added.

hancush commented 3 years ago

Actually, I just updated the title here. Moving back into the icebox until data comes in.