Open evamaxfield opened 2 years ago
What would be appropriate for matter full_text_uri
, in this example: (I'm presuming the corresponding ingestion_models.Matter
must change as well)
https://seattle.legistar.com/MeetingDetail.aspx?ID=930274&GUID=903D2508-9840-4878-8334-1AEF77335BB8
https://gist.github.com/dphoria/3134769fe44686a82fdca2a55b822397
I will take a look later myself. Just wanted to start the question / conversation.
Great question! Yes the ingestion model would need to be updated as well to add the same property / attribute.
Taking this meeting from legistar: https://seattle.legistar.com/MeetingDetail.aspx?ID=929921&GUID=3EB77948-2243-425A-9864-8CD868B96048&Options=&Search=
And selecting the first council bill (CB 120263), we get to: https://seattle.legistar.com/LegislationDetail.aspx?ID=5448143&GUID=4F8010D6-BEBB-46AF-BE22-F579AD681B68&Options=&Search=
I think what we want really just a link to that page / that above link since it has the full details. But if we wanted to get even more specific, I would say clicking "Reports" and then clicking "Legislation Text" or really any of the options gives us more of a "document view" like this: https://seattle.legistar.com/ViewReport.ashx?M=R&N=Text&GID=393&ID=4717976&GUID=660120D3-9C6F-4314-AFC7-A44217E71237&Title=Legislation+Text
This is really a bigger deal because like.... currently we don't even store that info to CDP at all, here is the corresponding meeting page for that meeting on seattle staging: http://councildataproject.org/seattle-staging/#/events/f3351cc9822f
notice that the minutes item CB 120263 doesnt have any attachments / documents.
Do we need a separate field for the full text, or could it just be another MatterFile
? If we want to handle the full text differently in the UI than other MatterFile
s than I'm all for adding full_text_uri
, but otherwise I think it could be another MatterFile
I think the above try-except block may be dropping some MatterFile / MinutesItemFile attachments that would be useful to keep and so we may want to try to fix it if we do see that behavior.
For this it's most likely failing due to a connection timeout or an error when making an http request. Since the only validation run on MatterFile
is resource_exists
, I think it has to be one of these two
Do we need a separate field for the full text, or could it just be another
MatterFile
? If we want to handle the full text differently in the UI than otherMatterFile
s than I'm all for addingfull_text_uri
, but otherwise I think it could be anotherMatterFile
I guess we could add this as a MatterFile
but there I feel like we would need to add an attribute of type
or something? Something to signify what each MatterFile
represents (i.e. just a report, an amendment, or the bill text)
I guess we could add this as a
MatterFile
but there I feel like we would need to add an attribute oftype
or something? Something to signify what eachMatterFile
represents (i.e. just a report, an amendment, or the bill text)
Since the full text uri is kinda distinct from other MatterFile
's, I think we could just add full_text_uri
to Matter
(also saves us a query if we want to fetch this for a specific Matter
).
Unless there are very discrete categories that we could classify MatterFile
into, I don't think we need MatterFile.type
and name
would be sufficient.
Yea the benefit to query time is also a major plus.
While looking through the frontend and design work on the legislation tracking project, I realized for the first time that I think we may be missing a crucial piece of information which is a link to the actual full matter text.
Current
Matter
is defined as: link but a matter definitely has full text and we should store a link to that full text. I proposefull_text_uri
or some varient of that.Additionally, while we look into this, it would be good to investigate which
MatterFile
's make it through the pipeline: https://github.com/CouncilDataProject/cdp-backend/blob/main/cdp_backend/pipeline/event_gather_pipeline.py#L1444I think the above try-except block may be dropping some
MatterFile
/MinutesItemFile
attachments that would be useful to keep and so we may want to try to fix it if we do see that behavior.