search

transparentdemocracy / voting-data

Voting behavior data extracted from plenary reports of the Belgian federal government.
5 stars 1 forks source link

Improve or drop the proposal discussion fallback mechanism #46

Open sandervh14 opened 1 month ago

sandervh14 commented 1 month ago

INFO:transparentdemocracy.plenaries.extraction:55_071_d21 does not have a discussion part. Using fallback.

This occurs very often. The reason normally is that there is simply no title for a section where the "article discussion" (what we write to proposal_discussion.description_nl/fr). But it causes very long descriptions for many proposal discussions, resulting in the "walls of text" team website complains about.

Summarization in #7 and #17 will of course fix this too. And starting to capture the proposal debate (the section before the article discussion) will also fix this, as now the fallback description includes that debate, whereas for descriptions that are genuine (not fallback) does not.

sandervh14 commented 1 month ago

Example case in our unit tests: 55_298_mg_10_m0.