search

hasadna / knesset-data

APIs and documentation to allow getting data from the Israeli Parliament (Knesset)
MIT License
10 stars 12 forks source link

Committee members data is wrong / not updated #133

Open shevyk opened 7 years ago

shevyk commented 7 years ago

e.g: the Legislative Committee knesset. gov: http://main.knesset.gov.il/Activity/committees/huka/Pages/default.aspx image

oknesset: https://oknesset.org/committee/5/ image

MKs Tali Polsakov and Michael Malkieli are members but are not on Oknesset MKs Yoav ben Zur and Rachel Azaria appear as a members on oknesset but are not.

alonisser commented 7 years ago

@OriHoch @shevyk Is this automatically scrapped? I don't think so.. I think currently this is a manual update done by someone

OriHoch commented 7 years ago

@RavivBarzilay is working on getting this data form the new APIs

OriHoch commented 7 years ago

related issues:

alonisser commented 7 years ago

@OriHoch @shevyk There is a bigger modeling problem here, I keep nudging about , just scraping won't solve this.. the problem is that committee members can change during a knesset period, Also there are a couple of statuses besides membership (replacements for example) while our modeling does not allow for membership periods in committees , so currently we always display wrong data. either we display mks that were and currently are not or the other way around..

Solving this should handle the modeling

OriHoch commented 7 years ago

@alonisser once the relevant tasks I wrote above will be done we will have access to the data from the new knesset api the new knesset api supports this model - it provides start and end time for each person position this will then be available in the datapackage zip and in the django models (in a new model, because current model as you say don't support it) this can be then be used directly from the datapackage or via Re:Dash

Anything we should change in the above plan ^ ? just trying to be practical here :+1:

alonisser commented 7 years ago

Actually I'm still don't understand separating models from Oknesset.. I thought the datapackage thing was about separating data and passing scraped data in predefined package. Separating models and models logic from the app does not look very good for me, and the whole inlined imports on every instance method in Oknesset does not look very pythonic to me. I view the missing tests (I guess since there is no point of testing them outside of oknesset app) as a hint that this separation isn't correct, since tests are supposed to be package contained.

But since this is the current implementation and ori wants to be practical: We need a new model of committee membership (start, end, mk, committee, type) , and changing the views to get and display according to committee membership . From the top of my head I don't think we need to change Committe model fields but change adapt some instance methods do need to change, including admin and admin edit screens for adding/removing a member from committee.

I can write the the needed changes in Oknesset, once I know the data structure retrieved

OriHoch commented 7 years ago

would be best to discuss in person what's the goal with Knesset-data separation. the darapackage is only one goal, there are others

regarding the inline imports - you are absolutely correct, it's bad and supposed to be temporary. the end goal is to have knesset-data-django as a complete django package which contains the knesset data DB, different frontend implementations could use this, preferably via api (Open Knesset website being one such frontend)