search

okestonia / opendata-issue-tracker

Community effort to open up more and more data in Estonia
http://opendata.ee
48 stars 5 forks source link

API / dataset for all Riigikogu votes #60

Closed taivop closed 5 years ago

taivop commented 7 years ago

The data is already available on riigikogu.ee, but only as websites that need to be scraped to get a full dataset.

I'd like to see, at minimum:

And if possible:

(this seems somewhat related to #30 but I think that issue doesn't cover the voting data)

andreskytt commented 7 years ago

I have seen the data scraped, trying to track it down

taivop commented 7 years ago

I was in contact with Riigikogu kantselei (Janek Solodin) and:

So I guess this issue is low priority now.

taivop commented 7 years ago

Votes: haaletused.json.gz. Download: mega.nz (6.2MB)

taivop commented 7 years ago

Transcripts: stenogrammid-13_04_17.sql.gz. Download: mega.nz (172MB)

tormi commented 7 years ago

they are planning to release a JSON API

@taivop, by using http://jsonapi.org/ ?

The agencies current practice of implementing API's is very inconsistent and to my knowledge there is no clear rule or even advice from RIA nor MKM about this. @andreskytt , what do you think can be done to improve this?

boamaod commented 7 years ago

We already played with the data a bit:

ottomattas commented 7 years ago

Just opened the API's test-environment to the public. It is only temporarily up and the documentation on it is minimal but I hope you can find something worth looking at.

https://aavik.riigikogu.ee

tormi commented 7 years ago

re: https://aavik.riigikogu.ee/api/ems/v1/raml/ems-api-v1.html

Teeme veel viimaseid parendusi lõpplahendusele ning andmestruktuuridele. Kui on huvi koostööd teha või mõtteid jagada, kirjuta vastutavale süsteemihaldurile aadressil otto.mattas@riigikogu.ee.

@ottomattas , have you considered using JSON API spec?

ottomattas commented 7 years ago

@tormi, I have. There are some concerns, though. We are trying to work with RIA on this matter and they today implicitly suggest using OpenAPI Specification which in turn may not be fully compatible with JSON API spec yet. (Priit Parmakson's not yet finalised guide for national API's: https://agiil.github.io/IT/API#api-spetsifitseerimine). The proposed usage (or scope of it) of Google API's design guide for Estonia's Open Data solutsions is another matter altogether.

But the idea of using JSON API is totally worth a mention. I will be taking another look at it when official JSON Schema is added to OpenAPI 3.0 repo.

boamaod commented 7 years ago

@ottomattas, there is probably some historical reason for having no formal agendas before 2012 autumn?

Compared to:

ottomattas commented 7 years ago

@boamaod, yes. Said data has different data structure comparing to other formal agendas (after autumn 2012) because of the development of internal systems at that time. It needs some work standardising the data to be decently published on the parliament's webpage (https://www.riigikogu.ee) and effectively through the API.

We are working on data standardisation as we speak, at the moment voting data is in focus. Hopefully, formal agendas get the treatment as well. (No plan to do so has been set yet, though.)

boamaod commented 7 years ago

@ottomattas, our hackathon experience was that agenda links together most of other data and this might be a reason for prioritising. :)

tormi commented 7 years ago

@ottomattas, Is limit parameter working for all resources? https://aavik.riigikogu.ee/api/ems/v1/document-hierarchy-tree?limit=1 seems to ignore it. And can I use pagination?

tormi commented 7 years ago

@ottomattas, can I fetch relationships in a consistent way? For example, _give me user's events_.

ottomattas commented 7 years ago

@boamaod, thanks for the feedback. This is really valuable input, we'll keep it mind. 👍

@tormi, at the moment limit parameter is only working for specific resources as we are still working on standardising the solution. Also, /document-hierarchy-tree gives only one result so there is no point in adding limit/skip etc explicitly for the parameters' functionality. Holding up to the standard is a legitimate reason, though.

I will introduce @indrekmaask as the developer for the parliament's open data project. Hopefully he can answer any technical questions (including the queries from @tormi about pagination and fetching relationships) more specifically.

As it seems there's an interest for parliament's open data project in general, maybe I should start an issue tracker of our own to keep track of the proposed ideas and related issues?

Hillehinsberg commented 7 years ago

@ottomattas and @indrekmaask - thanks for the work done and the input here! I suggest to keep the dialogue going on here, as lond as the interested users have already found this track. The link was disseminated to all Garage48 participants and wider group of those who are potentially interested in similar issues.

boamaod commented 7 years ago

@ottomattas, having special Riigikogu label to sort out relevant issues might be enough? I see that it already exists with one issue attached, added also to current issue.

tormi commented 7 years ago

@ottomattas, thanks!

maybe I should start an issue tracker of our own to keep track of the proposed ideas and related issues?

Yes, please! This place should only deal with onboarding and status of open data availability. When open data is available, we should point users to data and label issue accordingly ("Open data available" at https://github.com/okestonia/opendata-issue-tracker/projects/1). @Hillehinsberg, could you please label this issue accordingly? If data provider has it's own issue tracker, we should close the issue here and provide the path to app's tracker.

See also CONTRIBUTING.md (point 3) and #71.

indrekmaask commented 7 years ago

Hey all! I'd like to point out that the current API is written specifically for the website and not for public use. There's a lot to be done in order to conform to proper public API standards, starting with documentation, better defined resources, data presentation etc.

Few things are certain: 1) It will be a JSON REST API with hypermedia support (HATEOAS). 2) Written in Java (Spring technologies). 3) Documentation will be a mix of automatic and handwritten text (Spotify as an example: https://developer.spotify.com/web-api/endpoint-reference/). Possible tools: Swagger (http://swagger.io/), Spring REST Docs (https://projects.spring.io/spring-restdocs/)

boamaod commented 7 years ago

@ottomattas and @indrekmaask, it might be smart to use (or at least try to mimic) some of the existing standards for decision body data representation APIs, since it might make implementing 3rd party tools/code much easier. I'm not terribly aware of such standards, but looking at Finland as leader of Open Data in our region might be a good start. For example in 2016 six Finnish municipalities created a common standard exactly for that, it might be useful to discuss their experience before making final decisions. Sharing tools/code on shores of Finnish Gulf would be really promising option for governments/municipalities as well as journalists, data analysts, application developers, activists etc.

ping @juyrjola

ottomattas commented 7 years ago

Thanks for the input! You have helped us forward with the technical aspects. Also, showing active interest in what we do and are trying to accomplish, gives our organisation much needed attention regarding open data - this shows decision makers the subject is really relevant and worth pursuing. Hopefully the collaboration leads to something wonderful in the end!

Let's keep the momentum going: https://github.com/riigikogu-kantselei/api (there's no code yet, only a public issue tracker).

tormi commented 7 years ago

Time to close this one?

ottomattas commented 7 years ago

I believe so as the initial issue seems to be solved. Datasets requested in the issue are available.

Also, anybody interested in further developments can make their way to the parliament's repository for Open data API as said before.

infokujur commented 5 years ago

Palun uuendada infot opendata.riik.ee https://opendata.riik.ee/teabevaldajad/ @marmoluik Siis saame lõimu sulgeda.

marmoluik commented 5 years ago

Andmed on opendata.riik.ee keskkonnas uuendatud @infokujur

infokujur commented 5 years ago

@marmoluik Väga hea! Kas nüüd tulevad nad juba otse, või on seal veel ca kuu pikkune vahefaas? Kui nii, siis võiks dokumentatsioonis sellele viidata. Ja kas see on kõikide väljade osas või ainult dokumendiregistri osas vms

marmoluik commented 5 years ago

@infokujur Kuna nüüdseks on API lives, siis tegemist on live andmetega.

infokujur commented 5 years ago

Originally posted by @janesmae in https://github.com/riigikogu-kantselei/api/issues/13#issuecomment-446573489 @marmoluik Ehk see küsimus on ka lahendatud?

janesmae commented 5 years ago

Originally posted by @janesmae in riigikogu-kantselei/api#13 (comment) @marmoluik Ehk see küsimus on ka lahendatud?

Jah. Riigikogu Kantselei avaandmete API on nüüd toodangus ja asub: api.riigikogu.ee (vastavalt infole Avaandmete portaalis).