Closed joepio closed 4 years ago
Here are some results of the Agenda/Meetings for a specified month, aggregated per index:
Some conclusions based on oct 2018:
The indices in both V0 and V1 as of this moment. Beware that the index ori_combined_index_new_migrated
contains everything double, so make sure to exclude it from comparison.
I've used Zwolle oct 2018 meetings as test case to see which data in V1 is still missing. I tried extracting only Zwolle in V1, but without result. However, in local development I have managed to extract the 36 meetings that were missing, so it must be possible in production too. Most likely there is a problem in the ETL on production causing gaps like these. Hopefully fixes in #147 might help to prevent these gaps.
This comparison should be repeated when Neo4j has been replaced by Postgres (#175)
The new import has started. When it has finished, we will continue working on this comparison.
Current results for V1:
2018 in V1 seems to be very incomplete cause it has been partially processed. Focus has been on completing 2019. V0 shows a steep decline after the first three months of 2019, this is due to the fact that V0 stopped importing new stuff. Comparing to the first months of V0 all Agenda's per month (see earlier) we see that V1 contains about half the numbers of V0. In V0 the normal minimum counts start around 600 meetings averaging 850, except for summer recess months (holidays) which average at 200.
On 5th of June 2019 the total count for V1 Meeting's feb 2019 was 383 and has increased now to 505, however the equivalent count for V0 for that period was 771. On closer look it seems that V1 is actually has more complete municipalities (Maastricht V1: 55 vs V0: 38). However, as remarked earlier on the 5th of June, quite some municipalities are still missing! Except for Hilvarenbeek, the municipalities: zwolle, arnhem, enschede, utrecht are still not to be found in Februari. Municipality count Februari V0: 113, V1 back then: 53, V1 now: 63. This does not mean that these lacking municipalities are completely missing for all months, zwolle for example currently only shows (a few) meetings for may and june 2018.
The missing municipalities could relate to #229 , since iBabs isn't running. Are the missing municipalities iBabs municipalities @jurrian?
It certainly looks that way, the named municipalities above are all iBabs municipalities. The lack of those on June the 5th suggests that the problem in #229 persisted longer than we thought. When this problem is fixed, I predict that V1 will be more complete than V0 for the months that have been imported.
Hi there! We want to go live with the implementation of ORI within 1848.nl on monday. But I have a questions about the current ORI API and the further development. 1) If I search within ori.argu.co there are hardly any results for Utrecht or Provincie Utrecht. How is this possible and will this improve shortly? 2) The API currently contains 118 municipalities and 5 provinces. When will the database be complete? 3) Can you give information about what documenttypes are included? For example: collegebrieven, stemmingsuitslagen, debatverslagen, etc
Many thanks! PS: I'm new to Github so correct me if I fail to apply the etiquette :)
Hi Taluke,
Good luck with the launch!
Thanks joepio :) If I can help with pushing the local governments I'll be happy to do so. I bet the iBabs fix will not be ready before monday?
And another question: How far do the API-documents go back?
Maybe the ibabs fix will be implemented this week, but the documents still need to be imported. This will take more time. However, I'll let the devs know that the most important import task is the daily one, so your notification system will work as soon as possible!
Tell governments to get in touch with the ORI project lead from VNG, Sander Bakker. Sander.Bakker@vng.nl
We'll be importing as far back as we can! We don't know how far that is yet :).
On Thu, Sep 12, 2019, 11:27 Taluke notifications@github.com wrote:
Thanks joepio :) If I can help with pushing the local governments I'll be happy to do so. I bet the iBabs fix will not be ready before monday?
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/openstate/open-raadsinformatie/issues/146?email_source=notifications&email_token=AAQVBEJITREEAUYZN66GBSDQJIKQ3A5CNFSM4HTVUDQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RNHZA#issuecomment-530764772, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQVBEJOW7ODW5OK2JMGRS3QJIKQ3ANCNFSM4HTVUDQA .
I'm glad to hear you're on top of things @joepio!
For our work on HierOverheid we were also wondering when the v1 collection would become more complete. We'll keep working with the documents we can currently access and hope to know soon how our system will do on the as-of-yet unloaded docs.
Since we're primarily focused on the provincial docs, we currently don't have very much to work with:
green open osi_zuid-holland_20190809125059 llsuyALZSTqSpzFOGLv3Lg 1 1 441 0 6.1mb 3mb
green open osi_limburg_20190809125018 Ck9RDxk1Ty2J671pLW7tNQ 1 1 38 0 71.5kb 35.7kb
green open osi_flevoland_20190809125113 MLp1DKVbQteFdbg5dg2WIA 1 1 52 0 256.1kb 138.3kb
green open osi_noord-holland_20190809125038 PS8mYzsnTh2legb4st0CGA 1 1 2 0 49.6kb 24.8kb
green open osi_provincie-utrecht_20190809125121 JiGLqlFMRrCnEKmnuLFTcQ 1 1 61 0 396.6kb 200kb
Could you say a bit more about why there are so few documents in these indices?
Logs indicate that some meetings take up an absurd amount of time (hours instead of a minute, even with extreme specs and instance counts). Both @jurrian and @jorritb have spent quite some time on trying to find the underlying issue, but it turns out to be very difficult to identify the root cause. I think #224 is suspicious.
If the root cause of the performance issue is solved, we can quickly increase the document counts.
We're live with ORI on 1848.nl! From now we'll monitor the user feedback. Most important is that we have to be complete. @joepio Can you give an update on the iBabs-bug? Plus: What actions are being taken to add more provinces and municipalities? I'm pretty sure our users will ask me :)
This document fails to download: https://www.raadleiderdorp.nl/api/v1/meetings/1588/documents/37265
@Taluke The iBabs bug can be followed here: https://github.com/openstate/open-raadsinformatie/issues/240. Please create a new issue for the failing document (although I suspect that it is not caused by IRU, since the link is external).
[edit: correct issue]
Performance issues have been solved, imports are 10 - 100x as fast now. This means that quantity will very soon be way better :)
Current results for V1:
In my last comment the total number of meetings was 4286, this is now 8815. The last time has been about importing 2019. Comparing the first three months 2019 of V0 all Agenda's per month we see that V1 is still missing 100-300 meetings, however the months that follow show to be much more complete than V0, up to March 2020.
For a complete comparison of the feb 2019 sample see: https://docs.google.com/spreadsheets/d/1a1C6--9H0ERPsTR45SdMF4tDVKxsvf6NNBfwJWsyI-g/edit?usp=sharing
On 9th of sept the total count for feb 2019 was 505 which has increased to 622, but still less than 771 in V0. The comparison report above shows that a total of 44 municipalities are still missing for the month feb 2019. However, this is for the feb 2019 sample, municipalities missing in this month might be included in other months. The missing municipalities are almost all iBabs (40), this has probably to do with the problems we encountered with iBabs earlier.
The problem why we are falling short is probably because of the processing capacity of the ORI system. We have a daily cronjob which starts at midnight fetching the last month and coming month, the entities are mainly meetings and reports. Persons, organizations and such are processed in the monthly cron. The daily cron seems to be taking up 19 hours already (encounting) and, as the name suggests is ran every night. But if the daily cron already takes up this amount of time, there is no time to process previous years. Therefore I am thinking about making the daily cron weekly, which has as consequence that the data will be less current.
I don't think switching to a weekly cron job is wise. That will mean that the information is not very up to date and the use for ORI is quite dependent on getting all data asap (like for the purpose of 1848.nl re-use users want to have new information when it comes up). So going to weekly will make the dataset a lot less valuable.
Isn't it an option to get more processing capacity to make the daily cron job feasible?
Also, sinceit's running every day, it ould also help to make the period shorter, Ie. only last week and next week. Or maybe next two weeks, since next is usually not that much data ... (But then again, not sure what kind of impact that has on completeness)
Thanks for your input, I agree that making the range shorter (week instead of month) might be the best solution here. We will keep the daily.
Updated comparison Februari 2019: https://docs.google.com/spreadsheets/d/1a1C6--9H0ERPsTR45SdMF4tDVKxsvf6NNBfwJWsyI-g/edit?usp=sharing
Last time not all municipalities were indexed in the Feb 2019 sample, now they are all indexed. In most occurrences V1 is now equally as or more complete than V0 in this sample, which should be reflect the rest of 2019.
It took just a few days to import the whole of 2018. Here the comparison V0 vs V1 comparison for October 2018: https://docs.google.com/spreadsheets/d/1a1C6--9H0ERPsTR45SdMF4tDVKxsvf6NNBfwJWsyI-g/edit#gid=128243340
6 municipalities/provinces appear missing, I will reprocess them manually.
Over 1.000.000 resources have been loaded into v1!
A day later Elasticsearch now contains almost 1.500.000 resources. Over the whole period of 2000-2020, V1 now contains 78754 meetings vs 48579 in V0. See the report below for the monthly comparison: https://docs.google.com/spreadsheets/d/1a1C6--9H0ERPsTR45SdMF4tDVKxsvf6NNBfwJWsyI-g/edit#gid=83625813
The problems fixed most notably in milestones v1.3.4, v1.3.5 and v1.4.1 prevented reaching this goal earlier due to really slow processing. With these problems gone, importing until 2000 appeared to be much easier.
The goal of this issue was to measure completeness of V1 against V0 to determine whether V1 could be used and V0 deprecated. At this point we can say this goal has now been reached. Therefore this issue can be closed.
Awesome! Feels good to confidently close this one :)
When we make the V1 API the default one and use ori-search on zoek.openraadsinformatie.nl, we want to make sure that the data is at least as complete as in V0. In order to prove this, we need to provide some statistics about how many items are loaded per month, per type and per municipality.