search

mysociety / popit

DEPRECATED - Development on PopIt has stopped and it is no longer being maintained
https://goo.gl/Vvej4Q
Other
76 stars 33 forks source link

inconsistent results from day to day from search queries - indexing failing #781

Closed mhl closed 9 years ago

mhl commented 9 years ago

A comment from @andylolz:

this search showed ~690 results yesterday: http://yournextmp.popit.mysociety.org/api/v0.1/search/persons?q=identifiers.scheme:%22uk.org.publicwhip%22

... but today is return 440 results. Some other searches aren't returning expected results, e.g. this finds no one, but should be returning Hazel Blears:

Hazel Blears is also an example of someone who should be being returned by the previous search.

Some other name searches are still working, e.g.:

The Elasticsearch logs show lots of errors in trying to index people caused by running out of heap space, e.g.:

org.elasticsearch.index.engine.IndexFailedEngineException: [popit_yournextmp][1] Index failed for [person#6415]
        at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:498)
        at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:341)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:521)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.OutOfMemoryError: Java heap space

... which would explain this. However, @Flupsy has upped the amount of memory that's available to Elasticsearch several times recently, so I suspect something more fundamental is going wrong - see also the reindex of the South Africa PopIt instance, which is failing every night at the moment: https://github.com/mysociety/popit/issues/753

mhl commented 9 years ago

I think this has been fixed for the moment with @Flupsy increasing the memory available to Elasticsearch again and me reindexing the YourNextMP instance, although the underlying issue may well be: https://github.com/mysociety/popit/issues/782

andylolz commented 9 years ago

Btw that the search above is now returning 668 results instead of 690 is nothing to worry about (My 690 figure was from remembering approx 23 pages.)