koopjs / koop-escache

ElasticSearch cache for Koop (experimental).
Other
2 stars 1 forks source link

searches use scrolling for much win #5

Closed dmfenton closed 9 years ago

dmfenton commented 9 years ago

Instead of requesting everything at once, this pages and uses a search type of scan to avoid any scoring

Benefits:

See: https://www.elastic.co/guide/en/elasticsearch/guide/current/scan-scroll.html

dmfenton commented 9 years ago

Need to do some more profiling and optimizing

dmfenton commented 9 years ago

Tested against: 127.0.0.1:1337/agol/arcgis/780a3f7babe74f18aaebdd42dbe4a911/FeatureServer/0

Status Quo: 445,210 ms :turtle:

Batch size Response time (ms)
100 50,918
200 25,450
250 25,613
500 26,915
750 27,360
1,000 28,971
chelm commented 9 years ago

@dmfenton those response times rock. So will you rebase this with master? Then i'll merge it

dmfenton commented 9 years ago

Rolled everything into a 0.1.1 since we haven't released anything else.