search

DigitalGlobe / gbdxtools

(Deprecated) Python SDK for using GBDX
MIT License
74 stars 57 forks source link

Update paging ttl value to '10s' for a shorter duration #809

Closed jeffreyeriksondg closed 4 years ago

jeffreyeriksondg commented 4 years ago

Due to potential stress that a long-duration ttl for Elasticsearch scan+scroll session could put on data nodes, and given the number of queries for catalog data that are done using gbdxtools, I've made a change to reduce the default duration for the sessions from 5 minutes to 10 seconds, which should be long enough for most use-cases. And if a user wants/needs a longer ttl, they can still set that in their call.

drwelby commented 4 years ago

Thanks, can you aim this PR to dev?

drwelby commented 4 years ago

GBDXtools uses this TTL to load paginated results in 1000 feature batches. 10 seconds should be enough time to persist the query between batches.

If for some reason the batch can not execute in 10 seconds, the query will reset and the pagination will start over at the beginning. The query will then reload duplicate features. If more time is needed for a certain query, the ttl kwarg can be increased.