search

reddit / rollingpin

fast deploy to lots of servers
https://redditblog.com/2017/06/02/the-evolution-of-code-deploys-at-reddit/
Other
145 stars 33 forks source link

elasticsearch: don't json load doc id on failure #50

Closed foklepoint closed 3 years ago

foklepoint commented 6 years ago

This addresses the issue where ES would be down and not return a doc id, but we still attempted to read the returned response even though the response code was not valid

foklepoint commented 6 years ago

💇‍♂️

rram commented 6 years ago

My understanding of the issue is that Elasticsearch is hanging until a gateway timeout happens. This would fix the crash after the timeout but it'll still hang for a long time.

foklepoint commented 6 years ago

My understanding of the issue is that Elasticsearch is hanging until a gateway timeout happens. This would fix the crash after the timeout but it'll still hang for a long time.

This is indeed true :)

foklepoint commented 6 years ago

💇‍♂️ now we need to default deploy_annotation_id as None as well, otherwise on_build_sync fails

foklepoint commented 6 years ago

Couldn't find an easy way to adjust the timeout of reactor for ES :/

spladug commented 6 years ago

In our version of twisted the way to do a timeout is to do reactor.callLater(thetimeoutvalue) and then join on that deferred and the HTTP deferred at the same time. Whichever fires first is the winner. Cancel the other.

foklepoint commented 3 years ago

Closing for now, not getting a lot of use out of elasticsearch anymore anyways