search

openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Stream ordering. Fixes #116 #117

Closed mlandauer closed 4 years ago

mlandauer commented 4 years ago

This PR does two things to fix the problem:

  1. Creates a single place where the http requests are made. This is so that we, as much as possible, maintain the relative ordering of log outputs in stdout and stderr.
  2. Speed up the create event API call on the server by getting rid of a call to the kubernetes API. This is to ensure that the queue for the http requests doesn't get filled up with a backlog of events that haven't been sent yet. If this happens we run in to the same ordering problem as before.