Closed sasharevzin closed 4 years ago
@matschaffer Is it something we can do?
Yep, and it's something I'd like to do.
All the fixed measurement sources (pointcast, solarcast, etc) already go into elasticsearch (this data path https://docs.google.com/presentation/d/18RDRvmfqSg3fvxhU4Aw4yKs_9OdHCTBfR-Ace7KrmpI/edit#slide=id.p)
The main things that's missing is the bgeigie drive uploads.
If you want to add a step to processing that puts them in ES, I'd be fine with that.
One thing to watch out for is that in that diagram: note that ttserve publishes to both ingest and api (via POST /measurements).
I don't think we should send POST /measurements
to ES from api right now, otherwise we'll end up storing every measurement in ES twice.
I'd start with the drives, that will get most of our data into ES.
Once #606 is completed, we can look at removing the POST /measurements
publishing.
Closing this as I suspect you have your answer, if you want to open a follow up issue to publish drive upload data to elasticsearch, I'm all for that.
Sounds good. Thanks
I would suggest sending all measurements to elastic search for faster searches as we do in https://github.com/Safecast/safecastapi/blob/master/app/models/ingest_measurement.rb#L5