Safecast / safecastapi

The app that powers api.safecast.org
44 stars 25 forks source link

The measurements search improvements #679

Closed sasharevzin closed 4 years ago

sasharevzin commented 4 years ago

I would suggest sending all measurements to elastic search for faster searches as we do in https://github.com/Safecast/safecastapi/blob/master/app/models/ingest_measurement.rb#L5

sasharevzin commented 4 years ago

@matschaffer Is it something we can do?

matschaffer commented 4 years ago

Yep, and it's something I'd like to do.

All the fixed measurement sources (pointcast, solarcast, etc) already go into elasticsearch (this data path https://docs.google.com/presentation/d/18RDRvmfqSg3fvxhU4Aw4yKs_9OdHCTBfR-Ace7KrmpI/edit#slide=id.p)

The main things that's missing is the bgeigie drive uploads.

If you want to add a step to processing that puts them in ES, I'd be fine with that.

One thing to watch out for is that in that diagram: note that ttserve publishes to both ingest and api (via POST /measurements).

I don't think we should send POST /measurements to ES from api right now, otherwise we'll end up storing every measurement in ES twice.

I'd start with the drives, that will get most of our data into ES.

Once #606 is completed, we can look at removing the POST /measurements publishing.

matschaffer commented 4 years ago

Closing this as I suspect you have your answer, if you want to open a follow up issue to publish drive upload data to elasticsearch, I'm all for that.

sasharevzin commented 4 years ago

Sounds good. Thanks