search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
364 stars 132 forks source link

Add variant and metrics to events_plot mapping #223

Open fhaase2 opened 7 months ago

fhaase2 commented 7 months ago

We had some issues with duplicated plots in our ClearML deployment.

The reason was that Elastiscearch infered text for these fields and we do an aggregation query.

This PR sets them explicity to type keyword, which should make it more stable given that field inference works slightly different in different Elasticsearch versions.

Ponsaille commented 6 months ago

Happy new year! Were are we on this PR, would be really nice to have this additional stability! @jkhenning

jkhenning commented 5 months ago

Hi @Ponsaille, sorry for the delay 🙏 How would this work is you only add these to the schema? The server won't store them there, I think

evg-allegro commented 5 months ago

Hi @Ponsaille , @fhaase2 which version of ES do you use? We already have this mapping in events.json file and it is applied to all the events indices. While more specific mappings just add missing fields for specific event types. Thus the resulting indices have the mapping that is combination of events.json and then a more specific one like events_plot.json or events_log.json etc. This works for us in ES versions from 5.x to 7.x including. When you restart the apiserver container the mappings are applied as part of the apply_mappings script. Do you see any errors from it in the apiserver log?

fhaase2 commented 5 months ago

We use ES 8.5.1. And per default, the indices created automatically by ClearML have the wrong mappings in this version. So this PR makes sure that the created index has correct mapping, even in ES > 7.5.