ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
I read from someone from your team that not everything is query-able because of performance issues if everything is indexed. That makes sense in general. However, have you looked into MongoDB? It is a big data database (distributed) for JSON-like objects. Could it be feasible to use that as a back-end for all configuration/metadata-like data and it create a single index? In this spirit, maybe it would make sense to have far fewer config-concepts, but some that are powerful and merge the current concept's abilities? (Just food for thought.)
I read from someone from your team that not everything is query-able because of performance issues if everything is indexed. That makes sense in general. However, have you looked into MongoDB? It is a big data database (distributed) for JSON-like objects. Could it be feasible to use that as a back-end for all configuration/metadata-like data and it create a single index? In this spirit, maybe it would make sense to have far fewer config-concepts, but some that are powerful and merge the current concept's abilities? (Just food for thought.)