search

mozilla-ai / lumigator

Source code for Mozilla.ai's Lumigator platform
https://mozilla-ai.github.io/lumigator/
Apache License 2.0
63 stars 7 forks source link

Enable dummy experiment endpoint #377

Closed javiermtorres closed 1 day ago

javiermtorres commented 1 week ago

What's changing

An experiment endpoint reusing the jobs service is made available.

How to test it

N/A for the moment

Additional notes for reviewers

N/A for the moment

I already...

ividal commented 4 days ago

By the way, we're going to need to introduce some tests for this one - current tests (not even integration ones) do not cover this functionality.

javiermtorres commented 4 days ago

By the way, we're going to need to introduce some tests for this one - current tests (not even integration ones) do not cover this functionality.

Roger and wilco :)

veekaybee commented 4 days ago

I'd like to better understand the use-case for this. Generally, it would make sense to write a separate experiment service and we decide on how we'd like tracking to work at the experiment level. I'd be hesitant at this stage to mix jobs and experiments since they're different levels of hierarchy at the moment.

ividal commented 4 days ago

I'd like to better understand the use-case for this. Generally, it would make sense to write a separate experiment service and we decide on how we'd like tracking to work at the experiment level. I'd be hesitant at this stage to mix jobs and experiments since they're different levels of hierarchy at the moment.

The goal is to abstract API consumers from the changes around /experiments and `/jobs, and avoid breaking the API unless we are permanently removing functionality on purpose).

I completely agree with you that jobs are a different abstraction, which is why /experiments/evaluation and experiments/inference were not something that we intended folks to use, they were a temporary by-product.

Here, the functionality folks could rely on was "/experiments carries out some steps to let me specify some data, a model and get a score back", whereas jobs aims at giving the person full control of one of the steps. We can keep that spirit (and in fact it's what we want in the future, when tracking is integrated), so we don't need to have a dummy /experiments around.

In the near future, we'll have to decide how exactly experiments will collect/trigger/x multiple jobs and how we track them. Until then, the UI doesn't need to care and can still use them as an entrypoint to workflows.