Make injected variable have an overrides for a design

[proboscis]

When using injected variable as a run configuration, we often want to override some configurations by design

[proboscis]

When we define multiple experiments, the hyperparameters are frequently modified, and then utilized. Since we would have 100s of hyper parameter overrides, it would not be feasible to create such many designs and set it in a default_design_path.

One approach would be to make a InjectedWithDesign that overrides the provided design on use.

x = injected("x")
y:InjectedWithDesign = x + instances(...overrides)
# but what do I do if I want to get services out after running y, sharing the overrides?
z_from_y = injected("z") + y.overrides
# like this?

[proboscis]

Since injected already has operator implemented for inter injected operations, using '+' might look confusing. we can use some other operator like << for explicity. We can of course create a function that does it, but using it is tedious compared to using << operator, when it comes to defining quick experiments.

[proboscis]

Actually the code doesn't have to be explicitly parsable by python, we can always use natural languages in comment and ask GPT to parse it.

[proboscis]

Example script interface

exp_config = instances()
__meta_design__ = instances()
train = injected("train") # runnable
eval = injected("eval") # runnable
train_lr_01 = train << instances(learning_rate=0.1)
train_lr_02 = train << instances(learning_rate=0.2)
eval_lr_01 = eval << train_lr_01
eval_lr_02 = eval << train_lr_02 # or maybe train_lr_02.overrides

[proboscis]

script without such interface:

exp_config = instances()
exp_lr_01 = exp_config + instances(learning_rate=0.1)
exp_lr_02 = exp_config + instances(learning_rate=0.2)
train = injected("train") # select exp_lr_01/02 by pull down
eval = injected("eval") # select exp_lr_01/02 from pull down
ckpt_01 = train("ckpt_01") # it is possible to select exp_lr_02 from pull down menu although it's supposed be 01.
model_01 = load("ckpt_01")

[proboscis]

I believe it is okey to use the first interface, because you can always extract the overrides. However, when you start to not share the same default_design, it may become a problem for example, if you make do this,

default_design = instances()
x = injected("x")
x_01 = x << instances(name='hello')

default_design=instances()
x_02 = x_01  #different params

[proboscis]

The source of this problem is that, for application, is is okey to have many ways to run it. But for experiment, we want the results to look like an artifact that is immutable we don't want to switch the design to obtain a result for an experimet. It is better to have exactly one value associated with one variable.

[proboscis]

What do you do if you are combining multiple injecteds?

x = injected('x') << (lr=0.1)
y = injected('y') << (lr=0.2)
x + y ==??

[proboscis]

I can't solve this right now, and It seems it is better to have 100 designs. rather than implementing this interface

[proboscis]

Instead, make a 'Checkpoint' Injected that holds the design it is created so that everytime the injected value becomes consitent.

[proboscis]

Now, how do I casually support many designs for configurations? We can mark a design to be picked up by a configurator. like so:

exp_01 = instances .... + default_mark

Then how should a configurator pickup the designs?

[proboscis]

Strategy for picking up configurations.

1. make it injectable via __meta_design__

2. default strategy?

3. check attrs of parent modules recursively and accumulate

4. a parent's init's __meta_design__ can contain any moudles to look for designs

[proboscis]

For this, I should add a list of meta_design trace from a module where the lookup started, to be injected as __meta_designs__:List[ModuleVarSpec] so that we can refer to the context the run configurator is working on.