in-context-learning-2024 / in-context

5 stars 1 forks source link

Rework CI testing #52

Open nelson-lojo opened 6 months ago

nelson-lojo commented 6 months ago

CI testing could use a more extensible matrix for testing function classes and context models by leveraging some conventions:

We could have each model key as defined in src/models/__init__.py correspond to a tag equivalent to the key with spaces replaced by underscores (_) and similarly for function classes. We would then have "default" yaml configs named conf/train/<fc_tag>_<model_tag>.yml.

CI could then simply sed/awk steps: [0-9]* into steps: 5 in that target file.

nelson-lojo commented 4 months ago

Currently a stop-gap version of this is implemented in #36 , where models are extracted from conf/include/models/base.yml and composed.yml and tasks are extracted from conf/include/tasks.yml

nelson-lojo commented 4 months ago

I think a better long-term solution is to completely conventionalize the configuration system and document it well.

Along these lines, consider: Model Name Anchors: <family name>_<modifier 1>_<modifier n>_<additional extension> Task Name Anchors: <task name>_<modifier 1>_<modifier n>_{regression | classification | parity}

nelson-lojo commented 4 months ago

A potential tack-on feature is to remove static conf files from being needed at the outset, instead only saving the yaml in the directory where we save all artifacts since the only content of a conf file is 5 lines:

<<: *some_task
steps: 500001
log_freq: 100
checkpoint_freq: 50000
model: *some_model