Add Makefile with key project actions (build, test, lint, format).
Reorganise CI flow. Separate ml and ui jobs from nitta build/test.
Add popover with all scores to subforest screen.
Bump stackage snapshot, to be hls-powered.
Some haskell related refactoring.
Update tensorflow from 2.12 to 2.13. Add mac os specific dependencies.
Add hack to specify NITTA_RUN_COMMAND from env variable.
TODOs:
[ ] Docker image for development: add make target to multi-platform build and push docker to docker-hub
[ ] Replace black via ruff autoformatter in CI, Makefile, and dev-docker-image
[ ] Rename score field in NodeView to defScore (UI, Haskell, maybe ML backend)
[ ] Add score selector to subForest screen.
[ ] Change CLI argument port type from Int with magic -1 to Maybe Int
[ ] Push ML scores to STM during synthesis process (NITTA.Synthesis.Explore:subForestIO).
subForestIO
BackendCtx{nodeScores, mlBackendGetter}
tree@Tree{sSubForestVar} = do
(firstTime, subForest) <-
atomically $
tryReadTMVar sSubForestVar >>= \case
Just subForest -> return (False, subForest)
Nothing -> do
subForest <- exploreSubForestVar tree
putTMVar sSubForestVar subForest
return (True, subForest)
when firstTime $ traceProcessedNode tree
-- FIXME: ML scores are evaluated here every time subForestIO is called. how to cache it like the default score? IO in STM isn't possible.
-- also it looks inelegant, is there a way to refactor it?
let modelNames = mapMaybe (T.stripPrefix mlScoreKeyPrefix) nodeScores
if
| null subForest -> return subForest
| null nodeScores -> return subForest
| null modelNames -> return subForest
| otherwise -> do
MlBackendServer{baseUrl} <- mlBackendGetter
case baseUrl of
Nothing -> return subForest
Just mlBackendBaseUrl -> do
-- (addMlScoreToSubforestSkipErrorsIO subForestAccum modelName) gets called for each modelName
foldM (addMlScoreToSubforestSkipErrorsIO mlBackendBaseUrl) subForest modelNames
[ ] Dockerfile with dev containers and other related stuff (vscode workspace) should be relocated from ml/synthesis to a separate dir, perhaps.
[ ] Currently ML if the config is invalid, the new evaluate script will fail with generic errors (TypeError, KeyError, etc.). It would be useful to write a JSON Schema describing evaluation configs. It will help to provide relevant field descriptions in editors and to implement a config validation in the script itself.
[ ] Simplify environment setup and nitta -- MLBackend integration. It shouldn't have such complex, like it happens in Makefile:ml-nitta
[ ] During the model training, if we use completely different synthesis trees for validation dataset instead of a random split over all shuffled training data, the validation loss and mae will be significantly worse. It means that the current model doesn't generalize well to other synthesis trees, although synthesis may still succeed, as it was shown. Improving those new validation metrics is crucial for further ML synthesis development. Need more input node parameters? Adjusting random descent in MCTS-like style too to fix negative labels share? UPD: some related work was done and yielded significant results in regards to fixing this, but mode's generalization abilities still require improvement.
[ ] Add tensorflow version check during dev-image build (should be the same with pyproject.toml)
[ ] I think the best solution here is to add NITTA_ML_BACKEND_RUN_COMMAND to provide more control. But I move it to the new issue because:
To: https://github.com/ryukzak/nitta/pull/167/files
Changes overview:
NITTA_RUN_COMMANDfrom env variable.TODOs:
scorefield inNodeViewtodefScore(UI, Haskell, maybe ML backend)porttype fromIntwith magic-1toMaybe IntNITTA.Synthesis.Explore:subForestIO).Makefile:ml-nittaNITTA_ML_BACKEND_RUN_COMMANDto provide more control. But I move it to the new issue because: