Open hagrid67 opened 11 months ago
Yes, I can look into it. But do we really want to require a redis instance to be running just to run the checks in our build pipeline?
It seems that running Redis in github CI is well established.
I think we have plans to offer the evaluation service as a key feature of flatland, so that people can publish their benchmarks. We discussed at the last dev meeting starting work by the end of the year; Manuel is doing some planning. So I'd suggest that it's worth putting the effort into building some tests around it. The existing evaluator uses redis; this comes from the AICrowd architecture.
The notebook is one I built 2-3 years ago to run a simple test to start an evaluator service and then run a client against it. It "manually" starts and stops processes in the background so I'm sure it could be improved; eg we might want to run Redis in one container, and the evaluator in another, and then run the client in the foreground.
Until then, it seems fairly low effort to get this existing test working on github CI. It seems to run OK on my local Ubuntu 22.04 with redis running.
Objectives Re-enable
test_service.ipynb
intox notebooks
to test the evaluation serviceTodos
redis
container for the notebooks workflowredis_ip
test_service.ipynb
Minimal testing requirements N/A - this change enables an existing test to run
Additional context
checks.yml
notebooks have recently been enabled by Jeremy / @hagrid67 in PR #61 @fnberta would you mind taking a look at enablingredis
in the workflow please? (Or I'll take a look in a few days)