Extensive testing of (new) functionality - Githubissues

dres-dev / DRES

Distributed Retrieval Evaluation Server

MIT License

14 stars 3 forks source link

Extensive testing of (new) functionality #489

Open sauterl opened 2 months ago

sauterl commented 2 months ago

Extensively test (new) functionality:

Evaluation Template Building

[ ] VBS-Style template
[ ] LSC-Style template
[ ] Templates including perpetual tasks
[ ] Exotic Templates
[ ] Try to break the builder

Evaluation Execution

[ ] VBS-Style evaluation, SYNC
[ ] VBS-Style evaluation, ASYNC
[ ] LSC-Style evalution, SYNC
[ ] LSC-Style evaluation, ASYNC
[ ] Perpetual tasks, SYNC
[ ] Perpetual tasks, ASYNC
[ ] Exotic Templates Evaluation, SYNC
[ ] Exotic Templates Evaluation, ASYNC
[ ] Robustness of submission endpoint

While doing the above, we should make sure that

The frontend behaves as expected
The backend behaves as expected
Errors in frontend are understandable for end users
Errors in backend are somewhat understandable and reportable
The API(s) works as intended