Early pruning steps in the workflow

[mhoangvslev]

• Retrieve all timeout queries with this

  find experiments/bsbm/benchmark/evaluation/fedx -type f -empty | grep ".txt" | sort

yields result

experiments/bsbm/benchmark/evaluation/fedx/q01/instance_1/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_2/batch_4/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_2/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_2/batch_4/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_2/batch_4/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_3/batch_4/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_3/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_3/batch_4/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_3/batch_4/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_4/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_4/batch_4/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_4/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_4/batch_4/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_4/batch_4/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_7/batch_4/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_7/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_7/batch_4/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q04/instance_7/batch_4/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_4/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_4/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_4/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_0/batch_4/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_1/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_2/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_3/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_4/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_5/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_6/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_7/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_8/batch_3/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_2/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_2/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_2/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_2/attempt_3/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_3/attempt_0/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_3/attempt_1/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_3/attempt_2/results.txt
experiments/bsbm/benchmark/evaluation/fedx/q06/instance_9/batch_3/attempt_3/results.txt

• You notice that after batch 2, q06 times out for every instances, every attempts. It's safe to stop evaluate this query instances after batch2
• It's safe to skip evaluation for any query instance if it timed out for the previous batch.