search

SciML / DiffEqBenchmarkServer.jl

Server behind the benchmarking bot @DiffEqBot for benchmarking scientific machine learning (SciML)
https://benchmarks.sciml.ai/
3 stars 2 forks source link

Better scheduling algorithm #5

Open kanav99 opened 5 years ago

kanav99 commented 5 years ago

This bot may be used for testing purposes later, so we might need a new scheduling algorithm for the bot.

  1. We can run simultaneous tests.
  2. But the benchmark job should always run on the idle runner.
  3. Support multiple runners.
ChrisRackauckas commented 5 years ago

I think what's more important is to make sure that we can trigger jobs with a different tag on a different machine. We are getting a second computer setup on Thursday which will have a GPU, but not a great CPU. So we want to be able to set some scripts to run with just the diffeqgpu tag and use that runner.

kanav99 commented 5 years ago

Yeah we should do this on a priority basis. We can use Gitlab Runner tags (we don't use them right now) and develop an argument runbenchmarks(on = gpu) so that bot knows where we should run that.

ChrisRackauckas commented 5 years ago

For benchmarks, does it run both commits each time, or cache master? If it caches master, we need to make sure each script has a dedicated machine.

kanav99 commented 5 years ago

No, right now it runs both the branches.

ChrisRackauckas commented 5 years ago

Yeah and that's probably better. So okay yes a priority system is all that's necessary.