GitHub actions

[PragTob]

This is #321 but pushed by me, so maybe/hopefully it works? :thinking:

@pablocostass

[PragTob]

Well it's running that's something :D

[PragTob]

Grml probably overexertion:

  1) test .run_scenarios can run multiple benchmarks in parallel (Benchee.Benchmark.RunnerTest)
     test/benchee/benchmark/runner_test.exs:82
     Assertion with >= failed
     code:  assert length(run_times_for(new_suite, "")) >= 8
     left:  3
     right: 8
     stacktrace:
##[error]       test/benchee/benchmark/runner_test.exs:93: (test)

I'm too tired.. probably gotta retry that one more or make it more lenient :|

[pablocostass]

@PragTob Hang in there :muscle:

If after a re-run it still fails I can, as you suggested, make that test more forgiving :)

[PragTob]

@pablocostass I'll take a look at it right now :)

[PragTob]

wow, after all that caching it is quite a bit faster!

But now it started failing on mac for... different reasons :cry:

   1) test .run_scenarios each triggers for every invocation, scenario once (Benchee.Benchmark.RunnerTest)
     test/benchee/benchmark/runner_test.exs:632
     Assertion with >= failed
     code:  assert hook_call_count >= 2
     left:  1
     right: 2
     stacktrace:
       test/benchee/benchmark/runner_test.exs:701: (test)

[PragTob]

  1) test formatters have full access to the suite data, values in assigns (BencheeTest)
     test/benchee_test.exs:315
     Assertion with =~ failed
     code:  assert output =~ ~r"Run time: #{@rough_10_milli_s}$"m
     left:  "Operating System: macOS\nCPU Information: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz\nNumber of Available Cores: 4\nAvailable memory: 12 GB\nElixir 1.10.3\nErlang 23.0.2\n\nBenchmark suite executing with the following configuration:\nwarmup: 5 ms\ntime: 10 ms\nmemory time: 0 ns\nreduction time: 0 ns\nparallel: 1\ninputs: none specified\nEstimated total run time: 15 ms\n\nBenchmarking Sleeps...\nRun time: 20.47 ms\nAverage: 20.47 ms\nCustom value\n"
     right: ~r/Run time: ((8|9|10|11|12|13|14)\.\d{2} ms)$/m
     stacktrace:
       test/benchee_test.exs:351: anonymous fn/0 in BencheeTest."test formatters have full access to the suite data, values in assigns"/1
       test/benchee_test.exs:316: (test)

This one already has a retrying attached to it.

well... changed the timings

[PragTob]

So.. that didn't work.

  1) test formatters have full access to the suite data, values in assigns (BencheeTest)
     test/benchee_test.exs:315
     Assertion with =~ failed
     code:  assert output =~ ~r"Run time: #{@rough_10_milli_s}$"m
     left:  "Operating System: macOS\nCPU Information: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz\nNumber of Available Cores: 4\nAvailable memory: 12 GB\nElixir 1.10.3\nErlang 23.0.2\n\nBenchmark suite executing with the following configuration:\nwarmup: 20 ms\ntime: 100 ms\nmemory time: 0 ns\nreduction time: 0 ns\nparallel: 1\ninputs: none specified\nEstimated total run time: 120 ms\n\nBenchmarking Sleeps...\nRun time: 20.10 ms\nAverage: 21.45 ms\nCustom value\n"
     right: ~r/Run time: ((8|9|10|11|12|13|14)\.\d{2} ms)$/m
     stacktrace:
       test/benchee_test.exs:351: anonymous fn/0 in BencheeTest."test formatters have full access to the suite data, values in assigns"/1
       test/benchee_test.exs:316: (test)

I'm... I can't. I bet it's the damn bad time measurement on Mac. I guess I'll have to deactivate Mac for these measurement sensitive ones. Or wait... one more shot.

[PragTob]

Aight. this is it...

  1) test formatters have full access to the suite data, values in assigns (BencheeTest)
     test/benchee_test.exs:315
     Assertion with =~ failed
     code:  assert output =~ ~r"Average: #{@rough_30_milli_s}$"m
     left:  "Operating System: macOS\nCPU Information: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz\nNumber of Available Cores: 4\nAvailable memory: 12 GB\nElixir 1.10.3\nErlang 23.0.2\n\nBenchmark suite executing with the following configuration:\nwarmup: 60 ms\ntime: 120 ms\nmemory time: 0 ns\nreduction time: 0 ns\nparallel: 1\ninputs: none specified\nEstimated total run time: 180 ms\n\nBenchmarking Sleeps...\nRun time: 33.35 ms\nAverage: 41.30 ms\nCustom value\n"
     right: ~r/Average: ((28|29|30|31|32|33|34)\.\d{2} ms)$/m
     stacktrace:
       test/benchee_test.exs:352: anonymous fn/0 in BencheeTest."test formatters have full access to the suite data, values in assigns"/1
       test/benchee_test.exs:316: (test)

[PragTob]

Seems it needs to install elixir every time on windows and that takes some time... would be great if we could also cache that but not super high prio as it's kinda like before.

[pablocostass]

Seems it needs to install elixir every time on windows and that takes some time... would be great if we could also cache that but not super high prio as it's kinda like before.

@PragTob I mistook cache and setup-elixir requirements and I thought the former couldn't be used in Windows when that's the case of the latter, oopsie. I'll open a PR with the change later.