spray / spray-json

A lightweight, clean and simple JSON implementation in Scala
Apache License 2.0
972 stars 190 forks source link

Flaky performance test #293

Open jodersky opened 5 years ago

jodersky commented 5 years ago

The "not show bad performance characteristics when object keys' hashCodes collide" test in JsonParserSpec is flaky.

https://github.com/spray/spray-json/blob/0b893f26ed2fd71649389e7eccc7bba6f1507946/src/test/scala/spray/json/JsonParserSpec.scala#L117

In absolute terms, the regularTime is about 3E6 (3ms) on a modern laptop. This time is sufficiently large to make the test mostly pass on a dedicated machine. However, on a shared environment such as a Travis-CI build, it frequently causes the build to fail.

Potential solutions:

raboof commented 5 years ago

I think we should not put too much effort in these tests - let's first increase the number of keys (and/or be more lenient in the expectation) - if that's not enough I'd even be open to doing a couple of runs or removing the test wholesale.