search

heroku / barnes

Tell StatsD about request time, GC, objects and more. Latest Rails 4 and Ruby 2.1 support, and ancient Rails 2 and Ruby 1.8 support.
MIT License
0 stars 0 forks source link

Do not include gem in test environment #24

Closed Tolsto closed 4 years ago

Tolsto commented 5 years ago

The Readme currently suggest to just include the gem in the Gemfile. However, it's a bad idea to include it in the test environment. When running on the Heroku CI it will start a thread that makes calls to Kernel.rand(). Those calls will mess up the test seed and test failures on the CI will not be reliably reproducible locally anymore. It took us a long time to debug this issue and it would be good if the Readme would state this.

schneems commented 5 years ago

I don’t totally follow the failure mode. Are calls to “rand” failing for your app on Heroku CI but only with this gem?

Do you get a similar randomly failing error by adding this gem to a Rails new app and running on Heroku CI or is there something specific that triggers this behavior?

On Mon, Aug 26, 2019 at 4:08 AM Nils Mueller notifications@github.com wrote:

The Readme currently suggest to just include the gem in the Gemfile. However, it's a bad idea to include it in the test environment. When running on the Heroku CI it will start a thread that makes calls to Kernel.rand(). Those calls will mess up the test seed and test failures on the CI will not be reliably reproducible locally anymore. It took us a long time to debug this issue and it would be good if the Readme would state this.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/heroku/barnes/issues/24?email_source=notifications&email_token=AAAOSYB5O67IX4TMZGU25ULQGOMSHA5CNFSM4IPNRQQKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HHKQWOA, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAOSYAUUOUCNAYTZDIFUU3QGOMSHANCNFSM4IPNRQQA .

-- Richard Schneeman https://www.schneems.com

Tolsto commented 5 years ago

The Gem does not produce any errors on the CI. However, it starts a thread and makes calls to rand() from within that thread. This means that a test that runs in the main thread after one of those rand() calls and also makes a rand() call will receive a different random number than it would have received if barnes wouldn't run. However, because of the thread scheduler it's not predictable when exactly the thread will get resources to make the call to rand(). The result is that the same test in the test suite might get different random numbers for each suite run, even when using the same RNG seed in Rspec.