search

SerenityOS / serenity

The Serenity Operating System 🐞
https://serenityos.org
BSD 2-Clause "Simplified" License
30.54k stars 3.19k forks source link

LibM: lgamma([NAN, INFINITY, -INFINITY, -RealNumber]) causes stack overflow on x86_64 #8601

Closed ADKaster closed 3 years ago

ADKaster commented 3 years ago

On x86_64, any of the following cause a stack smash protector abort:

    EXPECT(isnan(lgamma(NAN)));
    EXPECT(isinf(lgamma(INFINITY)));
    EXPECT(isinf(lgamma(-INFINITY)));
    (void)lgamma(-2.5);
    (void)lgamma(-1.0);
ADKaster commented 3 years ago

cc @RealKC

ADKaster commented 3 years ago

Waaaait a minute. If you replace the calls to lgamma with lgamma_r, using a stack var for the sign pointer, it doesn't crash?

Sounds like an x86_64 loader problem rather than a libm problem now...