"Stack smashing detected" error in ubuntu docker container

[WY-CGhilardi]

After a system update of our libpistache-dev install, I am now running into stack smashing warnings like:

*** stack smashing detected ***: terminated
curl: (52) Empty reply from server
[1]+  Aborted                 ./test 9081

The first version I noticed this behavior was on

$ dpkg -s libpistache-dev | grep Version
Version: 0.2.3.20231022+git20231024.37ce2d4~ubuntu20.04.1

but can also reproduce on the latest release

$ dpkg -s libpistache-dev | grep Version
Version: 0.2.7.20240103+git20240106.1c733a1~ubuntu20.04.1

I was able to distill it down to a simplified MWE

Using the following test file and Dockerfile:

test.cpp

``` #include #include #include #include #include #include #include #include #include #include using namespace Pistache; void Empty_Route(const Rest::Request& request, Http::ResponseWriter response ){ if (request.headers().list().size() == 0) { response.send(Pistache::Http::Code::Ok); } response.send(Pistache::Http::Code::Ok); } int main( int argc, char *argv[] ) { using namespace Rest; std::cerr << "Started " << "\n"; // get port number if specified Port port(81); if( argc > 1 ) port = static_cast(std::stol(argv[1])); Address addr(Ipv4::any(),port); std::cerr << "opened port " << port.toString() << "\n"; Router router; auto opts = Http::Endpoint::options().maxRequestSize(45000000).threads(4); std::shared_ptr server = std::make_shared(addr); server->init(opts); //empty route part 1 Routes::Get(router, "/", Routes::bind(&Empty_Route)); Routes::Post(router, "/", Routes::bind(&Empty_Route)); server->setHandler(router.handler()); std::cerr << "server handler set\n"; server->serve(); } ```

Dockerfile

``` FROM ubuntu:20.04 ENV DEBIAN_FRONTEND=noninteractive # install pistache RUN apt-get update && apt install -y software-properties-common build-essential curl \ && add-apt-repository ppa:pistache+team/unstable \ && apt update \ && apt install -y libpistache-dev ADD test.cpp /tmp/test.cpp #compile test executable - simplified down from a larger CMake based build using -DCMAKE_BUILD_TYPE=Release RUN g++ /tmp/test.cpp --std=c++17 -L/usr/lib/x86_64-linux-gnu -lpistache -fstack-protector-strong -o /tmp/test # expose port used by rest api EXPOSE 9081 USER nobody ENTRYPOINT ["/tmp/test", "9081"] ```

After building and running the container, any request will throw that e.g. curl -i localhost:9081

I was able to get a valgrind dump and I believe the relevant portion is

==27== 
==27== Process terminating with default action of signal 6 (SIGABRT)
==27==    at 0x4BB800B: raise (raise.c:51)
==27==    by 0x4B97858: abort (abort.c:79)
==27==    by 0x4C0226D: __libc_message (libc_fatal.c:155)
==27==    by 0x4CA4CD9: __fortify_fail (fortify_fail.c:26)
==27==    by 0x4CA4CA5: __stack_chk_fail (stack_chk_fail.c:24)
==27==    by 0x110AA6: Pistache::Rest::Routes::bind<void, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter>(void (*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter))::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}::operator()(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter) const (in /tmp/test)
==27==    by 0x112397: std::_Function_handler<Pistache::Rest::Route::Result (Pistache::Rest::Request, Pistache::Http::ResponseWriter), Pistache::Rest::Routes::bind<void, Pistache::Rest::Request const&, Pistache::Http::ResponseWriter>(void (*)(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter))::{lambda(Pistache::Rest::Request const&, Pistache::Http::ResponseWriter)#1}>::_M_invoke(std::_Any_data const&, Pistache::Rest::Request&&, Pistache::Http::ResponseWriter&&) (in /tmp/test)
==27==    by 0x492149E: Pistache::Rest::Router::route(Pistache::Http::Request const&, Pistache::Http::ResponseWriter) (in /usr/lib/x86_64-linux-gnu/libpistache.so.0.2.7)
==27==    by 0x4921F25: Pistache::Rest::Private::RouterHandler::onRequest(Pistache::Http::Request const&, Pistache::Http::ResponseWriter) (in /usr/lib/x86_64-linux-gnu/libpistache.so.0.2.7)
==27==    by 0x48D0D6D: Pistache::Http::Handler::onInput(char const*, unsigned long, std::shared_ptr<Pistache::Tcp::Peer> const&) (in /usr/lib/x86_64-linux-gnu/libpistache.so.0.2.7)
==27==    by 0x49013AA: Pistache::Tcp::Transport::handleIncoming(std::shared_ptr<Pistache::Tcp::Peer> const&) (in /usr/lib/x86_64-linux-gnu/libpistache.so.0.2.7)
==27==    by 0x49056FC: Pistache::Tcp::Transport::onReady(Pistache::Aio::FdSet const&) (in /usr/lib/x86_64-linux-gnu/libpistache.so.0.2.7)
==27== 
==27== HEAP SUMMARY:
==27==     in use at exit: 107,132 bytes in 177 blocks
==27==   total heap usage: 360 allocs, 183 frees, 113,877 bytes allocated
==27== 
==27== Searching for pointers to 177 not-freed blocks
==27== Checked 33,760,840 bytes
==27== 

I can post the full log if helpful but it's about 1500 lines or so.

[kiplingw]

Hey @WY-CGhilardi. It's a build problem we think we've solved. Please use the latest unstable package off the PPA. @Tachi107 and I fixed this in #1182 and #1184. Let us know if that solves your problem.

[WY-CGhilardi]

@kiplingw I do still get this behavior on 0.2.7.20240103+git20240106.1c733a1~ubuntu20.04.1 , which is the latest I can see on the unstable PPA as of this morning.

[kiplingw]

Can you make sure you did a full rebuild of your environment? If you're using the Autotools regenerate your ./configure script.

[WY-CGhilardi]

Even forcing a clean docker build of the above Dockerfile (--no-cache) produces the same behavior. The example above does not use any build tools, just a straight g++ call , so it's pretty simple.

[kiplingw]

Can you try using with pkg-config?

[dennisjenkins75]

Can you repro that without stripping the symbols from pistache, so that the actual source line #s show up in the backtrace?

[WY-CGhilardi]

@dennisjenkins75 Not in a short amount of time, I am just consuming what is available on the PPA right now and there doesn't seem to be any kind of symbols package available through apt.

Do you have a dev docker container or some kind of setup script for deps if I am going to have to compile my own debug version of pistache? pistache-deps only seems to list googletest?

[kiplingw]

@Tachi107, if his usage of pkg-config solves the problem, then I think what we should consider doing is have Meson add the pre-processor defines for the build flags in include/pistache/config.h.

@WY-CGhilardi, make sure you've added debug symbols to the PPA sources:

deb http://ppa.launchpad.net/pistache+team/unstable/ubuntu mantic main main/debug

Did the problem disappear when you used pkg-config with your build environment?

For getting build dependencies, simply run the following:

$ sudo apt build-dep pistache

[Tachi107]

@WY-CGhilardi the issue is most likely here:

#compile test executable - simplified down from a larger CMake based build using -DCMAKE_BUILD_TYPE=Release
RUN g++ /tmp/test.cpp --std=c++17 -L/usr/lib/x86_64-linux-gnu -lpistache  -fstack-protector-strong -o /tmp/test

As mentioned in the README, the only way to properly use a precompiled version of Pistache is to use pkg-config (either directly or indirectly). You should change the compilation line above to:

RUN g++ /tmp/test.cpp --std=c++17 $(pkg-config --cflags --libs libpistache)  -fstack-protector-strong -o /tmp/test

[WY-CGhilardi]

@Tachi107 so that line is a very distilled version of the larger CMake build. In that build (which is still throwing the stack smash as well) we use

# Following instructions from here: https://github.com/pistacheio/pistache#cmake
find_package(PkgConfig)
pkg_check_modules(Pistache REQUIRED IMPORTED_TARGET libpistache)

For this smaller version, I was using the interpolated pkg-config line during my local testing, but the ubuntu image does not have pkg-config installed, so I wrote out the full version in place of the interpolation to save myself having to install it.

~In the docker container example, that seems to complain about a missing library or something~ EDIT: this required me to apt install libbrotli-dev first

#docker container
$ dpkg -s libpistache-dev | grep Version
Version: 0.2.7.20240103+git20240106.1c733a1~ubuntu20.04.1
$ pkg-config --cflags --libs libpistache
-DPISTACHE_USE_SSL -DPISTACHE_USE_CONTENT_ENCODING_BROTLI -DPISTACHE_USE_CONTENT_ENCODING_DEFLATE -lpistache

Additionally, looking across a few different machines, I end up with varying flags depending on which version is installed. e.g

#my laptop - what I used to hardcode compilation command
$ dpkg -s libpistache-dev | grep Version
Version: 0.1.1.20230324+git20230325.a68ad09~ubuntu20.04.1
$ pkg-config --cflags --libs libpistache
-L/usr/lib/x86_64-linux-gnu -lpistache 

#ec2 instance
$ dpkg -s libpistache-dev | grep Version
Version: 0.2.3.20231022+git20231024.37ce2d4~ubuntu20.04.1
$ pkg-config --cflags --libs libpistache
-lpistache

[WY-CGhilardi]

I tried dropping in the new pkg-config string into the compilation string and that seems to have corrected my smaller example. I guess I need to go back to the larger project and inspect the CMake outputs closer to figure out where it's breaking. I will keep this issue open for a bit and post anything relevant I can find in the meantime.

g++ /tmp/test.cpp --std=c++17 -DPISTACHE_USE_SSL -DPISTACHE_USE_CONTENT_ENCODING_BROTLI -DPISTACHE_USE_CONTENT_ENCODING_DEFLATE -lpistache -fstack-protector-strong -o /tmp/test

[kiplingw]

On Thu, 2024-01-11 at 12:31 -0800, WY-CGhilardi wrote:

Package libbrotlienc was not found in the pkg-config search path.

Good idea to always use the binary package for Pistache whenever possible because it will correctly pull the needed development and runtime dependencies.

[kiplingw]

@WY-CGhilardi, I'm going to close for now, but you're free to post additional comments if need be or the issue needs to be re-opened.

[Tachi107]

~In the docker container example, that seems to complain about a missing library or something~ EDIT: this required me to apt install libbrotli-dev first

Yeah, sorry, we forgot to add that package to the dependencies of libpistache-dev - if libbrotli .pc files are missing, pkg-config refuses to output the correct flags because of reasons. We've added it now in https://github.com/pistacheio/pistache/pull/1187 :)

[eipporko]

Do you know if this solution has actually been deployed? I'm using Pistache from the Ubuntu 22.04 repository, and I'm still encountering this issue.

[kiplingw]

Hey @eipporko. The solution was deployed two weeks ago. It should work now from the PPA when using via pkg-config.

[eipporko]

I tweaked my CMakeList.txt, and it's working now :) Sorry for any trouble!