Seg Fault

[jeffRTC]

@randombit

Decided to debug https://github.com/randombit/botan/issues/2583 by creating small app and I see it prints vector correctly but when it reach end of application, it crash with a seg fault.

Valgrind stack trace,

==2716398== 
Length of Vector : 4
Count :0
Count :1
Count :2
Count :3
==2716398== Invalid write of size 8
==2716398==    at 0x4842964: memset (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2716398==    by 0x566A82F: Botan::deallocate_memory(void*, unsigned long, unsigned long) (in /usr/lib/x86_64-linux-gnu/libbotan-2.so.12.12.1)
==2716398==    by 0x55E1A4D: ??? (in /usr/lib/x86_64-linux-gnu/libbotan-2.so.12.12.1)
==2716398==    by 0x40EC7B: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:155)
==2716398==    by 0x40EC29: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:730)
==2716398==    by 0x41112D: std::__shared_ptr<Botan::RSA_Public_Data const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1169)
==2716398==    by 0x411107: std::shared_ptr<Botan::RSA_Public_Data const>::~shared_ptr() (shared_ptr.h:103)
==2716398==    by 0x41109D: Botan::RSA_PublicKey::~RSA_PublicKey() (rsa.h:25)
==2716398==    by 0x410FC1: Botan::RSA_PrivateKey::~RSA_PrivateKey() (rsa.h:92)
==2716398==    by 0x410DC5: Botan::RSA_PrivateKey::~RSA_PrivateKey() (rsa.h:92)
==2716398==    by 0x410E8A: std::_Sp_counted_ptr<Botan::RSA_PrivateKey*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:377)
==2716398==    by 0x40EC7B: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:155)
==2716398==  Address 0x9419080 is not stack'd, malloc'd or (recently) free'd
==2716398== 
==2716398== 
==2716398== Process terminating with default action of signal 11 (SIGSEGV)
==2716398==  Access not within mapped region at address 0x9419080
==2716398==    at 0x4842964: memset (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2716398==    by 0x566A82F: Botan::deallocate_memory(void*, unsigned long, unsigned long) (in /usr/lib/x86_64-linux-gnu/libbotan-2.so.12.12.1)
==2716398==    by 0x55E1A4D: ??? (in /usr/lib/x86_64-linux-gnu/libbotan-2.so.12.12.1)
==2716398==    by 0x40EC7B: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:155)
==2716398==    by 0x40EC29: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:730)
==2716398==    by 0x41112D: std::__shared_ptr<Botan::RSA_Public_Data const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1169)
==2716398==    by 0x411107: std::shared_ptr<Botan::RSA_Public_Data const>::~shared_ptr() (shared_ptr.h:103)
==2716398==    by 0x41109D: Botan::RSA_PublicKey::~RSA_PublicKey() (rsa.h:25)
==2716398==    by 0x410FC1: Botan::RSA_PrivateKey::~RSA_PrivateKey() (rsa.h:92)
==2716398==    by 0x410DC5: Botan::RSA_PrivateKey::~RSA_PrivateKey() (rsa.h:92)
==2716398==    by 0x410E8A: std::_Sp_counted_ptr<Botan::RSA_PrivateKey*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:377)
==2716398==    by 0x40EC7B: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:155)
==2716398==  If you believe this happened as a result of a stack
==2716398==  overflow in your program's main thread (unlikely but
==2716398==  possible), you can try to increase the size of the
==2716398==  main thread stack using the --main-stacksize= flag.
==2716398==  The main thread stack size used in this run was 8388608.
==2716398== 
==2716398== HEAP SUMMARY:
==2716398==     in use at exit: 149,626 bytes in 1,143 blocks
==2716398==   total heap usage: 210,274 allocs, 209,131 frees, 89,801,475 bytes allocated
==2716398== 
==2716398== LEAK SUMMARY:
==2716398==    definitely lost: 0 bytes in 0 blocks
==2716398==    indirectly lost: 0 bytes in 0 blocks
==2716398==      possibly lost: 1,352 bytes in 18 blocks
==2716398==    still reachable: 148,274 bytes in 1,125 blocks
==2716398==                       of which reachable via heuristic:
==2716398==                         newarray           : 1,536 bytes in 16 blocks
==2716398==         suppressed: 0 bytes in 0 blocks
==2716398== Rerun with --leak-check=full to see details of leaked memory
==2716398== 
==2716398== For lists of detected and suppressed errors, rerun with: -s
==2716398== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

My code,

#include <iostream>
#include <string>
#include <vector>
#include <variant>
#include <set>
#include <mutex>
#include <csignal>
#include <future>
#include <memory>
#include <thread>
#include <unordered_map>
#include <fstream>
#include <sstream>
#include <botan/tls_server.h>
#include <botan/tls_callbacks.h>
#include <botan/tls_session_manager.h>
#include <botan/tls_policy.h>
#include <botan/auto_rng.h>
#include <botan/certstor.h>
#include <botan/pk_keys.h>
#include <botan/pkcs10.h>
#include <botan/pkcs8.h>
#include <botan/x509self.h>
#include <botan/x509path.h>
#include <botan/x509_ca.h>
#include <botan/x509_ext.h>
#include <botan/pk_algs.h>
#include <botan/ber_dec.h>
#include <botan/der_enc.h>
#include <botan/oids.h>
#include <botan/rsa.h>

namespace TLS
{
    typedef std::chrono::duration<int, std::ratio<31556926>> years;

    class callbacks : public Botan::TLS::Callbacks
    {
    public:
        std::string decodedMessage; // Store Decoded (In)
        std::vector<std::string> encodedMessages; // Store Encoded (Out)

    public:
        void tls_emit_data(const uint8_t data[], size_t size) override
        {
            encodedMessages.emplace_back(data, data + size);
        }

        void tls_record_received(uint64_t seq_no, const uint8_t data[], size_t size) override
        {

            decodedMessage += std::string(data, data + size);
        }

        void tls_alert(Botan::TLS::Alert alert) override
        {
            // handle a tls alert received from the tls server
            std::cout << alert.type_string() << "\n";
        }

        bool tls_session_established(const Botan::TLS::Session &session) override
        {
            // the session with the tls client was established
            // return false to prevent the session from being cached, true to
            // cache the session in the configured session manager
            return false;
        }
    };

    class credentials : public Botan::Credentials_Manager
    {
    private:
        struct certificate
        {
            std::vector<Botan::X509_Certificate> certs;
            std::shared_ptr<Botan::Private_Key> key;
        };

        std::vector<certificate> creds;
        std::vector<std::shared_ptr<Botan::Certificate_Store>> store;

    public:
        Botan::Private_Key *private_key_for(const Botan::X509_Certificate &cert,
                                            const std::string & /*type*/,
                                            const std::string & /*context*/) override
        {
            for (auto &&i : creds)
            {
                if (cert == i.certs[0])
                {
                    return i.key.get();
                }
            }

            return nullptr;
        }

        std::vector<Botan::X509_Certificate> cert_chain(const std::vector<std::string> &algos,
                                                        const std::string &type,
                                                        const std::string &hostname) override
        {
            BOTAN_UNUSED(type);

            for (auto &&i : creds)
            {
                if (std::find(algos.begin(), algos.end(), i.key->algo_name()) == algos.end())
                {
                    continue;
                }

                if (hostname != "" && !i.certs[0].matches_dns_name(hostname))
                {
                    continue;
                }

                return i.certs;
            }

            return std::vector<Botan::X509_Certificate>();
        }

        std::vector<Botan::Certificate_Store *> trusted_certificate_authorities(const std::string &type,
                                                                                const std::string & /*hostname*/) override
        {
            std::vector<Botan::Certificate_Store *> v;

            // don't ask for client certs
            if (type == "tls-server")
            {
                return v;
            }
            else
            {
                std::cout << "DEBUG: Request for Client Cert"
                          << "\n";
            }

            for (auto &&cs : store)
            {
                v.push_back(cs.get());
            }

            return v;
        }

        void createCert(std::string hostname)
        {
            /**
             * Check for existing Certs 
            **/

            for (auto &&i : creds)
            {

                if (!i.certs[0].matches_dns_name(hostname))
                {
                    continue;
                }

                // Found a match for hostname
                return;
            }

            /**
             * Initialize Root CA
            **/

            Botan::AutoSeeded_RNG rng;

            const Botan::X509_Certificate rootCert("myCA.pem");

            std::ifstream rootCertPrivateKeyFile("myCAKey.pkcs8.pem");

            Botan::DataSource_Stream rootCertPrivateKeyStream(rootCertPrivateKeyFile);

            std::unique_ptr<Botan::Private_Key> rootCertPrivateKey = Botan::PKCS8::load_key(rootCertPrivateKeyStream, "pass@");

            Botan::X509_CA rootCA(rootCert, *rootCertPrivateKey, "SHA-256", rng);

            /**
            * Generate a Cert & Sign with Root CA
            **/

            Botan::X509_Cert_Options opts;
            std::shared_ptr<Botan::Private_Key> serverPrivateKeyShared(new Botan::RSA_PrivateKey(rng, 4096));
            Botan::RSA_PrivateKey* serverPrivateKey = (Botan::RSA_PrivateKey*)serverPrivateKeyShared.get();

            opts.common_name = hostname;
            opts.country = "US";

            auto now = std::chrono::system_clock::now();

            Botan::X509_Time todayDate(now);
            Botan::X509_Time expireDate(now + years(1));

            Botan::PKCS10_Request req = Botan::X509::create_cert_req(opts, *serverPrivateKey, "SHA-256", rng);

            auto serverCert = rootCA.sign_request(req, rng, todayDate, expireDate);

            /**
             * Load Cert to In-Memory Database
            **/

            certificate cert;

            cert.certs.push_back(serverCert);
            cert.key = serverPrivateKeyShared;

            creds.push_back(cert);
        }
    };
}; // namespace TLS

TLS::credentials creds;

int main() {
    std::ifstream inFile;
    inFile.open("0.packet", std::ios::binary);

    std::stringstream strStream;
    strStream << inFile.rdbuf();

    std::string packet = strStream.str(); 

    TLS::callbacks tlsCallbacks;

    Botan::AutoSeeded_RNG rng;
    Botan::TLS::Session_Manager_In_Memory sessionMgr(rng);
    Botan::TLS::Strict_Policy policy;

    Botan::TLS::Server server(tlsCallbacks, sessionMgr, creds, policy, rng);

    creds.createCert("www.google.com");

    server.received_data(reinterpret_cast<const uint8_t*>(packet.c_str()), packet.size());

    if(!tlsCallbacks.encodedMessages.empty()) {
        std::cout << "Length of Vector : " << tlsCallbacks.encodedMessages.size() << "\n";

        int count = 0;

        for(const auto &i : tlsCallbacks.encodedMessages) {
            std::cout << "Count :" << count << "\n";

            count++;
        }
    }
}

[jeffRTC]

@nametoolong Do you have any idea either?

[jeffRTC]

@nametoolong @randombit

A user on Stackoverflow told me,

The Valgrind trace clearly shows that the issue is in Botan::deallocate_memory which is something that can only be fixed in the library itself. So either the library author needs to fix it or you need to fix it; a random person on Stack Overflow is almost certainly not going to fix it for you.

[jeffRTC]

@nametoolong @randombit

Can't Botan deallocate memory for globally defined credentials object ?

[randombit]

Sorry about the delay here. The problem is the globally defined object.

The problem is that the mlock pool is a singleton created on first use then destroyed sometime after main returns. First your object is created. It allocates memory. This results in the pool being created. Destruction happens LIFO. So first the pool is destroyed. Then your object is destroyed, and attempts to touch memory (to zero it) which has already been unmapped.

Workarounds are

• Create a Botan::Allocator_Initializer object to force initialization before your object is created (thus the pool lives until after your object has been destructed)
• Disable locking_allocator module
• Set env var BOTAN_MLOCK_POOL_SIZE to 0
• No global vars

In principle the locking allocator instead of munmaping the memory, just zeros it, and leave it to be unmapped by the OS on process exit. This might still break invariants, but not as badly. It also causes valgrind to reports leaks which is obnoxious.

I think because it was mmap'ed directly and not through malloc, valgrind doesn't track it.

[jeffRTC]

Thanks

[jeffRTC]

@randombit

So, I went with you first solution.

.......
.......
#include <botan/mem_ops.h>

Botan::Allocator_Initializer initAllocator;
TLS::credentials creds;

Compiled this with Clang, it outputs,

 -I/usr/include/botan-2

/usr/bin/ld: /tmp/example-10fe96.o: in function `Allocator_Initializer':
/usr/include/botan-2/botan/mem_ops.h:43: undefined reference to `Botan::initialize_allocator()'
clang: error: linker command failed with exit code 1 (use -v to see invocation)