Engine - Spike: Migrate Legacy Wazuh-Engine APIs to cpp-httplib with HTTP and Protobuf Support

[JcabreraC]

Description:

With the recent integration of cpp-httplib into Wazuh-Engine, there is a need to explore the migration of existing legacy APIs, which currently utilize a proprietary protocol, to the more standard HTTP protocol provided by cpp-httplib. This spike will investigate how to integrate existing Google Protobuf message definitions with cpp-httplib to streamline API communication within Wazuh-Engine.

Objectives:

• Evaluate the current implementation of legacy APIs and identify dependencies on the proprietary protocol.
• Explore cpp-httplib's capabilities in handling HTTP requests and responses in conjunction with Google Protobuf.
• Determine the feasibility and outline the steps required to migrate legacy APIs to use HTTP via cpp-httplib while retaining the Protobuf message format for data serialization and deserialization.

Tasks:

1. Assessment of Existing APIs: Review the current legacy APIs to understand their operations, data flows, and integration points with the Wazuh-Engine.
2. Technical Exploration: Dive deep into cpp-httplib's documentation and examples to understand how it can be used with Protobuf.
3. Integration Strategy Development: Develop a detailed plan for transitioning the identified APIs to cpp-httplib, including adjustments needed to support HTTP and Protobuf.
4. Prototype Implementation: If feasible, create a small prototype to demonstrate the integration of cpp-httplib with Protobuf in handling a simple API request and response.

Expected Outcome:

• A comprehensive report detailing the findings from the assessment and exploration phases.
• A clear strategy and roadmap for migrating legacy APIs to cpp-httplib with HTTP support, using Protobuf for message handling.
• A prototype that illustrates the practical application of the proposed integration approach.

Definition of Done:

• [x] Complete the assessment of existing APIs and their requirements.
• [x] Document the capabilities of cpp-httplib in relation to HTTP and Protobuf integration.
• [x] Outline a detailed migration strategy that includes technical specifications and step-by-step implementation guidance.
• [x] Successfully demonstrate a prototype that uses cpp-httplib with HTTP and Protobuf for API communication.

[JcabreraC]

ETA extended by not starting it by working on other urgent issues.

[JavierBejMen]

Research

cpp-httplib is an HTTP/HTTPS server and client library. It operates using blocking I/O at the transport layer, handling requests and responses synchronously. The library uses a thread pool to manage multiple connections, processing each one in a dedicated thread to handle requests concurrently.

Regarding the usage of Protocol Buffers (protobuf), cpp-httplib does not have native support for protobuf. However, it can be easily integrated with protobuf by manually deserializing the received message.

With this in mind, cpp-httplib is suitable to replace our current server and API. It provides the necessary functionality to handle HTTP requests efficiently and can be integrated with protobuf for structured data communication. We will continue with our strategy to enqueue events and petitions, leveraging the library's thread pool for concurrent request handling while our multithreading paradigm remains unchanged.

Decreased Scalability could be a concern due to the blocking nature of the library. However, as our server mainly pushes events to a queue for processing, this realistically would only affect the API, not the internal event-handling mechanism.

[JavierBejMen]

Protobuf usage

Testing code:

#include <gtest/gtest.h>

#include <eMessages/catalog.pb.h>
#include <eMessages/eMessage.h>
#include <httplib.h>
#include <google/protobuf/util/json_util.h>

TEST(PocTest, Test1)
{
    httplib::Server svr;

    svr.Post("/hi",
            [](const httplib::Request& req, httplib::Response& res)
            {
                res.set_content("Hello World!", "text/plain");
                com::wazuh::api::engine::catalog::ResourceGet_Request body;
                if (!body.ParseFromString(req.body))
                {
                    res.status = 400;
                    res.set_content("Failed to parse protobuf message", "text/plain");
                    return;
                }
                std::string jsonRequest;
                google::protobuf::util::MessageToJsonString(body, &jsonRequest);
                std::cout << "Server received request: " << jsonRequest << std::endl;

                com::wazuh::api::engine::catalog::ResourceGet_Response response;
                response.set_content("Hello World!");
                response.set_status(com::wazuh::api::engine::ReturnStatus::OK);

                std::string serializedResponse;
                if (!response.SerializeToString(&serializedResponse))
                {
                    res.status = 500;
                    res.set_content("Failed to serialize protobuf message", "text/plain");
                    return;
                }

                res.set_content(serializedResponse, "application/octet-stream");
            });

    std::thread t([&svr]() { svr.listen("0.0.0.0", 8080); });

    httplib::Client cli("localhost", 8080);
    com::wazuh::api::engine::catalog::ResourceGet_Request request;
    request.set_name("test");
    request.set_format(com::wazuh::api::engine::catalog::ResourceFormat::yaml);
    request.set_namespaceid("default");

    std::string serializedRequest;
    if (!request.SerializeToString(&serializedRequest))
    {
        FAIL() << "Failed to serialize protobuf message";
    }

    httplib::Headers headers = {{"Content-Type", "application/octet-stream"}};
    auto response = cli.Post("/hi", headers, serializedRequest, "application/octet-stream");

    if (response && response->status == 200)
    {
        com::wazuh::api::engine::catalog::ResourceGet_Response body;
        if (!body.ParseFromString(response->body))
        {
            FAIL() << "Failed to parse protobuf message";
        }

        std::string jsonResponse;
        google::protobuf::util::MessageToJsonString(body, &jsonResponse);
        std::cout << "Client received response: " << jsonResponse << std::endl;
    }
    else
    {
        FAIL() << "Failed to send request";
    }

    svr.stop();

    t.join();
}

Output:

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from PocTest
[ RUN      ] PocTest.Test1
Server received request: {"name":"test","format":"yaml","namespaceid":"default"}
Client received response: {"status":"OK","content":"Hello World!"}
[       OK ] PocTest.Test1 (3 ms)
[----------] 1 test from PocTest (3 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (3 ms total)
[  PASSED  ] 1 test.

[JavierBejMen]

Migration Plan

• Define a high-level Server interface to register handlers and routes (We currently have the apiserver module, we could rename it and expand it). If all messages in the Engine are going to be protobuf serialized we can make it so at the interface level alleviating server user code.
• Modify the API module so it uses the request/response interface exposed by the server instead of the WazuhRequest/WazuhResponse.
• Apply the same changes to VD API for homogeneity.
• Implement server initialization on main:
  • Define how many servers a thread usage we need to implement (Engine API, Engine events, VD)

Event testing API requires communication between the API request handler and the thread processing the event, this needs further research on the exact implementation.

[JavierBejMen]

Event test API PoC

The server serving the client needs to wait/synchronize until the event is processed, as we cannot respond to the client outside of the server handler life scope:

TEST(PocTest, Test2)
{
    httplib::Server svr;
    struct Callback
    {
        std::function<void(const std::string&)> callback;
    };
    std::shared_ptr<Callback> delegatedResponseHandler = std::make_shared<Callback>();

    std::condition_variable cv;

    svr.Post("/hi",
             [&, delegatedResponseHandler](const httplib::Request& req, httplib::Response& res)
             {
                 res.set_content("Hello World!", "text/plain");
                 com::wazuh::api::engine::catalog::ResourceGet_Request body;
                 if (!body.ParseFromString(req.body))
                 {
                     res.status = 400;
                     res.set_content("Failed to parse protobuf message", "text/plain");
                     return;
                 }
                 std::string jsonRequest;
                 google::protobuf::util::MessageToJsonString(body, &jsonRequest);
                 std::cout << "Server received request: " << jsonRequest << std::endl;

                 delegatedResponseHandler->callback = std::function<void(const std::string&)>([&](const std::string& message)
                 {
                     com::wazuh::api::engine::catalog::ResourceGet_Response response;
                     response.set_content(message);
                     response.set_status(com::wazuh::api::engine::ReturnStatus::OK);

                     std::string serializedResponse;
                     if (!response.SerializeToString(&serializedResponse))
                     {
                         res.status = 500;
                         res.set_content("Failed to serialize protobuf message", "text/plain");
                         return;
                     }

                     res.set_content(serializedResponse, "application/octet-stream");
                 });

                 std::cout << "Server sent callback" << std::endl;
                 cv.notify_one();

                 {
                     std::mutex m;
                     std::unique_lock<std::mutex> lk(m);
                     // Wait for signal
                     cv.wait(lk);
                     std::cout << "Server received signal" << std::endl;
                 }
             });

    std::thread tDelegatedResponse(
        [&, delegatedResponseHandler]()
        {
            std::mutex m;
            std::unique_lock<std::mutex> lk(m);
            // Wait for signal
            cv.wait(lk);
            if (!delegatedResponseHandler->callback)
            {
                FAIL() << "Delegated response handler not set";
            }
            delegatedResponseHandler->callback(std::string("Hello World!"));
            cv.notify_one();
        });
    std::thread t([&svr]() { svr.listen("0.0.0.0", 8080); });

    httplib::Client cli("localhost", 8080);
    com::wazuh::api::engine::catalog::ResourceGet_Request request;
    request.set_name("test");
    request.set_format(com::wazuh::api::engine::catalog::ResourceFormat::yaml);
    request.set_namespaceid("default");

    std::string serializedRequest;
    if (!request.SerializeToString(&serializedRequest))
    {
        FAIL() << "Failed to serialize protobuf message";
    }

    httplib::Headers headers = {{"Content-Type", "application/octet-stream"}};
    auto response = cli.Post("/hi", headers, serializedRequest, "application/octet-stream");

    if (response && response->status == 200)
    {
        com::wazuh::api::engine::catalog::ResourceGet_Response body;
        if (!body.ParseFromString(response->body))
        {
            FAIL() << "Failed to parse protobuf message";
        }

        std::string jsonResponse;
        google::protobuf::util::MessageToJsonString(body, &jsonResponse);
        std::cout << "Client received response: " << jsonResponse << std::endl;
    }
    else
    {
        FAIL() << "Failed to send request";
    }

    svr.stop();

    t.join();
    tDelegatedResponse.join();
}

[juliancnn]

LGTM!, Although I think we should add some more tasks to the epic:

• Create a new test tool or use an existing one, PostMan style, that allows us to test End2End endpoints, through datasets defined in json format. This tool should not only allow to test a request, but also a sequence of requests (for example create a test session and use it).
• Create the test datasets for our framework for each module, at least some basic tests and then expand it.
• Create a new workflow to launch these tests.

[JavierBejMen]

Tasks

• Define a high-level Server interface to register handlers and routes (We currently have the apiserver module, we could rename it and expand it). If all messages in the Engine are going to be protobuf serialized we can make it so at the interface level alleviating server user code.
• Implement the server using httplib and protobuf
  • Implement synchronous calls for the event test API
• Modify the API module so it uses the request/response interface exposed by the server instead of the WazuhRequest/WazuhResponse.
  • Updating all APIs
• Apply the same changes to VD API for homogeneity.
• Implement server initialization on main:
  • Define how many servers and thread usage we need to implement (Engine API, Engine events, VD)
• Update api-communication package and all python scripts dependent on it.

[JcabreraC]

Tasks

The following issues are the result of this research and the next steps to address them:

• 26371

• 26388

• 26389

• 26390

• 26391