Hi Andy, First of all, thank you for the great libwebsocket!. I was trying to understand from GitHub issues/mailing lists of lws if it’s safe to use multiple lws contexts on the same process inside different threads, where each thread is 100% independent and has its own service loop:

From here (2014) I’ve understand that it is safe: https://libwebsockets.org/pipermail/libwebsockets/2014-September/001340.html

and from here (2017) I’ve understand that it is NOT safe: https://github.com/warmcat/libwebsockets/issues/961

So, 1) For now (latest table version) is it safe to work with multiple lws contexts from different threads ? 2) I saw on https://github.com/warmcat/libwebsockets/issues/961 that you mentioned that one of the points is the old version of OpenSSL initialization and global variables, Is that problem still relevant? For which versions of OpenSSL? 3) For now (latest table version) there are more problematic points to work with multiple lws contexts from different threads?

BTW, I’ve tried to create some multithreaded example (below) that with 10 threads, each has lws context with its own service loop, each thread act like WS client calling 100 times to WS echo server and it worked just fine with no crashes.

Thanks in advance!

code:

// Simple LibWebSockets test client //////////////////////////////////////////////////////////////////////////

include

include "libwebsockets.h"

//////////////////////////////////////////////////////////////////////////

static int bExit; static int bDenyDeflate = 1; static int request_counter = 0;

static int callback_test(struct lws wsi, enum lws_callback_reasons reason, void user, void* in, size_t len);

//////////////////////////////////////////////////////////////////////////

// Escape the loop when a SIGINT signal is received static void onSigInt(int sig) { bExit = 1; }

//////////////////////////////////////////////////////////////////////////

// The registered protocols static struct lws_protocols protocols[] = { { "test-protocol", // Protocol name callback_test, // Protocol callback 0, // Data size per session (can be left empty) 512, // Receive buffer size (can be left empty)

    },
    { NULL, NULL, 0 } // Always needed at the end

};

// The extensions LWS supports, without them some requests may not be able to work static const struct lws_extension extensions[] = { { "permessage-deflate", lws_extension_callback_pm_deflate, "permessage-deflate; client_max_window_bits" }, { "deflate-frame", lws_extension_callback_pm_deflate, "deflate_frame" }, { NULL, NULL, NULL } // Always needed at the end };

// List to identify the indices of the protocols by name enum protocolList { PROTOCOL_TEST,

    PROTOCOL_LIST_COUNT // Needed

};

//////////////////////////////////////////////////////////////////////////

// Callback for the test protocol static int callback_test(struct lws wsi, enum lws_callback_reasons reason, void user, void* in, size_t len) { // The message we send back to the echo server const char msg[128] = "Simple webserver echo test!";

    // The buffer holding the data to send
    // NOTICE: data which is sent always needs to have a certain amount of memory (LWS_PRE) preserved for headers
    unsigned char buf[LWS_PRE + 128];

    // Allocating the memory for the buffer, and copying the message to it
    memset(&buf[LWS_PRE], 0, 128);
    strncpy((char*)buf + LWS_PRE, msg, 128);

    // For which reason was this callback called?
    switch (reason)
    {
            // The connection closed
    case LWS_CALLBACK_CLOSED:
            printf("[Test Protocol] Connection closed.\n");
            break;

            // Our client received something
    case LWS_CALLBACK_CLIENT_RECEIVE:
    {
            printf("[Test Protocol] Received data: \"%s\"\n", (char*)in);

    if (request_counter >= 150)
    {
                bExit = 1;
        return -1; // Returning -1 causes the client to disconnect from the server
    }
    else
    {
        request_counter++;
        lws_callback_on_writable(wsi);
    }

    }
            break;

            // Here the server tries to confirm if a certain extension is supported by the server
    case LWS_CALLBACK_CLIENT_CONFIRM_EXTENSION_SUPPORTED:
            if (strcmp((char*)in, "deflate-stream") == 0)
            {
                    if (bDenyDeflate)
                    {
                            printf("[Test Protocol] Denied deflate-stream extension\n");
                            return 1;
                    }
            }
            break;

            // The connection was successfully established
    case LWS_CALLBACK_CLIENT_ESTABLISHED:
            printf("[Test Protocol] Connection to server established.\n");

            printf("[Test Protocol] Writing \"%s\" to server.\n", msg);

            // Write the buffer from the LWS_PRE index + 128 (the buffer size)
            lws_write(wsi, &buf[LWS_PRE], 128, LWS_WRITE_TEXT);

            break;

            // The server notifies us that we can write data
    case LWS_CALLBACK_CLIENT_WRITEABLE:
    lws_write(wsi, &buf[LWS_PRE], 128, LWS_WRITE_TEXT);
            printf("[Test Protocol] The client is able to write.\n");
            break;

            // There was an error connecting to the server
    case LWS_CALLBACK_CLIENT_CONNECTION_ERROR:
            printf("[Test Protocol] There was a connection error: %s\n", in ? (char*)in : "(no error information)");
            break;

    default:
            break;
    }

    return 0;

}

//////////////////////////////////////////////////////////////////////////

// Main application entry void main_tr(void dataptr) { lws_set_log_level(LLL_ERR | LLL_WARN, lwsl_emit_syslog); // We don't need to see the notice messages

    signal(SIGINT, onSigInt); // Register the SIGINT handler

    // Connection info
    char inputURL[300] = "wss://echo.websocket.org";
    int inputPort = 80;

    struct lws_context_creation_info ctxCreationInfo; // Context creation info
    struct lws_client_connect_info clientConnectInfo; // Client creation info
    struct lws_context *ctx; // The context to use

    struct lws *wsiTest; // WebSocket interface
    const char *urlProtocol, *urlTempPath; // the protocol of the URL, and a temporary pointer to the path
    char urlPath[300]; // The final path string

    // Set both information to empty and allocate it's memory
    memset(&ctxCreationInfo, 0, sizeof(ctxCreationInfo));
    memset(&clientConnectInfo, 0, sizeof(clientConnectInfo));

    clientConnectInfo.port = inputPort; // Set the client info's port to the input port

    // Parse the input url (e.g. wss://echo.websocket.org:1234/test)
    //   the protocol (wss)
    //   the address (echo.websocket.org)
    //   the port (1234)
    //   the path (/test)
    if (lws_parse_uri(inputURL, &urlProtocol, &clientConnectInfo.address, &clientConnectInfo.port, &urlTempPath))
    {
            printf("Couldn't parse URL\n");
    }

    // Fix up the urlPath by adding a / at the beginning, copy the temp path, and add a \0 at the end
    urlPath[0] = '/';
    strncpy(urlPath + 1, urlTempPath, sizeof(urlPath) - 2);
    urlPath[sizeof(urlPath) - 1] = '\0';

    clientConnectInfo.path = urlPath; // Set the info's path to the fixed up url path

    // Set up the context creation info
    ctxCreationInfo.port = CONTEXT_PORT_NO_LISTEN; // We don't want this client to listen
    ctxCreationInfo.protocols = protocols; // Use our protocol list
    ctxCreationInfo.gid = -1; // Set the gid and uid to -1, isn't used much
    ctxCreationInfo.uid = -1;
    ctxCreationInfo.extensions = extensions; // Use our extensions list

    // Create the context with the info
    ctx = lws_create_context(&ctxCreationInfo);
    if (ctx == NULL)
    {
            printf("Error creating context\n");
            return NULL;
    }

    // Set up the client creation info
    clientConnectInfo.context = ctx; // Use our created context
    clientConnectInfo.ssl_connection = 0; // Don't use SSL for this test
    clientConnectInfo.host = clientConnectInfo.address; // Set the connections host to the address
    clientConnectInfo.origin = clientConnectInfo.address; // Set the conntections origin to the address
    clientConnectInfo.ietf_version_or_minus_one = -1; // IETF version is -1 (the latest one)
    clientConnectInfo.protocol = protocols[PROTOCOL_TEST].name; // We use our test protocol

    printf("Connecting to %s://%s:%d%s \n\n", urlProtocol, clientConnectInfo.address, clientConnectInfo.port, urlPath);

    // Connect with the client info
    lws_client_connect_via_info(&clientConnectInfo);
    //if (wsiTest)
    //{
    //      printf("Error creating the client\n");
    //      return NULL;
    //}

    // Main loop runs till bExit is true, which forces an exit of this loop
    while (!bExit)
    {
            // LWS' function to run the message loop, which polls in this example every 50 milliseconds on our created context
            lws_service(ctx, 50);
    }

    // Destroy the context
    lws_context_destroy(ctx);

    printf("\nDone executing.\n");

    return NULL;

}

define NUM_THREADS 10

int main() {

/ this variable is our reference to the second thread / pthread_t inc_x_thread[NUM_THREADS]; / create a second thread which executes inc_x(&x) / for (int i = 0; i < NUM_THREADS; i++) { if(pthread_create(&inc_x_thread[i], NULL,main_tr, NULL)) {

    fprintf(stderr, "Error creating thread\n");
    return 1;

}

}

/ wait for the second thread to finish / for (int i = 0; i < NUM_THREADS; i++) { if(pthread_join(inc_x_thread[i], NULL)) {

    fprintf(stderr, "Error joining thread\n");
    return 2;

}

}

return 0;

}

For others reading this via google, the main point is considering lws uses the single-threaded event loop style, the recommended answer is one context for the user code. That context can handle as many serving vhosts and discrete client connections as you like. None of the scenarios in the examples require multiple contexts... even multiple serving threads is done with one context.

I saw on #961 that you mentioned that one of the points is the old version of OpenSSL initialization and global variables, Is that problem still relevant? For which versions of OpenSSL?

You have the code... see for yourself

https://libwebsockets.org/git/libwebsockets/tree/lib/tls/openssl/ssl.c#n389

AFAIK openssl is the only point that may actually break it. Everything else in lws is dereferenced through the context cleanly.

For now (latest table version) there are more problematic points to work with multiple lws contexts from different threads?

It's going to be bloated... as an optimization on *nix where process fds are ordinals starting from 0, the context allocates some tables at creation for all possible fds the process may meet... you can find out how much with INFO logging. Since these are processwide assets, not just thread-specific, each context must allocate the whole thing. If your process - the whole process - actually only needs a few dozen fds max, you can artificially restrict this either with ulimit or in the context creation struct

https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-context-vhost.h#n358-361

Otherwise... this is FOSS. "Are there ever going to be problems" doesn't have an answer if you're choosing to do your own thing. The situation is as described, if you want to do it nobody will stop you. But if you find problems you may need to get your hands dirty.

Something else with this context-per-thread scheme is there no longer any global serialization from a single event loop, depending on what you're doing this may introduce a big burden for locking where none was needed before. But that's a trade-off you can figure out.

I was using a context-per-thread for a while on the client side but switched to a single context and service thread at the end. The only issue I have seen if calling lws_context_destroy(context) sometimes causes a double free crash especially when it failed to connect server.

On Tue, Mar 12, 2019 at 11:14 AM Andy Green notifications@github.com wrote:

For others reading this via google, the main point is considering lws uses the single-threaded event loop style, the recommended answer is one context for the user code. That context can handle as many serving vhosts and discrete client connections as you like. None of the scenarios in the examples require multiple contexts... even multiple serving threads is done with one context.

I saw on #961 https://github.com/warmcat/libwebsockets/issues/961 that you mentioned that one of the points is the old version of OpenSSL initialization and global variables, Is that problem still relevant? For which versions of OpenSSL?

You have the code... see for yourself

https://libwebsockets.org/git/libwebsockets/tree/lib/tls/openssl/ssl.c#n389

AFAIK openssl is the only point that may actually break it. Everything else in lws is dereferenced through the context cleanly.

For now (latest table version) there are more problematic points to work with multiple lws contexts from different threads?

It's going to be bloated... as an optimization on *nix where process fds are ordinals starting from 0, the context allocates some tables at creation for all possible fds the process may meet... you can find out how much with INFO logging. Since these are processwide assets, not just thread-specific, each context must allocate the whole thing. If your process - the whole process - actually only needs a few dozen fds max, you can artificially restrict this either with ulimit or in the context creation struct

https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-context-vhost.h#n358-361

Otherwise... this is FOSS. "Are there ever going to be problems" doesn't have an answer if you're choosing to do your own thing. The situation is as described, if you want to do it nobody will stop you. But if you find problems you may need to get your hands dirty.

Something else with this context-per-thread scheme is there no longer any global serialization from a single event loop, depending on what you're doing this may introduce a big burden for locking where none was needed before. But that's a trade-off you can figure out.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/warmcat/libwebsockets/issues/1516#issuecomment-472066545, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxPC9HDOJ2JeD67qbI5Y5OqmnqDEXCQks5vV9JfgaJpZM4bqtgQ .

@kzhdev a bit late to debug it now but this sounds like the openssl init stuff. Modern openssl reference-counts init / destroy apis and doesn't make this kind of confusion. Older openssl blows up even if two different libs in the same process init it.

For others reading this via google, the main point is considering lws uses the single-threaded event loop style, the recommended answer is one context for the user code. That context can handle as many serving vhosts and discrete client connections as you like. None of the scenarios in the examples require multiple contexts... even multiple serving threads is done with one context.

I saw on #961 that you mentioned that one of the points is the old version of OpenSSL initialization and global variables, Is that problem still relevant? For which versions of OpenSSL?

You have the code... see for yourself

https://libwebsockets.org/git/libwebsockets/tree/lib/tls/openssl/ssl.c#n389

AFAIK openssl is the only point that may actually break it. Everything else in lws is dereferenced through the context cleanly.

For now (latest table version) there are more problematic points to work with multiple lws contexts from different threads?

It's going to be bloated... as an optimization on *nix where process fds are ordinals starting from 0, the context allocates some tables at creation for all possible fds the process may meet... you can find out how much with INFO logging. Since these are processwide assets, not just thread-specific, each context must allocate the whole thing. If your process - the whole process - actually only needs a few dozen fds max, you can artificially restrict this either with ulimit or in the context creation struct

https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-context-vhost.h#n358-361

Otherwise... this is FOSS. "Are there ever going to be problems" doesn't have an answer if you're choosing to do your own thing. The situation is as described, if you want to do it nobody will stop you. But if you find problems you may need to get your hands dirty.

Something else with this context-per-thread scheme is there no longer any global serialization from a single event loop, depending on what you're doing this may introduce a big burden for locking where none was needed before. But that's a trade-off you can figure out.

Thanks for fast reply!, Maybe I got something wrong, but as I see it, using single lws context on one thread means that this thread should manage a buffer pool for outgoing requests. In that case, each thread that wish to send a message needs to allocate a buffer and send it to the lws thread using queue/ring buffer. Then, the lws thread should release it. For incoming requests the challenge might be even bigger since I need to: 1) Multiplex the message for the relevant thread which leads to manage some "requests" table 2) Trust my user (thread that is not lws) to release the allocated buffer

I just thought that it might be easier let any thread to operate on its own context with its own lws_context and service loop. Maybe I missed some API/library-ability that meant just to meet these goals?

Thanks again for your time.

"manage a buffer pool for outgoing requests"

Lws is very flexible, but whether a buffer pool is useful or not depends on what you're doing with it. There are several minimal examples that show how to use the lws ringbuffer apis if that's what you need. But it's not part of the core function of lws to do that.

On a server, there's commonly a pretence that you will never run out of memory, so this kind of idea that user code should be insulated from any regulation about that is common. If you want to send a big file, that approach for maximum laziness at the highest level code is dump it all into heap and some magic will send it and deallocate heap as it goes.

Lws approach is to not produce output for send until it can be sent, so even considering kernel buffers, there is a minimum of in-flight data around. That's why lws works fine on really constrained targets like ESP32. Following that scheme properly impacts the design of the top level code, when you produce output, how you manage state, and how concurrency is handled - ie, if threads are actually bringing anything to the party. Another knock-on from that design principle is the buffer you ask to send doesn't have to persist beyond the call to the send api, eg it can be on the stack, so there's nothing for user code to track and deallocate. You have to deal with state in your user code, but it's possible to make very tight and clean implementations this way.

warmcat / libwebsockets

multiple lws_context #1516

include

include

include

include

include

include "libwebsockets.h"

define NUM_THREADS 10