search

babelouest / ulfius

Web Framework to build REST APIs, Webservices or any HTTP endpoint in C language. Can stream large amount of data, integrate JSON data with Jansson, and create websocket services
https://babelouest.github.io/ulfius
GNU Lesser General Public License v2.1
1.07k stars 183 forks source link

[Feature request] How to use libulfius to implement a secure, asynchronious and scalable REST API? #263

Closed dancesWithCycles closed 1 year ago

dancesWithCycles commented 1 year ago

Is your feature request related to a problem? Please describe. Hi folks, Thank you so much for providing and maintaining this repository!

I startet a Proof of Concept (PoC) for a REST API with CRUD endpoints using libulfius insprired by the simple example. When I deploy that PoC and use the API endpoints simultaneously from several different web clients, the PoC get stuck and the clients result in a live or dead lock.

Describe the solution you'd like I have no solution yet. I am looking forward to use a REST API based on libulfius no matter how many web clients are calling API endpoints.

Describe alternatives you've considered An alternative is to use only one web client. Anyhow, that is not the use case.

Additional context I have deployed that PoC on a GNU/Debian operating system using a systemd service configuration.

I appreciate sharing your views on this.

Cheers!

dancesWithCycles commented 1 year ago

Hi folks, I just learned, what caused this issue and I like to share my observation.

My web client is a React.js app. This app is rendered in a so called StrictMode.

root.render(
    <React.StrictMode>
    <App />
    </React.StrictMode>
);

In a development environment the StrictMode makes the app render twice. You find more details at this link.

When my libulfius REST API service is called twice due to the render twice mode, I can see from the log entries that my service get stuck.

callback_get_entities() Started...
callback_get_entities() Started...
callback_get_entities() Done.

I get the same behavior when I substitude the React.js web client with the following command line instruction.

curl -X GET http://host:port/entities/info & curl -X GET http://host:port/entities/info

I conclude, my libulfius based REST API service always get stuck when an API endpoint is called while the last call of the same endpoin is not finished.

When I am using two different endpoints as part of the curl instruction

curl -X GET http://host:port/entities/info & curl -X GET http://host:port/entities/42/info

I am even running into a different issue that is logged like this.

callback_get_entities() Started...
callback_get_entity()  Started...
callback_get_entity() Done.
message type 0x31 arrived from server while idle
message type 0x32 arrived from server while idle
message type 0x54 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x31 arrived from server while idle
message type 0x32 arrived from server while idle
message type 0x54 arrived from server while idle
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
callback_get_entities() No data retrieved
callback_get_entities() Done.

I can also learn from the logs that my PostgreSQL database is complaining.

This is the /entities route: Dba error!

A part of the endpoints is a query at the database. I am wondering if two simultaneous database queries are rather the issue then two simultaneous API endpoint calls.

I am appreciating any comments about this. I will let you know about the latest investigations.

Cheers!

babelouest commented 1 year ago

Hello,

Concerning Ulfius' performance, I wouldn't say it's designed for massive parallel applications, because my goal at first is to provide an easy-to-use http framework in C.

The question about performance has been asked several times, check the issues for example: https://github.com/babelouest/ulfius/issues?q=is%3Aissue+performance+ , but I wouldn't know with absolute precision how fast can an ulfius program run.

It uses libmicrohttpd underneath with some specific configuration, especially one thread per client: MHD_USE_THREAD_PER_CONNECTION, see libmicrohttpd documentation for more details. If you know what you're doing, you can use ulfius_start_framework_with_mhd_options described here, which will allow you to use your own libmicrohttpd configuration underneath.

In your second message, you mention issues with the PostgreSQL database, which could be the cause of your deadlocks. When I use libpq, I need a mutex to make sure my single connection isn't used by 2 threads at the same time. You can check libpq's documentation for more details.

dancesWithCycles commented 1 year ago

In your second message, you mention issues with the PostgreSQL database, which could be the cause of your deadlocks. When I use libpq, I need a mutex to make sure my single connection isn't used by 2 threads at the same time. You can check libpq's documentation for more details.

Hi @babelouest , Thank you so much for your helpful reply. My investigation in the meantime supported that this issue is due to libpq and not due to libulfius. I am sorry for any inconvenience. Thank you again for the links and hints above. They sound helpful to continue my investigation. Cheeers!