[discussion] Performance - Postgres via UDS

[archenroot]

Hi guys,

I am considering some future work during this year to look at, nothing so much important, but related to overall performance, of the light4j platform as well as possible improvement. Although this is standard in JDBC world, one must know that TCP overhead is deadly. So why not replace it with UDS (Unix Domain Sockets) instead?

You can make your small test by yourself, just execute make: https://github.com/archenroot/ipc-bench

I plan to test some use case:

• HARDER with discuss-able effect especially when HA is required- Event store is not build via db clustering capabilities but instead just on 2-4 independent nodes and consume from event topic new events and persist them (I understand I am will need to handle error scenarios), but either I will use kafka or chronicle queue (it does not support in the moment network replication) for higher throughput, both can be seen as buffers, so when 1 node fails, instead of trying to identify and fix, you can initiate fast redeploy and reprocess buffered messages from T - 1 minute (T - time of failure) - EASIER with real effect - use it for READ only schemas used with CQRS pattern in place, this could have significant performance improvement for heavy READ consuming applications. You can really boost performance here ( I come from background we were handling over 10k requests per second for example), so UDS with PostgreSQL remaining on disk mounted to ram. While postgres supports this natively, I think it is wasted time and better to do it on OS level by creating tmpfs mount and put data files there.

I will leave this open as soon as I am going to possibly incorporate some work old few years, but still not finished (but the POC is fully operational and you can test it on local instances by yourself): https://github.com/impossibl/pgjdbc-ng/issues/169

The fork with POC: https://github.com/harbulot/pgjdbc-ng/tree/unixsocket_poc

Just again came to my head as possible improvement to make light-4j real warrior on all levels of operation.

Ladislav

[stevehu]

A lot of material to read as I never paid attention on UDS before. I don't mind to improve the performance while limiting to Linux as most cloud environments are in Linux anyway. Also, some new shared kernel container techniques will make UDS across containers which is very crucial. I don't like the VMs and I would deployed container on bare metal if possible to remove the extra layer. Security won't be the issue if the host is not shared.

I am planning to start light-rs which is rust based framework these days to aim another group of customers. This year will be a lot of fn:)

[archenroot]

@stevehu - Yeah, I actually use VM in headless mode just to access pure windows apps on linux desktop machines, but when it comes to backend in DC/Cloud, containers are the only option in these days, while I see it still as incubating technologies not serving full potential yet.

Hm, thx for pointing RUST here out, as I am not skilled to code in that language, but it already also took my attention multiple times in the past, I will need to write some real code and compare against java/scala to see how it looks, and more how it works. Even when I read some general description it looks quite promising, thx for hint.... GC per task, not enforced UTF-16, native code, heavy parallel (my personal interest), and much more....

In general the only issue with UDS is that drivers usually implement TCP based (but can differ on db type) communication.

[stevehu]

I would like to try the Postgres with UDS driver some time in the future for our own services. If it works between containers, that would be an option for other service to service communication.

[archenroot]

@stevehu - yeah.