High-level components on top of uvw

[scherepanov]

First, thanks for nice library! What I found, it is not trivial to work with event-loop based libraries. Even very simple task like open file and read to the end require some knowledge. Next, I found you are repeating same patterns again and again, in every new project. Like creating listener - accepting connections - processing with workers -writing to file etc. In most cases, everything is similar but not completely, and very repetitive. I think it can be a good idea to create library of high-level components on top of event loop library, uvw would be a best candidate. It should work similar to unix pipes. For example, you can read file and send it to network connection with unix shell: cat | nc .... Idea is to create set of components on top of uvw that can be connected similar way as unix pipes. Grand idea is to make event loop coding much simpler and faster, and hide all internals related to event loop structures, callbacks etc. In theory, each functional group of lubuv functionality (and uvw) should have one or two high level components on top.

Going a little bit into details of possible components design... Each component can have several inputs and outputs, similar to stdin, stdout, stderr, i.e. similar to file descriptors. Each input/output is directional (i.e. either input or output), and implement interface that allow to connect output of one component to input of another. This pipe interface is a key to all design. Pipe interface, like unix pipe, should be able to suspend receive of data, propagate error etc. Components will send/receive data through input/output, similar how it is implemented in uvw. And, components can send/receive messages to any other component with commands. I already have in mind several types of components, mostly based on things that I am implementing again and again, and need to implement again and again in new projects. Short list include file reader, file writer, queue, network sender, network receiver, listener, compressor, decompressor, worker, database reader, database writer, hash table (to resolve id->string reference in data stream), and many more... All universe of unix commands can give sample of what can be done. Components should be able to dynamically connect to each other. Ideally I should be able to describe in XML what components I want to use and how to connect their inputs and outputs. I should be able to combine several components into one that can be used. For example, read from file and decompress on several worker is a single combined component that can be reused. Components named, can accept arguments from command line. In theory, you can supply in one command line argument XML description of components and their connections, and with other arguments configure components. For example, program like read file - multithreaded decomrpess - write to file - can be just a set of command-line arguments, one describing 3 components and how they connect, other gives parameters to components. Components will be dynamically created, connected, configured, started, and whole thing will run in event loop with highest possible efficiency. I think it is possible to create pipe interface between components that will not introduce significant performance overhead. I can go on with all the ideas.... Well, sad thing that I cannot really do open source projects due to my employment contract, though in past I was able to get permission to contribute to some degree. Let me know if you are interesting in this project. Prime component is pipe interface, rest of work would be implementing components with pipe interface. Thanks for reading!

[skypjack]

It sounds interesting, even though the risk is that it's not that is to find a good general purpose design for this stuff. What I don't understand is:

Well, sad thing that I cannot really do open source projects due to my employment contract, though in past I was able to get permission to contribute to some degree.

I published this as an open source project but it started as a tool for a paid work. I'm no longer using it and I struggle to keep it up-to-date. Are you suggesting that you can't help and I should work on your idea in my free time? Because in this case I can stop you right away, I don't have enough bandwidth also for this in my free time. I'm sorry.

[scherepanov]

Yes, it is confusing. I am pretty sure you can do some minimal job (i.e. establish project), and then in matter of single day I will tell if I can commit to it. I am pretty sure I will be able to, and will find very quickly. I do understand that spending only your free time on this project is unrealistic and is not going to happen. Good thing I will find very quickly if a project is a go. Just do a stub for a project on github. Chose project name by yourself. I am intended to do "a lot", though you are absolutely by far more advanced in new c++ features (I am doing c++ part time, my primary expertise is completely different). I think it will be a good idea if you will write pipe interface (i.e. abstract class). I will take on writing all actual components, based on interface. Of course, I do look for your comments on code. I am pretty much looking for guidance on implementing library with all modern c++ whistles, pipe interface, and critique on components I will write. I think it will take some time investment from your side on start, on architecture level, then I will do rest, with some help from your side. I think I have a pretty good idea on feature set of a project. My problem is to nicely express them with modern c++.

[scherepanov]

BTW anyway I am intended to do this project as prop code... I can see using open source as a way to get better code.

[skypjack]

Got it, yeah, if we found a small group of persons that are interested/can contribute to the project, I'm all in at partecipating. 👍 I can help but I can't do it all for myself, as I said. Do you have something that works/on which to start reasoning already or are these only ideas at the moment?

[scherepanov]

Those are only ideas at the moment. I came to a point when I want to rewrite all my prop libraries, and it is a patchwork of multi-threading, home-grown loops, queues, worker pools, networking etc. I do want to rewrite on more standardized approach, and this idea came up. I kinda pretty sure it will make my life much easier. Practically everything that I did can be much nicer implemented with linked high-level components on top of UVW, providing even better performance. You have a lot of followers. Let's wait, may be someone will be interesting in participating. I will be "very active" contributor.

[skypjack]

Ok. I'll be out for a week anyway. Let's see if someone pops up in the meantime, otherwise we'll create a repo and try to arrange something. 😉

[escherstair]

I'm not sure I understood completely the idea, but as far as I understand it sounds interesting. You refer to a library of C++ high-level components, I think. Right?

[scherepanov]

Yes, I want library of high-level components. One component that comes to mind is reading file from beginning to end. It is not exactly trivial task with event loop. Components should be able to connect to each other through standardized interface, provide functionality similar to what you get with unix pipes. I got this idea from my code - I am repeatedly coding same components with some variations - file readers, network listeners, network senders, network receivers, queues, workers, compressors etc. I want to take several required components, connect them, and get working program. There are more details, but general idea is here.

[skypjack]

Apparently, either there isn't much interest or people have already created these kinds of classes and don't intend to share them. 🤷‍♂️ I'm closing this issue but feel free to continue the discussion here if you want to proceed. Thank you.