christophberger
commented
7 years ago Hi Richard,
Thanks for your message.
This blog is a hobby of mine, and I do not claim to be an expert in any of the numerous topics I cover in my blog posts. Hence I cannot offer professional advice, although I am always happy to help where I can.
Regarding your particular request, I have read through the description two or three times, and I do not see myself in a position to provide useful answers.
First, I would need to know and understand your scenario in much more detail than what can be described in an email. For example, using etcd for recovery purposes might or might not be a good idea, it depends on many other details of the scenario.
Second, questions around best strategies or best tools for a particular purpose are generally difficult to answer without actually evaluating a few candidates in your particular scenario. The point is, each scenario has its own „best“ strategies and tools, and this depends on numerous factors that would have to be taken into account when selecting possible candidates.
As far as go-mangos is concerned, the protocols it implements define basic communication patterns at a rather low level. You can pick a pattern that fits your scenario and implement an application-level protocol on top of it. The surveyor pattern, for example, provides basic node failure detection through timeouts, but this is only one small building block of a complete recovery strategy that you would need to build on top of that.
The system you plan to build seems complex enough to justify searching for some higher-level library or framework instead. You mentioned that your system would be similar to krakend, so maybe krakend might in fact be a good basis for your system.
Two other feature-rich toolkits are go-kit https://github.com/go-kit/kit and micro https://github.com/micro/micro. They may contain some or most of the features you would have to implement yourself on top of go-mangos.
I cannot really comment on node-red, cascade, goflow, or noflo. Each of these looks very interesting but I really cannot tell whether they might fit nicely into a go-mangos based architecture. For example, cascade already uses another messaging system, ZeroMQ. And the two visual tools, node-red and noflo, seem to be quite self-contained applications with their own infrastructure. Goflow sounds interesting but from the readme I cannot even see if this supports a multi-node architecture at all. The sample code indicates that the flow runs within a single process using goroutines and channel communication. It would seem to me that this is meant to run on a single multicore node rather than on separate nodes (but I might be wrong).
I am afraid this is not quite the answer you were looking for, but I hope I could at least provide some input from a different viewpoint that might help you moving forward.
Good luck with your project!
Best, Christoph
Am 02.02.2017 um 12:19 schrieb roscopecoltran notifications@github.com:
Hi,
Hope you are all well !
Following your great articles/examples about go-mangos and image processing, I have some quick questions about a more complex and advanced image pipeline processing combining go-mangos (ref to previous articles about messaging https://github.com/AppliedGo/messaging) and image processing (link https://appliedgo.net/imageprocessing/).
So if I am correct, the surveyor protocol allows to create nodes of surveyors allowing to send either a json payload/or a binary content to some respondents, to send the request to all respondents and expected a response with a timeout.
What if you want to build up an image processing pipeline with the surveyor protocol with mixed content with dynamic service trees ?
Eg. Image processing pipeline
Note: in our case, it would be more a multi-form data request https://cloud.githubusercontent.com/assets/24416962/22547142/99a90136-e940-11e6-8b2b-fedaf2e07e3d.png Multi-part FORM with a JPEG/PNG image:
Input JPEG/JSON Payload (base 64 encoded JPEG)
/ \ GrayScale Resize (200x200)
/ / \ Gateway_1 Gateway_2 Gateway_3 The goal is to create a flexible distributed api gateway, like krakend https://github.com/devopsfaith/krakend, but with some distributed image based pre-processing steps and aggregate some classifiers JSON responses (rest caffe, torch7)../Search "tcp://:9995" ./InputPreprocessing "tcp://localhost:9995" ./RecursiveSearch "tcp://:9996" "tcp://localhost:9995" ./InstanceMatching "tcp://localhost:9996" (ex match) ./CategoryMatching "tcp://localhost:9996" (ex. caffe) Search / \ Node1 Node2
/ \ Service1 Service2What would be the best messaging strategy for sharing temporarily those pre-preprocessed inputs with the surveyor protocol?(like resizing, convert to grayscale, or to convert the raw jpeg body into a base64, or to wrap the input into a json payload) ? web sockets ? Is it realistic in terms on memory consumption ?
How to manage the service trees dynamically if a pre-processing step failed or is discarded ? Maybe by saving the respondents network configuration host:ports in etcd https://github.com/coreos/etcd ?
What would be the best serialization package to use with go-mangos (msgpck, cap n proto) ?
Are tools like red-node https://github.com/node-red/node-red, cascades https://github.com/cascades-fbp/cascades, or goflow https://github.com/trustmaster/goflow, noflo https://noflojs.org/noflo-ui/ the closest road to build such distribution patterns with go-mangos ?
Thanks for any feedback or insights about the best approach :-)
Cheers, Richard
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AppliedGo/imageprocessing/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/ADqH8lgn50yY7VHsqQPbnlB3VyWQhLcoks5rYbu6gaJpZM4L0_Uq.
Hi,
Hope you are all well !
Following your great articles/examples about go-mangos and image processing, I have some quick questions about a more complex and advanced image pipeline processing combining go-mangos (ref to previous articles about messaging) and image processing (link).
So if I am correct, the surveyor protocol allows to create nodes of surveyors allowing to send either a json payload/or a binary content to some respondents, to send the request to all respondents and expected a response with a timeout.
What if you want to build up an image processing pipeline with the surveyor protocol with mixed content with dynamic service trees ?
Eg. Image processing pipeline
Note: in our case, it would be more a multi-form data request
Multi-part FORM with a JPEG/PNG image:
The goal is to create a flexible distributed api gateway, like krakend, but with some distributed image based pre-processing steps and aggregate some classifiers JSON responses (rest caffe, torch7).
Questions:
What would be the best messaging strategy for sharing temporarily those pre-preprocessed inputs with the surveyor protocol?(like resizing, convert to grayscale, or to convert the raw jpeg body into a base64, or to wrap the input into a json payload) ? web sockets ? Is it realistic in terms on memory consumption ?
How to manage the service trees dynamically if a pre-processing step failed or is discarded ? Maybe by saving the respondents network configuration host:ports in etcd ?
What would be the best serialization package to use with go-mangos (msgpck, cap n proto) ?
Are tools like red-node, cascades, or goflow, noflo the closest road to build such distribution patterns with go-mangos ?
Thanks for any feedback or insights about the best approach :-)
Cheers, Richard