Auto scaling Nodered Viseo

[jacyuan]

Hello,

Thanks a lot for this great tool of building chatbot. I've got one question related to the use of viseo with docker and kubernetes/openshift.

Indeed, we packaged viseo into a docker image, and launch the image with the help of kubernetes/openshift, with the possibility of auto scaling the container.

It seems that (please correct me if it's not the case), in Viseo, some conversation states (such as prompt sessions) are stored temporally in memory of the current running container, waiting for the user reply in order to continue the following conversation.

In case if we use more than one (viseo) container for backend (which is necessary under some circumstances), and with a simple task dispatcher (dispatch randomly) to transfer input message to viseo docker containers, the final user's gonna lose his/her current conversation context.

Do we have some possibilities to externalize (saving into a db or file maybe) these conversation states / sessions ?

Thank you very much

[JpEncausse]

Hi @jacyuan, Yes it's a limitation of the framework, we store a callback in memory for prompt message so we can get all the previous data on prompt.

A workaround is to set a session affinity (to conversation id) and keep the conversation on the same docker.

We have plan to store this callback in a common database or redis but it's tricky because

• we won't store the full object (risk of regression)
• we don't know how to let choose the storage
• building a kind of callback is not that simple

So we wait to have a real life usecase with a customer to implement it.

What kind of chatbot are you working on ? Did you do some loadtesting ? Would be interested in your docker architecture

[jacyuan]

Hello @JpEncausse, thank you for your reply.

I agree that to save conversation callbacks into an external layer is complicated (since there are a lot of context, etc, to store / and to restore other than callbacks themselves).

I think we're gonna check two solutions currently:

• try to see whether we can customize our load balancer (under openshift), in order to dispatch requests based on convId.
• otherwise, try to add more resource for a single container

Our context: We implement chatbots mainly for internal usages. Usually to replace classical faq sites, and sometimes, with some api calls. And we haven't yet launched a complete loadtesting for it. We packaged nodered viseo into a docker image, and leave openshift to manage all pipeline cicd aspects.

[JpEncausse]

Yes we introduce a unique node id in previous version to be able to "call back" a node with a given content. I think we will store in external layer the flow data but we have to figureout what to strip from the flow data.

What's you company ? Do you have some metrics about request to the server ? Because

• you can setup a failover server (yes you lost prompt but you handle crash / load issues)
• a Node-RED server should handle 30req/s on simple flow (without API calls)
• you can also build a bot per "feature" or "language" or "personality" to split the traffic. We use that kind of strategy for some customers.

[jacyuan]

Sorry @JpEncausse for the late reply.

In our case, we can have multiple running instances of nodered (managed by auto scaling under openshift), in case that one instance crashes, other ones will take control (but yes, prompts sessions will be lost). In most case till now, we don't really have heavy load to manager in current projects (but I'd rather like to be ready for future projects).

I see the idea to split loads by feature to different instances / nodes, somehow, like micro services if I understand well what you said. It could absolutely be a good idea if the management domain under chatbot continues to grow !

For further details, I'll contact you by mail later if you wish.