What is the Guidance for Deploying An Agent that will have Access to Docker API?

[javaniecampbell]

I have been playing with autogen and absolutely love it. I was ready to deploy it and then I realized that even though it is early stages, there was no guidance for deployment with consideration for the use_docker flag to be true or image specified in the cloud.

Is it possible to provide some guidance on how to do the following:

• [ ] How to deploy to Azure?
• [ ] How to deploy to AWS ?
• [ ] How to deploy to GCP?
• [ ] How to deploy and allow communication to Docker API in cloud provider?

[thinkall]

Hi @javaniecampbell , thanks for reaching out. Set use_docker=true means running code generated by autogen in a docker container. It will pull python image by default, you can also specify image name. To sum up, it’s a safe code execution env.

As for Azure, AWS and GCP, it depends. If you’re using VMs which support docker, the experience should be the same as your local machine. If you want to deploy your application to containers in the cloud, then it looks like you don’t need to set use_docker to True.

Would be helpful if you could share more details about your use case.

[javaniecampbell]

Hi @thinkall thanks for responding greatly appreciated, the infrastructural use case, I have in mind is using autogen as a microservice api that has isolation execution via the capability of using docker with I find as an idle security, yes I do agree it is absolutely safe and I would want to extend that safety to a real world in the cloud. VMs a pretty expensive for deployment I am wondering how I am capable of using in PaaS model. If docs could explore the deployment options and best practice for deployment would assist in how to go about the deployments 🙏🏾.

The next use case is possible of using autogen in serverless framework like Azure Functions to be able to offload the work of the agents to be able to have a background work.

My practical application is for generating requirements based on series of questions & answers to determine what the user want for planning then create a preview of idea being asked to confirm accuracy of requirements.

[wayliums]

Hi @thinkall thanks for responding greatly appreciated, the infrastructural use case, I have in mind is using autogen as a microservice api that has isolation execution via the capability of using docker with I find as an idle security, yes I do agree it is absolutely safe and I would want to extend that safety to a real world in the cloud. VMs a pretty expensive for deployment I am wondering how I am capable of using in PaaS model. If docs could explore the deployment options and best practice for deployment would assist in how to go about the deployments 🙏🏾.

The next use case is possible of using autogen in serverless framework like Azure Functions to be able to offload the work of the agents to be able to have a background work.

My practical application is for generating requirements based on series of questions & answers to determine what the user want for planning then create a preview of idea being asked to confirm accuracy of requirements.

@javaniecampbell so you would create an API which contain multi agents to solve a problem? Or is that kinda like a chat room still?

[stevetru1]

@javaniecampbell We have some prior experience deploying agents in Azure Functions, it is possible but there are some tricks involved to ensure fast spin-up times, making sure you are not caching results, etc. This isn't something we wanted to open-source "As-Is" without further security and verification guards but if it is a path you are looking to go down I can verify it will work for you.

For the purposes of future development/extension of the AutoGen framework, is the reason you want serverless execution to scale up/out on the problemset beyond what a single VM can handle? I'd love to learn more about why you want to head that direction.

[javaniecampbell]

Hi @thinkall thanks for responding greatly appreciated, the infrastructural use case, I have in mind is using autogen as a microservice api that has isolation execution via the capability of using docker with I find as an idle security, yes I do agree it is absolutely safe and I would want to extend that safety to a real world in the cloud. VMs a pretty expensive for deployment I am wondering how I am capable of using in PaaS model. If docs could explore the deployment options and best practice for deployment would assist in how to go about the deployments 🙏🏾.

The next use case is possible of using autogen in serverless framework like Azure Functions to be able to offload the work of the agents to be able to have a background work.

My practical application is for generating requirements based on series of questions & answers to determine what the user want for planning then create a preview of idea being asked to confirm accuracy of requirements.

@javaniecampbell so you would create an API which contain multi agents to solve a problem? Or is that kinda like a chat room still?

@wayliums yes I would want to create an API to be consumed by external systems which will either be triggered via an event from a queue or service bus to solve a problem in one of my use cases. The other is use it with botframework as the brains for solving a problem given using multi agents to with single responsibility to for workflow orchestration of sorts

[javaniecampbell]

@javaniecampbell We have some prior experience deploying agents in Azure Functions, it is possible but there are some tricks involved to ensure fast spin-up times, making sure you are not caching results, etc. This isn't something we wanted to open-source "As-Is" without further security and verification guards but if it is a path you are looking to go down I can verify it will work for you.

For the purposes of future development/extension of the AutoGen framework, is the reason you want serverless execution to scale up/out on the problemset beyond what a single VM can handle? I'd love to learn more about why you want to head that direction.

@stevetru1 awesome news, if you could share that knowledge, it would be great for those gotchas I love to learn how you accomplished it, if I can learn from your experience would be great.

For background, I have been working on project that generate microservices I want to extend it have the agents get 75% percent of the boilerplate code written that can modified if possible and saved to blob storage or git and do that in way that isn't too I/O intensive hence I thought a serverless might be the best way to manage that load that I could scale out/ fan out that workload especially if I have multiple requests that require several app platforms to generate it microservices needed or have them orchestrated etc amongst other aspects of the application I have mind.

I wanted to take advantage of several things that come with serverless approach:

1. The overhead of not maintaining a VM
2. Less infrastructure to deal with
3. Scaling up/out costs less than a VM currently
4. Easier to extend
5. Can take advantage of orchestration framework like Azure Durable Functions for complex workflows without too much boilerplate code for application architecture
6. Better for event driven approaches in my line of work
7. Throw-away nature of not having a dedicated service for background processing

Etc

I could go on but ideally it's the main workflow for budget friendly infrastructure for me currently, makes it easier to extend without troubling existing deployments for a event driven architecture I have and a level of flexibility to manage easier in scale up or out without too much boilerplate code to deliver value.

If this is solved I believe it can be some learnings for deploying in a kubernetes cluster as well.

[sonichi]

@victordibia @pcdeadeasy for awareness

[HAL9KKK]

very intersting topic, is there any guideline to make it happen, step by step?

[pcdeadeasy]

@HAL9KKK what do you need step wise instructions for? Can you explain what you want to deploy?

[abhishek-ravikumar-kulkarni]

@pcdeadeasy we would step wise instruction about deploying these ai agents on cloud, and also about what are the best practices for the same?

[tyler-suard-parker]

@javaniecampbell We have some prior experience deploying agents in Azure Functions, it is possible but there are some tricks involved to ensure fast spin-up times, making sure you are not caching results, etc. This isn't something we wanted to open-source "As-Is" without further security and verification guards but if it is a path you are looking to go down I can verify it will work for you.

For the purposes of future development/extension of the AutoGen framework, is the reason you want serverless execution to scale up/out on the problemset beyond what a single VM can handle? I'd love to learn more about why you want to head that direction.

@stevetru1 I am having the hardest time getting AutoGen to run on an Azure Functions App. I keep getting an "unable to read database" error. Do you have any pointers for me on how to get this to work?

[sonichi]

Maybe because caching? Have you tried setting cache_seed to None to disable caching?