Full support for Kubernetes Deployment

[VenkateshSrini]

Hi, From the Aspire project I would like to generate the deployment and service . yaml. If my app has a console application, then if that is not skipped then I need a Kubernetes Job deployment. I also need to have a facility for specifying of that is a cron job. I need a way to mount my application appSetting.json from Config map. If there are any secrets used in application then I want the K8s secrets also to be generated

[CosminSontu]

In supporting kubernetes deployment it would be great if the component author could control the templating of kubernetes resouces and also control variations per environment.

A low hanging fruit would be to generate yaml files in a Kustomize folder structure. Kustomize deployent support is built-in into kubectl. From my point of view, helm is more appropriate for publishing generic purpose components for wide audiences (nginx, redis, postgres...) , it's overkill as a default for deploying your app's components.

I think supporting Kubernetes as a target will increase traction.

1. Kubernetes is Cloud Portable.

2. Aspire reduces or elliminates the need for sidecars which leads to leaner cluster workloads. (See requests for dapr components (including sidecar container) and you will realise that for a mid-size app with ~10 components (averaging 2 replicas each) you will allocate ~5Gb RAM (requests) with a limit of 20Gb RAM (limits) and ~2CPUs(requests) -> 6CPUs(limits) - only for Dapr to run) based on : https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-production/

This would be a much saner option than targeting Dapr, at least for me, and it would be the trigger for adoption.

[davidfowl]

I would love one of you all to help spike an implementation of a deployment tool that can take this manifest. Here's an example:

https://github.com/dotnet/aspire/blob/50410e1ac8f14b5339ffbe9b2563dfff8db74ef3/samples/eShopLite/AppHost/aspire-manifest.json

The idea would be to parse this and build up a model, then spit out k8s manifest or helm manifests. It would be huge step forward in understanding what we could build here!

cc @brendandburns for thoughts as well.

[nabsul]

Suggestion: instead of spitting out yaml, how about accessing k8s directly via the library/API?

Not exactly the same, but I talk about something similar here: https://nabeel.blog/2022/04/30/yaml-alternative (Especially the "skip yaml" part)

I'd love to collaborate on this, if time permits.

[davidfowl]

Suggestion: instead of spitting out yaml, how about accessing k8s directly via the library/API?

Sounds like a reasonable approach as well. You can just use the Kubernetes API to do the magic.

There are some steps missing though. Take a look at how azd works:

https://learn.microsoft.com/en-us/dotnet/aspire/deployment/azure/aca-deployment-azd-in-depth?tabs=windows

1. You need a container registry
2. You need to build containers for each service
3. Push to the container registry
4. Tell the apphost to generate the manifest
5. Make sure the cluster can pull from the registry
6. Read the manifest and call the k8s API server to deploy

Spitting out manifests or helm charts is more gitops friendly.

We'll be able to natively support steps 2-4 with aspire via msbuild. The tool needs to handle manifest -> target translation

[CosminSontu]

I agree, kustomize / helm are more gitops friendly. I'm thinking flux here which is supported in AKS as a cluster extension.

If yaml is an issue, maybe json can be used for storing metadata (System.Text.Json can be used). Briefly looking through kustomize repo, I see it supports json files.

For now, I will be looking at Aspire to understand how the orchestration code feeds deployment metadata into deployment tooling (starting with the links above).

[davidfowl]

For now, I will be looking at Aspire to understand how the orchestration code feeds deployment metadata into deployment tooling (starting with the links above).

https://github.com/dotnet/aspire/blob/8203451250b032d0e7d0e98c1159d47ada9901d6/src/Aspire.Hosting.Azure/AzureResourceExtensions.cs#L21-L31

[hahn-kev]

looking at that aspire-manifset there's a number of places that it's referencing a postgres connection string, where is it expected that the real connection string comes from? Is there an appsettings json file I'm missing? Or would it be expected that the k8s tool has logic for how to create a postgres connection string?

[davidfowl]

Or would it be expected that the k8s tool has logic for how to create a postgres connection string?

This is the current idea. In the AZD scenario, it queries the deployed resource after provisioning to get the connection string to inject into the containers that need it.

That said, there's another unimplemented idea for container resources that looks like this:

"postgres" : {
   "type": "container.v0",
   "connectionString": "Host={bindings.tcp.host};Port={bindings.tcp.port};UserName=pg;Password={GeneratePassword};",
   "bindings": {
      "tcp": {
         "scheme": "tcp",
         "protocol": "tcp",
         "transport": "tcp"
      }
   }
}

The idea would be for the resource to define the connection string format and replace those tokens with the resolved values (whatever host and port they were deployed to or running on).

I think to make k8s work with external services we have a couple of options:

1. Map those resource types to well-known containers and connection string formats
2. Ask the user for the connection string and treat them as if they are going to be deployed separately.

[RMcD]

Should this integrate with Porter/Duffle/CNAB?

[davidfowl]

@RMcD All 3? No 😄. We should scope this down to what the majority of people are using with a concrete end to end scenario in mind. I don't want to get distracted with what is possible (its possible to do anything really). Let's nail down the workflow for building an app, and deploying to k8s. First manually then using CI/CD and gitops

[deanro]

We use EKS in AWS, so I'd be interested in trying out there. As more Kubernetes support arrives, I'd be willing to try it out and potentially even help with AWS specific options if required.

[pauldotyu]

My 2 cents here. I think generating vanilla K8s manifests based on the Aspire manifest would be the best starting point. Creating a Helm chart would be a bit overkill especially for those in the Kustomization camp and with vanilla K8s manifests, one can just as easily create the Kustomizations using a kustomize create --autodetect command if folks want to go that route.

[VenkateshSrini]

I agree with @pauldotyu. We should start small just by generating the Yaml. I think connecting to cluster via Kubernetes API is sort of over kill. It would require to have the credentials be available to ASPIRE for the SDK to do the A&A. This might not go well with enterprise. For one of the client that I work they use OKTA and multifactor auth for even kubectl access. So we should have those enhancements only after we are able to give reasonably good output YAMLS.

[VenkateshSrini]

I would love one of you all to help spike an implementation of a deployment tool that can take this manifest. Here's an example:

https://github.com/dotnet/aspire/blob/50410e1ac8f14b5339ffbe9b2563dfff8db74ef3/samples/eShopLite/AppHost/aspire-manifest.json

The idea would be to parse this and build up a model, then spit out k8s manifest or helm manifests. It would be huge step forward in understanding what we could build here!

cc @brendandburns for thoughts as well.

The Details provided here is not sufficient to generate the deployment yaml. We need to know the image with complete path. Second we also need to know the exposed port of the application. We need to know the type of application whether it is web or console. Based on whether it is console or Web we will have to see if we have generate job or generate an deployment

[davidfowl]

The details here should be a good starting point.

We need to know the image with complete path.

To know the complete path, the deployment tool must build know the target container registry and must build the container image using dotnet publish --os linux --arch x64 /t:PublishContainer -c Release. You can also publish to a container registry using /p:ContainerRegistry=registry (see https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?pivots=dotnet-8-0 for more details).

This should give you the full image path.

Second we also need to know the exposed port of the application.

For projects 8080 (for http, and ignore https for now), I would start with this since .NET 8+ containers expose this port. Container resources will have a containerPort property on the binding definition.

We need to know the type of application whether it is web or console

There's no job concept so everything is a long running pod (deployment). If there are no bindings then we don't need to create a service.

We don't need to build something enterprise ready in the MVP, we're looking to build a PoC to understand the gaps in the aspire manifest generation 😄 .

Here's AZD's logic for processing the manifest (in golang 🥲 ): https://github.com/Azure/azure-dev/blob/f0e015f2e2810132d2d9b5181b99b530b38cce46/cli/azd/pkg/apphost/generate.go#L215

[CosminSontu]

The details here should be a good starting point.

We need to know the image with complete path.

To know the complete path, the deployment tool must build know the target container registry and must build the container image using dotnet publish --os linux --arch x64 /t:PublishContainer -c Release. You can also publish to a container registry using /p:ContainerRegistry=registry (see https://learn.microsoft.com/en-us/dotnet/core/docker/publish-as-container?pivots=dotnet-8-0 for more details).

This should give you the full image path.

Second we also need to know the exposed port of the application.

For projects 8080 (for http, and ignore https for now), I would start with this since .NET 8+ containers expose this port. Container resources will have a containerPort property on the binding definition.

We need to know the type of application whether it is web or console

There's no job concept so everything is a long running pod (deployment). If there are no bindings then we don't need to create a service.

We don't need to build something enterprise ready in the MVP, we're looking to build a PoC to understand the gaps in the aspire manifest generation 😄 .

Here's AZD's logic for processing the manifest (in golang 🥲 ): https://github.com/Azure/azure-dev/blob/main/cli/azd/pkg/apphost/generate.go#L215

For Kubernetes deployment target I suggest the following prerequisites should be specified.

• kubectl cli: "kubectl" // docker cli isn't needed since dotnet cli can publish directly into an image registry
• image registry url: "..."
• image registry username: "..."
• image registry password: "..."
• kubernetes_context: "..."

Deployment should fail fast and warn if any of these are unavailable / unreachable

[prom3theu5]

Just spent a couple of hours playing around with a possible way to handle kustomize manifest generation

see prom3theu5/aspirational-manifests

What I'm doing is performing polymorphic deserialization of the aspire manifest file, and then, along with the implementation for type specific handlers for each component type, writing Kustomize simple manifests containing a deployment and service.

Handlers are mapped to resource types here

With loading of the aspire manifest here

Because the aspire manifest doesn't include port mappings directly for apps - i'm making the assumption services will be running on port 80 This could easily be expanded by passing in a custom configuration object containing resource key to port mappings, and other settings etc

Also - right now I've made no attempt to construct a db connection string, I'm simply passing all the env variables from the aspire manifest on each resource into config map generators which get mapped on the deployments

We'd likely want to expand the configuration input of the tool to include protected / sensitive values, and have those written to secrets instead.

Image name is also inferred as resource key (the name of the resource) with latest tag The idea being a simple image update can be added to the parent kustomization.yaml file to update all the image tags for instance

images:
  - name: apigateway
    newTag: 1.0.1
  - name: catalogservice
    newTag: 1.0.1

Although the path is there on the resource, so perhaps we interrogate the csproj xml directly?? I guess that is dependant on if an end user is utilising built in container build support.

We could have the templates as code, however while playing around with them was much easier to use Handlebars templating, simply passing in the data we need for transformation

[davidfowl]

Very nice starting point @prom3theu5!

Because the aspire manifest doesn't include port mappings directly for apps - i'm making the assumption services will be running on port 80

The port is 8080 for older .NET projects. We'll be doing work to put this value in the manifest so it doesn't need to be guessed or assumed by the tool.

Also - right now I've made no attempt to construct a db connection string, I'm simply passing all the env variables from the aspire manifest on each resource into config map generators which get mapped on the deployments

I think this is one of the harder things to do that isn't consistent across any of the hosting models. For container resources we plan to have define the connection string format with place holders that the tool should replace after determining what those values are.

Here's an example: https://github.com/dotnet/eShop/compare/main...davidfowl/custom-resource#diff-0480fce9a853ffae810b350c0c31097423ca45247c8fdce9d82c95407638ba45R43

We also need to make sure that service discovery works when we generate the k8s service manifest (cc @ReubenBond). eShopLite has an example of a working k8s deployment manfiest.

We'd likely want to expand the configuration input of the tool to include protected / sensitive values, and have those written to secrets instead.

Yes, this is something @mitchdenny, @ellismg and I have been discussing with the app model. How do we describe that a "secret needs to go here", so the deployment tooling could ask for or generate the secret value, store it securely and make it available to the resources that need it.

Image name is also inferred as resource key (the name of the resource) with latest tag The idea being a simple image update can be added to the parent kustomization.yaml file to update all the image tags for instance

Seems like a fine start. I think the tool can query the project for this information as part of producing the image.

Although the path is there on the resource, so perhaps we interrogate the csproj xml directly?? I guess that is dependant on if an end user is utilising built in container build support.

This would be out default. We wouldn't be parsing XML, you can resolve msbuild properties from the outside as JSON now:

https://github.com/Azure/azure-dev/blob/f00f37d70ae38a8b53d28650b0398ea4d097ec7d/cli/azd/pkg/tools/dotnet/dotnet.go#L208C42-L208C42.

Learn more here https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-8#cli-based-project-evaluation

We could have the templates as code, however while playing around with them was much easier to use Handlebars templating, simply passing in the data we need for transformation

azd is using go templates 😄 (https://github.com/Azure/azure-dev/blob/f00f37d70ae38a8b53d28650b0398ea4d097ec7d/cli/azd/resources/apphost/templates/resources.bicept)

Let's see if we can get eShopLite working in a k8s environment using this prototype as a base. Thanks for kick starting this!

[prom3theu5]

Thanks @davidfowl :)

The port is 8080 for older .NET projects. We'll be doing work to put this value in the manifest so it doesn't need to be guessed or assumed by the tool.

Perfect. For now i'll switch this from 80 to 8080 - should we expose 8443 as well?

I think this is one of the harder things to do that isn't consistent across any of the hosting models. For container resources we plan to have define the connection string format with place holders that the tool should replace after determining what those values are.

I agree - that would make things a lot simpler - looking at the example, from that it would be trivial to implement in the templates

We also need to make sure that service discovery works when we generate the k8s service manifest (cc @ReubenBond). eShopLite has an example of a working

Those manifests are good starting points really - next step I think we should use them as a base by turning them into common manifest types with mustache variables. We can then replace the simple deployments and services I threw together, which should get us a step closer to getting eShopLite generating and running

this would be out default. We wouldn't be parsing XML, you can resolve msbuild properties from the outside as JSON

Wow - this is pretty awesome, didn't know about this - thanks!

azd is using go templates 😄

Ahh nice - I wish there was a lib with support for them in c# - another gap in the eco system there :P I chose handlebars because the syntax is very similar - but we could use Liquid, or something else if people preferred Either way - glad that they are using templates there, easy to maintain :)

[mitchdenny]

Ahh nice - I wish there was a lib with support for them in c# - another gap in the eco system there :P

I here there is this template syntax called Razor which is quite rich ;)

[prom3theu5]

Ahh nice - I wish there was a lib with support for them in c# - another gap in the eco system there :P

I here there is this template syntax called Razor which is quite rich ;)

Haha there is Unfortunately writing yaml in it looks terrible in your IDE with highlighting 😛

---

Restructured things a little to make it more maintainable I've implemented extraction of container information as well now using dotnet msbuild --getProperty:<name>. Really handy that, especially with it returning a json payload of 'Properties' if you request more that one getProperty argument.

I've fallen back to project name in kebab-case format if the image isn't set in the csproj - which I think container building support does

[davidfowl]

What did you get working so far?

[prom3theu5]

Almost ready to test a deployment now of eShopLite.

I've added aspnetcore URL for http only to the environment variables produced for services, to ensure services do run on 8080 at startup

I haven't included https or cert as cert-manager can handle that in the load balancer externally of the container, unless we want internal mutual tls on container to container. Is that a gRPC requirement? (Think it may be)

Have Postgres and redis processing too now, along with their respective handlers.

Little ens are back from school so will be in 3-4 hours when they've gone to bed.

I thought of an issue a few mins ago while in car home. The current implementation of the property service I added doesn't construct the relative path to the projects directly as it has no concept of the working / launch directory of the tool. Small fix, AppContext.BaseDirectory should handle that. will get it done soon as back on my Mac 😃

[davidfowl]

I haven't included https or cert as cert-manager can handle that in the load balancer externally of the container, unless we want internal mutual tls on container to container.

Cut the TLS support for the MVP.

Is that a gRPC requirement?

You shouldn't need to do anything special to make grpc work over http in k8s (with broken load balancing 😄).

[prom3theu5]

Perfect. Makes life a whole heap easier for now

Thanks

Will update progress after tests later

[pjmagee]

Suggestion: instead of spitting out yaml, how about accessing k8s directly via the library/API?

Not exactly the same, but I talk about something similar here: https://nabeel.blog/2022/04/30/yaml-alternative (Especially the "skip yaml" part)

I'd love to collaborate on this, if time permits.

A lot of companies follow gitops for releases via helm or kustomize, e.g flux or argoCD. Not that im against this, but helm charts are familiar and popular for K8s deployments.

[ellismg]

@prom3theu5 - When you complete your spike, would love to take a look and start thinking about how we might be able to work this into azd.

For the azd end, we sort of imagined that the way we'd have this work would be to allow you to change the host of the Aspire project in the azure.yaml from containerapps to aks. In that world, instead of generating our ACA manifests + bicep to construct an ACA container, we'd produce k8s manifests (or helm charts, or some other IaC flavor) and bicep to create an AKS cluster).

Thanks for getting this started.

[wbreza]

@davidfowl Here on the azd team we're just starting on adding support for Helm & Kustomize and this would be a great use case for us to target and to see how we can help.

[prom3theu5]

When you complete your spike, would love to take a look and start thinking about how we might be able to work this into azd.

For the azd end, we sort of imagined that the way we'd have this work would be to allow you to change the host of the Aspire project in the azure.yaml from containerapps to aks. In that world, instead of generating our ACA manifests + bicep to construct an ACA container, we'd produce k8s manifests (or helm charts, or some other IaC flavor) and bicep to create an AKS cluster).

Thanks for getting this started.

That would be fantastic. I've written this in C#, but its all possible in go too.

This is where we are so far.

We have service discovery, redis cache and database (postgres) all working. Cart is broken - Grpc error, not investigated so far, but upon navigating to the cart, the grpc error is thrown. Maybe messaging related as we currently aren't deploying Rabbit - however, that's a small t-shirt as its basically a clone of Redis config wise, with a new data Augment method to handle keys ending in messaging I guess.

Had to start to get a little creative in the Project processor, which handles env variables in the manifests If the binding value included the type things might be a little easier - and if port numbers were available somewhere. You can see here: ProjectProcessor.cs

The methods executed by AugmentData transform the env vars to kube vars.

Db Connection isn't handled really - it just sets it to a value we know will work as thats hard coded in the postgres-server template file.

I replaced the https link to basketservice in the yarp config in api gateway with http, as service discovery complained because we aren't doing anything with tls yet, and I forced http only by overriding the URLs, but that was fine afterwards.

There are a few issues in eShopLite actually

The migrator isn't built as a container - but it is included in the aspire manifest - so interrogating that without the tool having a way to either skip certain projects via arguments, having more information in the manifest that labels it as a non-deployable, or checking properties in the project to see if it actually builds a container (I guess) leaves us with some broken manifests. I guess this should be a job really - but its how we get there...

Again - order processor doesn't get built as a container, so same issue.

At the moment manifests get built for deployments for both of those, and deploy but will be in a constant state of backoff with errors on image pulls 👯

CatalogService also fails to build unless you pass -p:ErrorOnDuplicatePublishOutputFiles=false but it runs fine once its built and pushed

All in all - not a bad night tbh.

[davidfowl]

@prom3theu5 Great work!

Had to start to get a little creative in the Project processor, which handles env variables in the manifests If the binding value included the type things might be a little easier - and if port numbers were available somewhere. You can see here: ProjectProcessor.cs

I think you're going to have to do 2 passes over the model. The first pass the process varibles so you can resolve references after you have collected them all, the second pass to do the templating.

I replaced the https link to basketservice in the yarp config in api gateway with http, as service discovery complained because we aren't doing anything with tls yet, and I forced http only by overriding the URLs, but that was fine afterwards.

You can skip the https binding I see you do it based on the configuration entry right now 😄.

The migrator isn't built as a container - but it is included in the aspire manifest - so interrogating that without the tool having a way to either skip certain projects via arguments, having more information in the manifest that labels it as a non-deployable, or checking properties in the project to see if it actually builds a container (I guess) leaves us with some broken manifests. I guess this should be a job really - but its how we get there...

Currently this is a website no? Why isn't it a container?

Again - order processor doesn't get built as a container, so same issue.

We should fix that, it should be a container. There's currently an issue with workers having bindings in the manifest (which they shouldn't).

CatalogService also fails to build unless you pass -p:ErrorOnDuplicateFileOutput=false but it runs fine once its built and pushed

What commands did you run?

[prom3theu5]

What commands did you run

I initially used: dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true in the CatalogService project directory, however that produces:

Running again with -p:ErrorOnDuplicatePublishOutputFiles=false gets a successful build.

Currently this is a website no?

It is - and it isn't 😛 It performs database migration and then seeds the database in a background service during application initialisation, and exposes a health check, but other than that - I'm not sure it does anything else. The application does not stop after seeding though, and its healthcheck endpoint remains in place

This is its current csproj

No container support directly - I just manually ran it with the catalogdb connection string injected, and postgres proxied through kubectl to my local machine With container support being build into .net now - you can technically just run the publish command and it will build a container, but that won't push it to a registry, and we will still have issues when it comes to parsing the properties to get container image, registry repo and tag etc

[prom3theu5]

Fully working locally - including orders with messaging now

@davidfowl

I think you're going to have to do 2 passes over the model. The first pass the process varibles so you can resolve references after you have collected them all, the second pass to do the templating.

Thats exactly the path I took Was going to clean it up once I got the basic config through, i've done that now here in the file Manifest File Parser I have two collections of handler types, one for components, and another for bindings. I iterate over the collection of env variables after being deserialized, before returning to the command handler. That way by the time they get there, they are transformed - including https endpoints.

There is now nothing hard coded in this apart from username / password for rabbit and postgres. Nothing stopping it being used for any manifest really as long as the project source is there too so that it can read container details from msbuild properties.

I think this has proven that things have been thoroughly thought out before you've released this to the public, as its fantastic that we are able to get here from just a json file - well done to all those involved.

There are only two small issues with generation really (aside from implementing tls) but i'm not convinced they are manifest issues - in fact i'd argue that its possibly the responsibility of the tooling to capture them. They are:

• no way to determine the username and password for postgres to build a valid connectionstring. We can make those either read from env variables or passed in as parameters / prompts in the tool though
• same as above, but for RabbitMq. We also have to use a string mask check on the type of messaging broker to properly setup the connection protocol in the connectionstring.

For both of these then, if passed in / read by the tool, we'd remove them from config map generation, and add them to secret generation.

---

If anyone also wants to test / look where we can improve things, and maybe discuss the config parsing etc that'd be cool, and here is a small guide to get you going:

Clone dotnet/aspire and prom3theu5/aspirational-manifests to the same root folder (wherever your working folder is - i.e. ~/git

I've used Docker-Desktop with k8s enabled, should work fine for podman / rancher-desktop though.

Process:

Start a local registry with:

docker run -d -p 5001:5000 --restart always --name registry registry:2

Small edits to aspire/samples/eShopLite

Add the following to CatalogDb/CatalogDb.csproj (could have added to build command - but need the tool to parse the properties from the project file)

<ContainerRegistry>localhost:5001</ContainerRegistry>
<ContainerRepository>catalogdbapp</ContainerRepository>

And for OrderProcessor/OrderProcessor.csproj

<ContainerRegistry>localhost:5001</ContainerRegistry>
<ContainerRepository>order-service</ContainerRepository>

I also change the ApiGateway appsettings to replace https://basketservice with http://basketservice. This could be done manually in the config map though in the generated manifest file, or in a configMapGenerator with merge set in the top level output kustomization.yml file

Then building all containers with from the root of the eShopLite sample with:

pushd ./ApiGateway
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true
popd

pushd ./CatalogDb
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true
popd

pushd ./CatalogService
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true -p:ErrorOnDuplicatePublishOutputFiles=false
popd

pushd ./BasketService
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true
popd

pushd ./OrderProcessor
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true
popd

pushd ./MyFrontend
dotnet publish --os linux --arch x64 -p:PublishProfile=DefaultContainer -p:PublishSingleFile=true --self-contained true
popd

(Notice the extra property on CatalogService as it references CatalogDb which also tries to output appsettings.json to the output folder resulting in a duplicate file violation)

Now debug aspir8 (aspirational manifests) as there is a launchSetting that will debug in the correct folder if you cloned them side by side.

Grab https://k9scli.io/ if you don't have it - makes working with k8s awesome.

You will find created manifests now in aspire/samples/eShopLite/AppHost/output Make sure you are on your docker-desktop context: kubectl config use-context docker-desktop Deploy those with kustomise build . | kubectl apply -f - or kubectl apply -k .

fire up k9s with k9s and you'll see everything deploy

Press shift + f on the frontend service and it'll ask you which port you'd like to port forward the service to locally (defaults to 8080)

open http://localhost:8080 (or your custom port) and you'll have the working deploy

[davidfowl]

I think this has proven that things have been thoroughly thought out before you've released this to the public, as its fantastic that we are able to get here from just a json file - well done to all those involved.

Thank you! That's a complement. We learnt a lot from making azd work so I'm happy that applies here as well!

@prom3theu5 Can we make the tool automate everything? Including the container build and push to a registry (via dotnet commands)? That would make it more equivalent to the azd experience. I think we need to fill in some of the gaps by prompting or by asking for a configuration file. Something akin to this.

There's discussion about secret handling here https://github.com/dotnet/aspire/issues/770.

Once again thank you for driving this forward, I am hoping we can integrate this quickly once we figure out enough to make the MVP work!

Everyone else on the issue, it would be great if you could try this out. It'll help us flesh out any big missing items and also polish up smaller ones.

Thanks!

[prom3theu5]

You're welcome. It's been fun to work on

Can we make the tool automate everything? Including the container build and push to a registry (via dotnet commands)? That would make it more equivalent to the azd experience.

Yeah sure. I'll add in some support for this in the morning. I'll add a couple of extra commands like azd has such as init, up and down, keeping e2e as it but renaming to generate manifests or something like that. That way you would have the option to do end to end with up or just produce kustomize manifests separately

If I get time tomorrow too I'll add in single project scaffolding via passing in a csproj, but that may have to wait till day after

[VenkateshSrini]

I think this has proven that things have been thoroughly thought out before you've released this to the public, as its fantastic that we are able to get here from just a json file - well done to all those involved.

Thank you! That's a complement. We learnt a lot from making azd work so I'm happy that applies here as well!

@prom3theu5 Can we make the tool automate everything? Including the container build and push to a registry (via dotnet commands)? That would make it more equivalent to the azd experience. I think we need to fill in some of the gaps by prompting or by asking for a configuration file. Something akin to this.

There's discussion about secret handling here #770.

Once again thank you for driving this forward, I am hoping we can integrate this quickly once we figure out enough to make the MVP work!

Everyone else on the issue, it would be great if you could try this out. It'll help us flesh out any big missing items and also polish up smaller ones.

Thanks!

This is also my point. The manifest should be self contained. I'm even looking at something like a plug in architecture where if we there could be some sort of hooks to this K8s Yaml generation. This will help in enterprise level customization.

My second point is when adding Aspire support to project, can we have multiple sub options like Add Aspire support for Docker, Add container support for Kubernetes, Add container support for service fabric etc:-. That way the when Manifest for YAML generation is created it could gather all the relevant information and put in the single manifest. The generators then just consume the manifest and generate the output

[davidfowl]

This is also my point. The manifest should be self contained. I'm even looking at something like a plug in architecture where if we there could be some sort of hooks to this K8s Yaml generation. This will help in enterprise level customization.

Maybe you can build on top of what @prom3theu5 has built and help us understand exactly what that means. I'm not sure I fully understand from this description.

My second point is when adding Aspire support to project, can we have multiple sub options like Add Aspire support for Docker, Add container support for Kubernetes, Add container support for service fabric etc:-. That way the when Manifest for YAML generation is created it could gather all the relevant information and put in the single manifest. The generators then just consume the manifest and generate the output

I don't think we'll do anything like this until it's pretty much VERY well understood what that means (like in version 2).

[prom3theu5]

Got no where near as much free time today as I'd like to have

I've expanded the end to end command though and included two more services (aspire manifest composition and container composition)

End to end process now takes the AppHost directory, generates the manifest, then allows you to select which resources from the manifest to process

It then builds and pushes the containers for the selected components. If encountering Error on duplicate file output it'll prompt to retry with the flag set to false, also if it encounters CONTAINERS1016 (not logged in) it should (not tested) prompt to login, and execute with std output and error redirected to null to surpress any credential output On hitting CONTAINERS1013, registry not found - it'll terminate printing a message to check your registry.

After processing all container actions, it'll move onto what it previously did - generate kustomize manifests

You can see the overall process below - one run with selective services, then a full run with the duplicate erroring catalog-service:

https://github.com/dotnet/aspire/assets/1518610/2dec8ea5-20fc-4e8b-9680-894cec2f5d48

Plan is to add a new init command which will initialise a json config file in the folder, prompting / allowing to set overrides for things like ports (until exposing of containerPort that's been merged in makes it out in a release), as well as registry, repository, maybe a project to image name mapping table? It's all up for discussion. Then when you run e2e it'll merge the file over arguments to customise how commands like container builds run etc, I.e if no values are provided fall back so what's in the csproj

[timheuer]

Awesome work @prom3theu5! I just was playing around with it (submitted a PR for something I ran into on Windows). After that banging my head (our output in dotnet tool install was driving me crazy sending me wrong ways), I was able to do your video demo (using the aspire-starter template).

Is the kustomize build . needed?

I also noticed -- and you have noted in your last comment about capturing inputs etc -- that the resulting deployments don't have the right image registries noted -- here's an example of one of mine:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apiservice
spec:
  minReadySeconds: 60
  replicas: 1
  selector:
    matchLabels:
      app: apiservice
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: apiservice
    spec:
      containers:
      - name: apiservice
        image: :latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        envFrom:
        - configMapRef:
            name: apiservice-env
      terminationGracePeriodSeconds: 180

notice image just has the :latest without the registry URI. Assuming this is one of the inputs you were mentioning before.

Great progress!

[prom3theu5]

Thanks Tim 😄

Is the kustomize build . needed?

No not all I was just showing that the generated manifests were buildable by kustomize

The issue with the image is strange and sounds like a path related bug in the ContainerDetailsService implementation. I did put something in that replaces backslashes with environment separator in as a dirty attempt to normalise paths between OSes that needs revisiting. Was a hack at best at the time ^^.

Something must be wrong with the resolution of the project path when it combines working directory with it from the aspire manifest as it was supposed to fall back to project-name in kebab case if no ContainerImageName property was set in the project file. I'll address tomorrow as away from computer tonight.

The idea behind the input file / json config would also expand on this. I was thinking something along the lines of at a minimum allowing overriding any ContainerRegistry and ContainerRepository MSbuild props, ports and possibly mapping overrides for container names, although not sure how valuable that would be.

I think maybe a template path in there too that allows overriding where aspirate loads it's manifest templates from pre-transform, which would allow someone to edit the templates and store them somewhere for their own use cases etc, expanding on the base templates we have.

[davidfowl]

Something must be wrong with the resolution of the project path when it combines working directory with it from the aspire manifest as it was supposed to fall back to project-name in kebab case if no ContainerImageName property was set in the project file. I'll address tomorrow as away from computer tonight.

Instead of doing that, just use the name in the manifest as the resource name as the fallback. You can set the image name if it hasn't already been set by the project.

[prom3theu5]

That's a good shout.

I think it's what the container support in the .net sdk does if you don't specify a custom image name, which is why I did that. Might be wrong though lol

Looking at the container details service, when I reworked it I actually didn't put a fallback back in. That would explain empty images names ^^, so definitely a bug.

Does need a fall back.

We would have to introduce the check for container details earlier which extracts values from props. Currently it happens in generate manifests within the project processor. Moving it to BuildAndPush in the processor pre execution of build would allow us to first see if the value was set, and if not use the resource key from the manifest. Sounds much cleaner. We'd want to also return the container details with the current boolean from that method upstream into the e2e command so that it can be passed downstream into the manifest generation after build cycles complete, so that it doesn't need to recall the props extraction where it needs it now. Or cache it 😃

[CosminSontu]

The idea behind the input file / json config would also expand on this. I was thinking something along the lines of at a minimum allowing overriding any ContainerRegistry and ContainerRepository MSbuild props, ports and possibly mapping overrides for container names, although not sure how valuable that would be.

I suggest a fail-fast approach (with proper error messages, indicating the required properties to be defined) in case those properties are missing. A permanent warning in case ContainerRegistry is not defined, saying something like : "Local registry is being used. Please use for specifying a remote registry."

For brevity, these are all container image related properties in csproj:

<PropertyGroup>
   <ContainerRepository>sampleservice</ContainerRepository>
   <ContainerImageTags>5.0;latest</ContainerImageTags>
   <!--<ContainerImageTag>5.0</ContainerImageTag>--> <!-- variation accepting a single tag -->
   <!--<ContainerRegistry>registry.mycorp.com:1234</ContainerRegistry>--> <!-- local registry if not specified -->
 </PropertyGroup>

Nice work!

[prom3theu5]

A permanent warning in case ContainerRegistry is not defined, saying something like : "Local registry is being used. Please use for specifying a remote registry."

I've just completed the rework - and fixes - for gathering of container properties pre build, and implemented fallback resource key for image name if not specified Afaik - ContainerRegistry is the only 'required' parameter, builds will still pass without it - but the images would be stored locally only.

I've implemented a check there, and we fail fast now with a message stating it must be set, and the project that it is missing in Once the init command is added, which would allow you to define an override / fallback yourself through guided composition, we'd have to plug those values into this check to ensure its not overridden.

Next pass after lunch will be passing the details from the cache (dictionary) into the build and push method in the Container Composition service, so that we can augment the build command to pass all our gathered parameters as properties.

Tim also suggested that we allow reading of trimming / self contained / single file etc. I'll cover those same time, to augment the builds, as I think thats a good idea

---

Update

Covered all props apart from SelfContained in the last lot of work I merged, which cleans up all the command building with an argument builder implementation.

Haven't covered SelfContained as eShopLite throws an error when trying to build the CatalogService due to it referencing CatalogDb "non-self contained executable cannot reference self contained executable"

However both projects do not contain true in their project files, and dotnet msbuild --getProperty:SelfContained returns false for them both, but you cant publish the container without the CatalogService value being set to true during build.

For now - i've left self contained defaulting to true in all builds. I guess the worse that can happen there is you get a bunch of runtime libs in your container where the base image may also have the runtime installed??

---

Update 2

Now supports the init command.

aspirate init

from within your AppHost directory or

aspirate init -p /path/to/AppHost

which will guide you through creating the aspirate.json bootstrap file (purely optional). The supported options are

• ContainerRegistry: setting this means you do not need one in your csproj, and if it isn't found - all builds will use this.
• ContainerTag: will override the container tag used if not in your csproj - if not specified in settings, will fall-back to latest.
• TemplatePath: this customises the path used when loading templates that get transformed to manifests, you can take the templates folder from the source, and modify to your hearts content with all your custom changes, and as long as you don't remove the placeholders, aspirate will use those instead of its built in. More on this and possible use cases (such as adding jobs to create databases etc) when we have docs....

this means the process from going from zero, to a dev instance of the starter template is just

dotnet new aspire-starter --use-redis-cache --output TestAspire
cd ./TestAspire/AppHost
aspirate init
[enter y and 'localhost:5001' fall-back registry - no to rest]
aspirate e2e -o ./output
kubectl apply -k ./output

Maybe time for a release on nuget, or github packages? May pick up adoption rate until azd has native support, and we can get some feedback ^^

[davidfowl]

Some feedback

• I would rename the e2e command
• as a future idea, allow users to scaffold the template for a particular resource as a starting point for overriding it.
• We should work out container resources, this doesn’t work in azd today either.

This looks amazing!

Maybe time for a release on nuget, or github packages? May pick up adoption rate until azd has native support, and we can get some feedback ^^

Yes! Put it on NuGet so we can start getting others to test it!

I haven’t seen more feedback here but I hope others have a chance to play with it.

PS: I think we do eventually want this to be part of the project for cloud specific azure k8s deployments. It does get weird if you have both azure resources and k8s as your target. I think you might use azd for that.

[prom3theu5]

as a future idea, allow users to scaffold the template for a particular resource as a starting point for overriding it.

That's crazy - I literally just implemented that and merged 👍 So when you bootstrap now with init it prompts you:

/Users/prom3theu5/git/personal/aspirational-manifests/src/Aspirate.Cli/bin/Debug/net8.0/Aspirate init

Aspirate supports setting a fall-back value for projects that have not yet set a 'ContainerRegistry' in their csproj file.
Would you like to set a fall-back value for the container registry? [y/n] (n): n

Aspirate supports setting a fall-back value for projects that have not yet set a 'ContainerTag' in their csproj file.
Would you like to set a fall-back value for the container tag? [y/n] (n): n

Aspirate supports setting a custom directory for 'Templates' that are used when generating kustomize manifests.
Would you like to use a custom directory (selecting 'n' will default to built in templates ? [y/n] (n): y
Please enter the path to use as the template directory: ./templates

(✔) Done: Set 'TemplatePath' to '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

Selected Template directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates' does not exist.
Would you like to create it? [y/n] (y): y

(✔) Done: Directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates' has been created.

Selected Template directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates' is empty.
Would you like to populate it with the default templates? [y/n] (y): y

(✔) Done: copied template 'redis.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: copied template 'rabbitmq.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: copied template 'deployment.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: copied template 'postgres-server.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: copied template 'kustomization.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: copied template 'service.hbs' to directory '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/templates'.

(✔) Done: Configuration for aspirate has been bootstrapped successfully at '/Users/prom3theu5/git/personal/aspire/samples/eShopLite/AppHost/./aspirate.json'.

 🚀 Execution Completed

That way - you can reuse an existing directory you have setup for templates, or scaffold with the bootstrap and get to editing

I would rename the e2e command

Yeah it's a bit 'meh' isn't it. Any suggestions? render? (make-it-so? 😄 )

Yes! Put it on NuGet so we can start getting others to test it!

Awesome - I'll add a couple of actions in the morning to handle cicd, and manual workflow executions.

We should work out container resources, this doesn’t work in azd today either.

I'll have a play around tomorrow to see what gets added to the manifest for them

[pjmagee]

e2e 

Why not use apply, similar to kubectl? It is kind of applying what is requested so i still feel like it's correct?

[davidfowl]

generate - Spit out kube manifests apply - apply to the cluster (dont create manifest)

[hahn-kev]

kustomize uses the term build for rendering out the resources, which you would then apply.

[davidfowl]

What does helm use?

[prom3theu5]

kustomize uses the term build for rendering out the resources, which you would then apply.

That's right - but kubectl actually has build in support for kustomize with -k, i.e. kubectl apply -k ./someFolder and destroy support too

What does helm use?

Helm uses install, but you first have to add a repo. Its similar to adding a private nuget feed, then installing a global tool from it I guess, so two commands in total

I used apply because its along the lines of what anyone utilising kubectl would expect.

generate - Spit out kube manifests apply - apply to the cluster (dont create manifest)

Was already working on those when you posted 😄 Release 1.0.1 has just deployed to nuget, which includes:

• Renamed the e2e command to 'generate'
• introduce the 'apply' command to allow deployment of manifests
• introduce the 'destroy' command to allow removal of deployed manifests

Both the apply and destroy command will first use kubectl to gather possible contexts defined in your current active kubeconfig, and present a menu allowing you to choose which context you'd like to use operate on, first confirming that you actually want to work

Full Command List is now:

---

I guess now that its up on nuget, and gone through a lot of changes, its time for a new video. Here is from zero to hero lol (dotnet new aspire-starter through to deploy and removal)

https://github.com/dotnet/aspire/assets/1518610/8e6563a6-b5d4-4421-8d0f-3e92e44147aa