search

Azure / azure-dev

A developer CLI that reduces the time it takes for you to get started on Azure. The Azure Developer CLI (azd) provides a set of developer-friendly commands that map to key stages in your workflow - code, build, deploy, monitor, repeat.
https://aka.ms/azd
MIT License
410 stars 198 forks source link

azd init fails on eshop - multiple binding ports #3123

Closed davidfowl closed 9 months ago

davidfowl commented 11 months ago

Tried to run azd init on eshop https://github.com/dotnet/eShop/pull/136

Errors:

Here's the manifest:

{
  "resources": {
    "redis": {
      "type": "container.v0",
      "image": "redis:latest",
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 6379
        }
      },
      "connectionString": "{redis.bindings.tcp.host}:{redis.bindings.tcp.port}"
    },
    "EventBus": {
      "type": "container.v0",
      "image": "rabbitmq:3-management",
      "env": {
        "RABBITMQ_DEFAULT_USER": "guest",
        "RABBITMQ_DEFAULT_PASS": "{EventBus.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5672
        },
        "management": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http",
          "containerPort": 15672
        }
      },
      "connectionString": "amqp://guest:{EventBus.inputs.password}@{EventBus.bindings.management.host}:{EventBus.bindings.management.port}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "postgres": {
      "type": "container.v0",
      "image": "ankane/pgvector:latest",
      "env": {
        "POSTGRES_HOST_AUTH_METHOD": "scram-sha-256",
        "POSTGRES_INITDB_ARGS": "--auth-host=scram-sha-256 --auth-local=scram-sha-256",
        "POSTGRES_PASSWORD": "{postgres.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5432
        }
      },
      "connectionString": "Host={postgres.bindings.tcp.host};Port={postgres.bindings.tcp.port};Username=postgres;Password={postgres.inputs.password};",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },
    "CatalogDB": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "IdentityDB": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "OrderingDB": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "WebHooksDB": {
      "type": "postgres.database.v0",
      "parent": "postgres"
    },
    "identity-api": {
      "type": "project.v0",
      "path": "../Identity.API/Identity.API.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__IdentityDB": "{IdentityDB.connectionString}",
        "BasketApiClient": "{basket-api.bindings.http.url}",
        "OrderingApiClient": "{ordering-api.bindings.http.url}",
        "WebhooksWebClient": "{webhooksclient.bindings.http.url}",
        "WebhooksApiClient": "{webhooks-api.bindings.http.url}",
        "WebAppClient": "{webapp.bindings.https.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "basket-api": {
      "type": "project.v0",
      "path": "../Basket.API/Basket.API.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__redis": "{redis.connectionString}",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "Identity__Url": "{identity-api.bindings.http.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http2"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http2"
        }
      }
    },
    "catalog-api": {
      "type": "project.v0",
      "path": "../Catalog.API/Catalog.API.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "ConnectionStrings__CatalogDB": "{CatalogDB.connectionString}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "ordering-api": {
      "type": "project.v0",
      "path": "../Ordering.API/Ordering.API.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "ConnectionStrings__OrderingDB": "{OrderingDB.connectionString}",
        "Identity__Url": "{identity-api.bindings.http.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "order-processor": {
      "type": "project.v0",
      "path": "../OrderProcessor/OrderProcessor.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "ConnectionStrings__OrderingDB": "{OrderingDB.connectionString}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "payment-processor": {
      "type": "project.v0",
      "path": "../PaymentProcessor/PaymentProcessor.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "webhooks-api": {
      "type": "project.v0",
      "path": "../Webhooks.API/Webhooks.API.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "ConnectionStrings__WebHooksDB": "{WebHooksDB.connectionString}",
        "Identity__Url": "{identity-api.bindings.http.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "mobile-bff": {
      "type": "project.v0",
      "path": "../Mobile.Bff.Shopping/Mobile.Bff.Shopping.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "services__catalog-api__0": "{catalog-api.bindings.http.url}",
        "services__catalog-api__1": "{catalog-api.bindings.https.url}",
        "services__identity-api__0": "{identity-api.bindings.http.url}",
        "services__identity-api__1": "{identity-api.bindings.https.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "webhooksclient": {
      "type": "project.v0",
      "path": "../WebhookClient/WebhookClient.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "IdentityUrl": "{identity-api.bindings.http.url}",
        "CallBackUrl": "{webhooksclient.bindings.http.url}",
        "services__webhooks-api__0": "{webhooks-api.bindings.http.url}",
        "services__webhooks-api__1": "{webhooks-api.bindings.https.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "webapp": {
      "type": "project.v0",
      "path": "../WebApp/WebApp.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "ConnectionStrings__EventBus": "{EventBus.connectionString}",
        "IdentityUrl": "{identity-api.bindings.http.url}",
        "CallBackUrl": "{webapp.bindings.https.url}",
        "services__basket-api__0": "{basket-api.bindings.http.url}",
        "services__basket-api__1": "{basket-api.bindings.https.url}",
        "services__catalog-api__0": "{catalog-api.bindings.http.url}",
        "services__catalog-api__1": "{catalog-api.bindings.https.url}",
        "services__ordering-api__0": "{ordering-api.bindings.http.url}",
        "services__ordering-api__1": "{ordering-api.bindings.https.url}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    }
  }
}
rajeshkamal5050 commented 11 months ago

https://github.com/dotnet/eShop support requires adding rabbitmq https://github.com/Azure/azure-dev/issues/3060 and anything new which comes up. Adding to the Germanium bucket for now. We can pull into appropriate milestones once we get the dependent resource types added.

@weikanglim noticed difference in validation errors from preview 1.

main fails with ERROR: unsupported resource type: rabbitmq.server.v0 preview2 fails with ERROR: configuring ingress for resource EventBus: the transport property of all bindings should match`

Has anything changed on Aspire in upcoming preview 2?

cc @ellismg @vhvb1989 @davidfowl @mitchdenny

davidfowl commented 11 months ago

I just wanted to track the work to make eShop work with azd. I think there are lots of little small issues we are running into like the ones filed above.

ellismg commented 10 months ago

Should be much improved with #3131, but I think we will still have problems with the end to end because of this:

    "EventBus": {
      "type": "container.v0",
      "image": "rabbitmq:3-management",
      "env": {
        "RABBITMQ_DEFAULT_USER": "guest",
        "RABBITMQ_DEFAULT_PASS": "{EventBus.inputs.password}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5672
        },
        "management": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http",
          "containerPort": 15672
        }
      },
      "connectionString": "amqp://guest:{EventBus.inputs.password}@{EventBus.bindings.management.host}:{EventBus.bindings.management.port}",
      "inputs": {
        "password": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 10
            }
          }
        }
      }
    },

I suspect that we don't do the right thing when faced with this bindings block today, since we don't yet support multiple ingress ports on ACA. Support for that is now in preview for ACA, but it would be helpful for me to understand what the intention of this block is. Is the intention we'd build something that looks like this for the Ingress block:

Is the intention that we use the http ingress for the "management" binding (so we listen on both 80 and 443) and forward that to 15672 and then use the additional ports feature to also expose 5672? If so, is it correct that the connection string references the "management" binding and not the "tcp" binding? I would expect the port in the connection string needs to be 5672? If the connection string is indeed correct, what value should be returned for port here? Is it 15672 because that's the containerPort is (seems wrong, because we won't configure ingress on that port, right?) or is it 443 or 80 (and how did we decide?)

@davidfowl / @mitchdenny interested in your help in making sense of how we want to configure ACA here...

mitchdenny commented 10 months ago

I think this is one of those times that you might end up having to ask the user what they want. You might take a stab at figuring it out and then just present it to the user for confirmation. In this particular case I'd probably bind the management port to HTTP and then use the preview features in ACA for the TCP port.

At the end of the day we are ultimately constrained by what the underling hosting platform can support in terms of what containers we can run. So if ACA in its current GA state can't host Rabbit MQ then that might just be a limitation we have for now until support for specifying multiple bindings in ACA is better supported.

Maybe, given ACA does support this in preview, the prompt is "do you want us to use ACA preview features to support this container?"

Mortana89 commented 9 months ago

I am facing the same issue with azd init; ERROR: evaluating value for servicesmetering.api0: unknown resource referenced in binding expression: metering This is an Aspire project with 27 services (yes, testing the boundaries here!). One gateway (ocelot) and then api and worker services. We only use the keyvault reference, for the rest we use Aspire as the tool to run all services at once. Was hoping azd init would prep our bicep template to facilitate migration to ACA :)

ellismg commented 9 months ago

@Mortana89 - If I had to wager a guess, I think the problem here is that you have a resource in your apphost named metering.api (e.g. something like .AddProject<...>("metering.api")? and then you reference that resource from another resource.

I think what is happening here is we have some expression in the manifest that looks like:

{metering.api.url}

Which azd is interpreting incorrectly (since it breaks this apart into parts based on the . separators).

You might for now be able to rename the resource to something like metering-api and have more luck.

It would be helpful for me if you could confirm my suspicion.

@mitchdenny How might we want to handle cases where the resource has a dot in its name and we need to generate a path expression like this? I feel like we need to have the manifest writer do some escaping here?

Mortana89 commented 9 months ago

You are right, thanks for the swift response! Replacing dots with dashes did the thing!

davidfowl commented 9 months ago

FWIW in the next preview the naming of resources is restricted to avoid these.

ellismg commented 9 months ago

Okay, I ran this agian today on https://github.com/dotnet/eShop/commit/3b49f61a888656b038b5f044b0d4c5096fc9f073 and with the latest AZD and we just have this error now:

ERROR: configuring ingress for resource EventBus: the transport property of all bindings should match

The problem here is due to the binding section of the EventBus container resource:

      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "containerPort": 5672
        },
        "management": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http",
          "containerPort": 15672
        }
      },

This is trying to expose two ports on the service - and presently we only support using a single port. ACA does have support for multiple ports (in preview): https://learn.microsoft.com/azure/container-apps/ingress-how-to#use-additional-tcp-ports, so we need to update azd to use this feature.

One question I do have for @davidfowl is how he imagines we would configure ACA for the management binding. I am guessing we would want to use the normal ACA HTTP ingress stuff to expose the management plane of HTTP on port 80 (and have it bind that to the 15672 container port) and then for the TCP binding do an additional binding for port 5672, but I'm not sure if we need to find a way to also have port 15672 open for http traffic open as well?

I also now recall an offhanded comment from @davidfowl that mentioned that there might be some issues with using the ACA additional TCP ports feature, so if you learned anything from your experiments, let me know and I can figure out how to synthesize it in ACA.

rajeshkamal5050 commented 9 months ago

Remaining items for making eshop work,

vhvb1989 commented 9 months ago

based on Matt's comment above:

ACA does have support for multiple ports (in preview): https://learn.microsoft.com/azure/container-apps/ingress-how-to#use-additional-tcp-ports, so we need to update azd to use this feature.

@davidfowl @mitchdenny , are you expecting azd to support this feature? If yes, the aspire manifest should label which is the main ingress for the appContainer. All other bindings would go to additionalPortMappings and be limited to what's documented here: https://learn.microsoft.com/azure/container-apps/ingress-overview#additional-tcp-ports

davidfowl commented 9 months ago

Yes we should support this feature.

vhvb1989 commented 9 months ago

good.

azd just need to know, from the manifest, which ports are additionalPorts and which one is the main ingress. According to the docs, there could be at most 5 additionalPorts :)

Do you have an issue on Aspire to define additionalPorts as bindings?

mitchdenny commented 9 months ago

We have an issue tracking marking endpoints as public. I think this needs to fold into that same API design.

rajeshkamal5050 commented 9 months ago

@vhvb1989 @mitchdenny @davidfowl I am assuming eshop works now? Can we close this one?

Given that for the respective feature work items we have,

vhvb1989 commented 9 months ago

closing this one, as the original issue has been addressed and new issues have been created