Can't create FargateService

polkx commented 4 months ago

When I try to create a FargateServiceArgs class the Error shows up and tells me that I must add taskDefinition or taskDefinitionArgs. Even when I add taskDefinition, error still shows up. Example:

Dependencies:

//> using scala "3.3.1"
//> using options -Werror -Wunused:all -Wvalue-discard -Wnonunit-statement
//> using plugin "org.virtuslab::besom-compiler-plugin:0.2.1"
//> using dep "org.virtuslab::besom-core:0.2.1"
//> using dep "org.virtuslab::besom-awsx:2.5.0-core.0.2"

Main class:

import besom.*
import besom.api.aws.ecs.Cluster
import besom.api.awsx.ecs.{FargateService, FargateServiceArgs}
import besom.api.awsx.ecs.inputs.{
  FargateServiceTaskDefinitionArgs,
  TaskDefinitionContainerDefinitionArgs,
  TaskDefinitionPortMappingArgs
}

@main def main = Pulumi.run {
  val cluster = Cluster("cluster")

  val taskDefinition =
    FargateServiceTaskDefinitionArgs(
      container = TaskDefinitionContainerDefinitionArgs(
        name = "my-task-def",
        image = "nginx",
        memory = 512,
        portMappings = List(TaskDefinitionPortMappingArgs(containerPort = 80))
      )
    )

  val service = FargateService(
    "my-service",
    FargateServiceArgs(
      cluster = cluster.arn,
      // taskDefinition = "my-task-def",
      taskDefinitionArgs = taskDefinition,
      desiredCount = 1
    )
  )

  Stack.exports(url = service.urn)
}

Stack trace:

Previewing update (dev):
     Type                        Name                   Plan       Info
 +   pulumi:pulumi:Stack         aws-hello-fargate-dev  create     2 errors; 37 messages
 +   └─ awsx:ecs:FargateService  my-service             create     

Diagnostics:
  pulumi:pulumi:Stack (aws-hello-fargate-dev):
    2024.02.21 11:44:24:850 scala-execution-context-global-20 [resource: my-service[awsx:ecs:FargateService]] ERROR besom.internal.ResourceDecoder.resolve:91
        Resolve resource: received error from gRPC call: 'UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.', failing resolution
    2024.02.21 11:44:24:877 main  ERROR besom.internal.BesomModule.run:69
        io.grpc.StatusRuntimeException: UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
                at io.grpc.Status.asRuntimeException(Status.java:537)
                at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
                at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
                at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
                at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
                at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
                at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
                at java.lang.Thread.run(Thread.java:1583)

    Exception in thread "main" io.grpc.StatusRuntimeException: UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
        at io.grpc.Status.asRuntimeException(Status.java:537)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:538)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

    Error: Either `taskDefinition` or `taskDefinitionArgs` must be provided.: Error: Either `taskDefinition` or `taskDefinitionArgs` must be provided.
        at new FargateService (/snapshot/awsx/bin/ecs/fargateService.js:47:19)
        at awsx:ecs:FargateService (/snapshot/awsx/bin/resources.js:25:45)
        at construct (/snapshot/awsx/bin/resources.js:43:12)
        at Provider.construct (/snapshot/awsx/bin/index.js:39:52)
        at Server.<anonymous> (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:315:52)
        at Generator.next (<anonymous>)
        at fulfilled (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

    error: besom.internal.ResourceDecoder.resolve:91 [resource: my-service[awsx:ecs:FargateService]] Resolve resource: received error from gRPC call: 'UNKNOWN: Either `taskDefinition` or `taskDefinitionArgs` must be provided.', failing resolution
    error: an unhandled error occurred: '/opt/homebrew/bin/scala-cli /opt/homebrew/bin/scala-cli run .' exited with non-zero exit code: 1

Another example, the same result:

import besom.*

import besom.api.awsx.ecs.FargateService

@main def main = Pulumi.run {
  val service = FargateService("my-service")
  Stack.exports(url = service.urn)
}

pawelprazak commented 4 months ago

Thank you. We need to investigate why is this happening.

Note to self: the relevant code in the provider: https://github.com/pulumi/pulumi-awsx/blob/5586b5776d22c437044f2e20b23a1768d786fb97/awsx/ecs/ec2Service.ts#L53

keynmol commented 3 months ago

Glad I found this issue!

I have a service definition like this:

//> using scala "3.3.3"
//> using options -Werror -Wunused:all -Wvalue-discard -Wnonunit-statement
//> using dep "org.virtuslab::besom-core:0.2.2"
//> using dep "org.virtuslab::besom-aws:6.23.0-core.0.2"
//> using dep "org.virtuslab::besom-awsx:2.5.0-core.0.2"
//> using dep "com.lihaoyi::upickle::3.2.0"

import besom.*
import besom.api.aws, besom.api.awsx, awsx.ecr, awsx.lb, awsx.ecs
import besom.api.awsx.ecs.inputs.*
import besom.api.awsx.lb.ApplicationLoadBalancerArgs
import besom.api.aws.ecs.inputs.ServiceNetworkConfigurationArgs
import besom.api.aws.cloudwatch.LogGroupArgs
import besom.json.*

@main def main = Pulumi.run {
  val repository =
    ecr.Repository(
      "sentiment-service-repo",
      ecr.RepositoryArgs(forceDelete = true)
    )

  val image = ecr.Image(
    "sentiment-service-image",
    ecr.ImageArgs(
      repositoryUrl = repository.url,
      context = p"../app",
      platform = "linux/amd64"
    )
  )

  val vpc = awsx.ec2.Vpc("sentiment-service-vpc")

  val loadBalancer = lb.ApplicationLoadBalancer(
    "sentiment-service-lb",
    ApplicationLoadBalancerArgs(subnetIds = vpc.publicSubnetIds)
  )

  val cluster = aws.ecs.Cluster("sentiment-service-cluster")

  val logGroup =
    aws.cloudwatch.LogGroup(
      "sentiment-service-log-group",
      LogGroupArgs(retentionInDays = 7, name = "sentiment-service-logs")
    )

  val service =
    image.imageUri.flatMap: image =>
      ecs.FargateService(
        "sentiment-service-fargate",
        ecs.FargateServiceArgs(
          networkConfiguration = ServiceNetworkConfigurationArgs(
            subnets = vpc.publicSubnetIds,
            assignPublicIp = true,
            securityGroups =
              loadBalancer.defaultSecurityGroup.map(_.map(_.id)).map(_.toList)
          ),
          cluster = cluster.arn,
          taskDefinitionArgs = FargateServiceTaskDefinitionArgs(
            containers = Map(
              "sentiment-service" -> TaskDefinitionContainerDefinitionArgs(
                image = image,
                name = "sentiment-service",
                cpu = 128,
                memory = 512,
                essential = true,
                logConfiguration = TaskDefinitionLogConfigurationArgs(
                  logDriver = "awslogs",
                  options = JsObject(
                    "awslogs-group"         -> JsString("log-group"),
                    "awslogs-region"        -> JsString("us-east-1"),
                    "awslogs-stream-prefix" -> JsString("ecs")
                  )
                ),
                /*portMappings = List(
                  TaskDefinitionPortMappingArgs(
                    targetGroup = loadBalancer.defaultTargetGroup
                  )
                )*/
              )
            )
          )
        )
      )

  Stack(logGroup).exports(
    image = image.imageUri,
    service = service,
    vpc = vpc.id,
    cluster = cluster.id,
    url = p"http://${loadBalancer.loadBalancer.dnsName}"
  )
}

And I get this exception IF I do any of the steps below:

Diagnostics:
  pulumi:pulumi:Stack (besom-smithy4s-kiss-dev):
    Error: Exactly one of [container] or [containers] must be provided: Error: Exactly one of [container] or [containers] must be provided
        at normalizeTaskDefinitionContainers (/snapshot/awsx/bin/ecs/containers.js:37:15)
        at new FargateTaskDefinition (/snapshot/awsx/bin/ecs/fargateTaskDefinition.js:34:79)
        at new FargateService (/snapshot/awsx/bin/ecs/fargateService.js:40:30)
        at awsx:ecs:FargateService (/snapshot/awsx/bin/resources.js:25:45)
        at construct (/snapshot/awsx/bin/resources.js:43:12)
        at Provider.construct (/snapshot/awsx/bin/index.js:39:52)
        at Server.<anonymous> (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:315:52)
        at Generator.next (<anonymous>)
        at fulfilled (/snapshot/awsx/node_modules/@pulumi/pulumi/provider/server.js:18:58)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

Here are the things that trigger this error:

Uncommenting the portMapping block
Not using flatMap with imageUri, and instead referencing it directly

Now, I've only been using Pulumi for like 2 days, so I don't know if it's a semantics problem or what, but the exception is very confusing.

Note that there's a pulumi-java issue: https://github.com/pulumi/pulumi-java/issues/1325 and this awsx one https://github.com/pulumi/pulumi-awsx/issues/820 aaaand I'm only now noticing that @pawelprazak is aware :D

keynmol commented 3 months ago

IIRC I've tried a couple of workarounds from both issues but neither of them worked with Besom.

lbialy commented 3 months ago

Hey, I think we have this fixed on current main (with some workarounds for upstream issues). We will cut a release soon. @pawelprazak can you take a look on this and advise if anything can be done to work around this issue on 0.2.2? Paweł has made a big refactoring in serde layer that brought us to be completely up to date with upstream Pulumi serde and this was done to fix this issue mostly. This change is however binary incompatible on core-providers edge so we have to release it as a part of 0.3.x. I think we should do this really fast, considering I found a nasty grpc memoization issue when writing post-Scalar blogpost (also already fixed).

pawelprazak commented 3 months ago

Hi @keynmol, thank you for the reproduction and I'm sorry you've stumbled upon this cursed issue :)

What we know:

awsx provider (technically it is a multi-language component) has a known issue when used from SDKs other than TS/nodejs, unfortunately I was not able to get the to root cause upstream yet
there should be a way to work around those issues (at least the ones I know of) (https://github.com/polkx/besom-bootzooka/blob/main/aws-fargate-bootzooka/Main.scala#L137)
there is a new-ish serialization framework used underneath by Pulumi, and it was ported to besom, awaiting release (https://github.com/VirtusLab/besom/pull/414), it should help, but is not required for the solution

The key workaround was to use explicit val task = awsx.ecs.FargateTaskDefinition(...) instead of the one inlined in the arguments. Please let me know if the issue persists after this change, and I'll try to help to find more workarounds if necessary.

The workarounds might work on the current 0.2.2, but unfortunately there are known issues that required binary compatibility breaking changes and are awaiting for 0.3.0, that we plan to release soon.

P.s. I'm very curious if the flatMap will no longer be necessary with the workaround described above, because if not I would worry me ;)

keynmol commented 3 months ago

Good to hear 👍 I'm doing this for a blogpost as well so can easily wait.

I will try the workaround, and will excitedly wait for the 0.3.0 :)

keynmol commented 3 months ago

Actually it seems I was able to achieve what I wanted (I think, the architecture looks and works the way I wanted) by

removing target group from port mapping, only keeping containerPort

adding loadBalancers block to FargateService:

.. 60           loadBalancers = List(
.. 61             ServiceLoadBalancerArgs(
.. 62               containerName = "sentiment-service",
.. 63               containerPort = 80,
.. 64               targetGroupArn = loadBalancer.defaultTargetGroup.arn
.. 65             )
.. 66           ),

lbialy commented 3 months ago

Btw you should not need to flatMap on imageUri property to use its value as an argument for another Args. Args are built in a way that allows them to consume both raw values and values wrapped in Outputs (opaque type Input[A] = A | Output[A] essentially). We are working on adding a warning when you call a resource constructor in a body of lambda passed to flatMap on Output because, and this is a problem well understood in Scala community thanks to CT discourse, it is impossible for Pulumi to form a static plan of deployment when dynamic, monadic APIs are used (so, basically, applicative vs monadic from mill vs sbt discourse). In dry run computed properties (Outputs on resources) are unknown and behave like Option None - they short circuit and do not run flatMaps. Therefore they cut out subtrees of the deployment plan and that's exactly why we don't really flatMap anywhere beside computing derived properties that we later feed to args of another resource constructor that is called statically, not in a flatMap.

lbialy commented 3 months ago

That's a gotcha coming from the elasticity of Pulumi that Scala's ease of use of flatMap sort of exacerbates. Compile warn should help push people to use this power only when really necessary.

pawelprazak commented 3 months ago

Glad to hear you were able to workaround the issues. Feel free to reach out if any more problems crop up. I'll leave this open until release of 0.3.0, but I'm afraid that the underlying issue appears to be upstream unfortunately.

pawelprazak commented 2 months ago

@lbialy WDYT, can we close this now that the 0.3.x was released? I think we've done what we could on our side and the main problem in this thread is upstream, in the awsx provider.

lbialy commented 1 month ago

It is an open problem that is not fixed. I think we should leave this as a lower priority bug report and check after a new release of pulumi/pulumi-eks (there has been a new one few hours ago - 2.5.2 btw but I don't expect it to fix this problem).

VirtusLab / besom

Can't create FargateService #398