search

aws / aws-sdk-java

The official AWS SDK for Java 1.x (In Maintenance Mode, End-of-Life on 12/31/2025). The AWS SDK for Java 2.x is available here: https://github.com/aws/aws-sdk-java-v2/
https://aws.amazon.com/sdkforjava
Apache License 2.0
4.13k stars 2.83k forks source link

Cannot override resource requirements #2610

Closed khanetor closed 3 years ago

khanetor commented 3 years ago

I cannot override the resource requirements when launching a Batch job. The job keeps using the default resource specification in job definition.

Describe the bug

// First I create the resource requirements
val resourceRequirements = Seq(
  new ResourceRequirement().withType(ResourceType.VCPU).withValue("16"),
  new ResourceRequirement().withType(ResourceType.MEMORY).withValue("32000")
)

// Then I create a container override
val overrides = new ContainerOverrides()
  .withCommand(command.filter(_.nonEmpty): _*)
  .withResourceRequirements(resourceRequirements: _*)

// Then I create a job submit request
val request = new SubmitJobRequest()
  .withJobName(...)
  .withJobQueue(queue)
  .withJobDefinition(jobDefinition)
  .withContainerOverrides(overrides)

Expected Behavior

A job should be created with 16 vcpu and 32000MB memory.

Current Behavior

A job is created with 2 vcpu and 8192MB memory, the default in job definition.

There is no error. A job is submitted successfully, but with the resource requirements ignored.

Steps to Reproduce

Context

I have forecasting jobs that need very high concurrency, thus I need many vcpu. Fargate cannot satisfy this requirement because 4vcpu is the maximum.

Your Environment

debora-ito commented 3 years ago

Hi @nlhkh thank you for reaching out.

Can you provide the verbose wire logs? I'm interested to see what is being sent in the wire for that SubmitJob request. https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#sdk-net-logging-verbose

Please remove any sensitive data the wire logs may contain (like access keys).

github-actions[bot] commented 3 years ago

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

khanetor commented 3 years ago

Hi @nlhkh thank you for reaching out.

Can you provide the verbose wire logs? I'm interested to see what is being sent in the wire for that SubmitJob request. https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#sdk-net-logging-verbose

Please remove any sensitive data the wire logs may contain (like access keys).

I have setup logging as such, but I don't see any output printing out. Keep in mind that the jobs were scheduled successfully, so there should not be any error message. The problem is that setting vcpu and memory has no effect.

val vcpuRequirement = new ResourceRequirement()
      .withType(ResourceType.VCPU)
      .withValue(vcpu.toString)

val memoryRequirement = new ResourceRequirement()
      .withType(ResourceType.MEMORY)
      .withValue((memory * 0.95 * 1024).toInt.toString)

val overrides = new ContainerOverrides()
      .withCommand(command.filter(_.nonEmpty): _*)
      .withResourceRequirements(vcpuRequirement, memoryRequirement)

The job will be launched with the default requirements in job definition.

debora-ito commented 3 years ago

With the wirelogs I was hoping to see if the settings were being sent in the request to the service, was not looking for error messages.

khanetor commented 3 years ago

With the wirelogs I was hoping to see if the settings were being sent in the request to the service, was not looking for error messages.

Hi @debora-ito, sorry for the late reply.

Here is the log when I use the deprecated methods to set VCPU and MEMORY:

val overrides = ContainerOverrides.builder()
      .command(command.filter(_.nonEmpty): _*)
      .vcpus(vcpu)
      .memory(memoryInGB * 1024 * 9 / 10)
      .build()
...
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Binding validation: 88ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Static validation: 0ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Instance member validation: 0ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Provider verification: 0ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Delayed Binding initialization: 1ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Static member injection: 0ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Instance injection: 0ms
2021-09-13 14:52:28 DEBUG c.g.inject.internal.util.Stopwatch  Preloading singletons: 0ms
2021-09-13 14:52:28 INFO  akka.event.slf4j.Slf4jLogger  Slf4jLogger started
2021-09-13 14:52:28 DEBUG akka.event.EventStream EventStream(akka://application) logger log1-Slf4jLogger started
2021-09-13 14:52:28 DEBUG akka.event.EventStream EventStream(akka://application) Default Loggers started
2021-09-13 14:52:28 DEBUG a.s.Serialization(akka://application) akka.serialization.Serialization(akka://application) Replacing JavaSerializer with DisabledJavaSerializer, due to `akka.actor.allow-java-serialization = off`.
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:52:28 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:52:28 INFO  play.api.Play  Application started (Dev) (no global state)
2021-09-13 14:52:28 DEBUG i.n.u.i.l.InternalLoggerFactory  Using SLF4J as the default logging framework
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  -Dio.netty.noUnsafe: false
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  Java version: 11
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  sun.misc.Unsafe.theUnsafe: available
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  sun.misc.Unsafe.copyMemory: available
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  java.nio.Buffer.address: available
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  java.nio.Bits.unaligned: available, true
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @837c323
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent0  java.nio.DirectByteBuffer.<init>(long, int): unavailabl
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  sun.misc.Unsafe: available
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  maxDirectMemory: 1073741824 bytes (maybe)
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.tmpdir: /var/folders/rz/j1sp0c394pn5sv5pttp2pkt40000gn/T (java.io.tmpdir)
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  Platform: MacOS
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.maxDirectMemory: -1 bytes
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-09-13 14:52:28 DEBUG io.netty.util.internal.CleanerJava9  java.nio.ByteBuffer.cleaner(): available
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.noPreferDirect: false
2021-09-13 14:52:28 DEBUG i.n.c.MultithreadEventLoopGroup  -Dio.netty.eventLoopThreads: 16
2021-09-13 14:52:28 DEBUG i.n.u.i.InternalThreadLocalMap  -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-09-13 14:52:28 DEBUG i.n.u.i.InternalThreadLocalMap  -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-09-13 14:52:28 DEBUG io.netty.channel.nio.NioEventLoop  -Dio.netty.noKeySetOptimization: false
2021-09-13 14:52:28 DEBUG io.netty.channel.nio.NioEventLoop  -Dio.netty.selectorAutoRebuildThreshold: 512
2021-09-13 14:52:28 DEBUG i.n.util.internal.PlatformDependent  org.jctools-core.MpscChunkedArrayQueue: available
2021-09-13 14:52:28 DEBUG io.netty.handler.ssl.OpenSsl  netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
2021-09-13 14:52:28 DEBUG controllers.AssetsConfiguration  Using the following cache configuration for assets:
     enableCaching = false
     enableCacheControl = false
     defaultCacheControl = public, max-age=3600
     aggressiveCacheControl = public, max-age=31536000, immutable
     configuredCacheControl:
2021-09-13 14:52:29 DEBUG software.amazon.awssdk.request  Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=batch.eu-west-1.amazonaws.com, encodedPath=/v1/submitjob, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])
2021-09-13 14:52:29 DEBUG io.netty.handler.ssl.JdkSslContext  Default protocols (JDK): [TLSv1.3, TLSv1.2] 
2021-09-13 14:52:29 DEBUG io.netty.handler.ssl.JdkSslContext  Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]
2021-09-13 14:52:29 DEBUG io.netty.channel.DefaultChannelId  -Dio.netty.processId: 16451 (auto-detected)
2021-09-13 14:52:29 DEBUG io.netty.util.NetUtil  -Djava.net.preferIPv4Stack: false
2021-09-13 14:52:29 DEBUG io.netty.util.NetUtil  -Djava.net.preferIPv6Addresses: false
2021-09-13 14:52:29 DEBUG io.netty.util.NetUtilInitializations  Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0)
2021-09-13 14:52:29 DEBUG io.netty.util.NetUtil  Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128
2021-09-13 14:52:29 DEBUG io.netty.channel.DefaultChannelId  -Dio.netty.machineId: a0:78:17:ff:fe:80:7c:99 (auto-detected)
2021-09-13 14:52:29 DEBUG io.netty.util.ResourceLeakDetector  -Dio.netty.leakDetection.level: simple
2021-09-13 14:52:29 DEBUG io.netty.util.ResourceLeakDetector  -Dio.netty.leakDetection.targetRecords: 4
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.numHeapArenas: 10
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.numDirectArenas: 10
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.pageSize: 8192
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxOrder: 11
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.chunkSize: 16777216
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.smallCacheSize: 256
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.normalCacheSize: 64
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.cacheTrimInterval: 8192
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.useCacheForAllThreads: true
2021-09-13 14:52:29 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2021-09-13 14:52:29 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.allocator.type: pooled
2021-09-13 14:52:29 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.threadLocalDirectBufferSize: 0
2021-09-13 14:52:29 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.maxThreadLocalCharBufferSize: 16384
2021-09-13 14:52:29 DEBUG io.netty.buffer.AbstractByteBuf  -Dio.netty.buffer.checkAccessible: true
2021-09-13 14:52:29 DEBUG io.netty.buffer.AbstractByteBuf  -Dio.netty.buffer.checkBounds: true
2021-09-13 14:52:29 DEBUG i.n.util.ResourceLeakDetectorFactory  Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@70c469fc
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd] REGISTERED
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd] CONNECT: batch.eu-west-1.amazonaws.com/34.243.50.129:443
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf] REGISTERED
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf] CONNECT: batch.eu-west-1.amazonaws.com/34.243.50.129:443
2021-09-13 14:52:29 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.maxCapacityPerThread: 4096
2021-09-13 14:52:29 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.maxSharedCapacityFactor: 2
2021-09-13 14:52:29 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.linkCapacity: 16
2021-09-13 14:52:29 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.ratio: 8
2021-09-13 14:52:29 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.delayedQueue.ratio: 8
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd, L:/192.168.100.22:53986 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] ACTIVE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] ACTIVE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] WRITE: software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$StreamedRequest(DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /v1/submitjob HTTP/1.1
Host: batch.eu-west-1.amazonaws.com
amz-sdk-invocation-id: 514753db-39fb-c528-b9cb-d5ee3e755f3b
amz-sdk-request: attempt=1; max=4
Authorization: AWS4-HMAC-SHA256 Credential=AKIASMV6TJBB3WPVSTDK/20210913/eu-west-1/batch/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=f6a73984ad38b72664fb669171636ac19b607f81ab2115f75489e090962b478d
Content-Length: 430
Content-Type: application/x-amz-json-1.1
User-Agent: aws-sdk-java/2.17.37 Mac_OS_X/11.5.2 OpenJDK_64-Bit_Server_VM/11.0.12+7-LTS Java/11.0.12 scala/2.13.6 vendor/Azul_Systems__Inc. io/async http/NettyNio cfg/retry-mode/legacy
X-Amz-Date: 20210913T115229Z)
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] FLUSH
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] FLUSH
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] WRITE: DefaultHttpContent(data: UnpooledHeapByteBuf(ridx: 0, widx: 430, cap: 430/430), decoderResult: success), 430B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 7b 22 6a 6f 62 4e 61 6d 65 22 3a 22 66 65 74 63 |{"jobName":"fetc|
|00000010| 68 2d 70 69 63 6e 69 63 2d 64 61 79 2d 32 30 32 |h-picnic-day-202|
|00000020| 31 30 39 31 33 2d 30 32 35 32 32 39 22 2c 22 6a |10913-025229","j|
|00000030| 6f 62 51 75 65 75 65 22 3a 22 73 69 6e 67 75 6c |obQueue":"singul|
|00000040| 61 72 69 74 79 2d 66 65 74 63 68 2d 71 75 65 75 |arity-fetch-queu|
|00000050| 65 2d 64 65 76 22 2c 22 6a 6f 62 44 65 66 69 6e |e-dev","jobDefin|
|00000060| 69 74 69 6f 6e 22 3a 22 70 72 6f 70 68 65 74 2d |ition":"prophet-|
|00000070| 6a 6f 62 2d 64 65 66 69 6e 69 74 69 6f 6e 2d 64 |job-definition-d|
|00000080| 65 76 22 2c 22 63 6f 6e 74 61 69 6e 65 72 4f 76 |ev","containerOv|
|00000090| 65 72 72 69 64 65 73 22 3a 7b 22 76 63 70 75 73 |errides":{"vcpus|
|000000a0| 22 3a 32 2c 22 6d 65 6d 6f 72 79 22 3a 31 38 34 |":2,"memory":184|
|000000b0| 33 2c 22 63 6f 6d 6d 61 6e 64 22 3a 5b 22 66 65 |3,"command":["fe|
|000000c0| 74 63 68 5f 64 61 74 61 2e 70 79 22 2c 22 2d 2d |tch_data.py","--|
|000000d0| 63 75 73 74 6f 6d 65 72 3d 70 69 63 6e 69 63 22 |customer=picnic"|
|000000e0| 2c 22 2d 2d 74 6f 6b 65 6e 3d 61 36 63 6e 33 77 |,"--token=a6cn3w|
|000000f0| 70 30 36 6c 35 35 64 32 41 43 54 49 37 33 67 32 |p06l55d2ACTI73g2|
|00000100| 4d 54 52 69 72 61 4b 49 66 31 22 2c 22 2d 2d 66 |MTRiraKIf1","--f|
|00000110| 72 65 71 3d 64 61 79 22 2c 22 2d 2d 63 75 72 72 |req=day","--curr|
|00000120| 65 6e 63 79 3d 47 42 50 22 2c 22 2d 2d 67 72 6f |ency=GBP","--gro|
|00000130| 75 70 69 6e 67 3d 73 74 6f 72 65 2c 63 61 74 65 |uping=store,cate|
|00000140| 67 6f 72 79 2c 73 75 62 63 61 74 65 67 6f 72 79 |gory,subcategory|
|00000150| 2c 70 72 6f 64 75 63 74 5f 63 6f 64 65 22 2c 22 |,product_code","|
|00000160| 2d 2d 6d 65 74 72 69 63 73 3d 73 61 6c 65 73 5f |--metrics=sales_|
|00000170| 70 63 73 22 2c 22 2d 2d 69 73 6f 2d 77 65 65 6b |pcs","--iso-week|
|00000180| 22 5d 7d 2c 22 74 69 6d 65 6f 75 74 22 3a 7b 22 |"]},"timeout":{"|
|00000190| 61 74 74 65 6d 70 74 44 75 72 61 74 69 6f 6e 53 |attemptDurationS|
|000001a0| 65 63 6f 6e 64 73 22 3a 31 38 30 30 7d 7d       |econds":1800}}  |
+--------+-------------------------------------------------+----------------+
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] FLUSH
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd, L:/192.168.100.22:53986 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 DEBUG io.netty.handler.ssl.SslHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-09-13 14:52:29 DEBUG io.netty.handler.ssl.SslHandler  [id: 0x42acd4cd, L:/192.168.100.22:53986 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd, L:/192.168.100.22:53986 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x42acd4cd, L:/192.168.100.22:53986 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] WRITE, EmptyLastHttpContent, 0B
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] FLUSH
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Mon, 13 Sep 2021 11:52:29 GMT
Content-Type: application/json
Connection: keep-alive
x-amzn-RequestId: 761f7af4-9d74-4748-ab69-d8fc63395c1c
Access-Control-Allow-Origin: *
x-amz-apigw-id: FmY7qFxAjoEFfGA=
Access-Control-Expose-Headers: X-amzn-errortype,X-amzn-requestid,X-amzn-errormessage,X-amzn-trace-id,X-amz-apigw-id,dat
X-Amzn-Trace-Id: Root=1-613f3b7d-4f528fa05d00daad633a7e0a
content-length: 182
2021-09-13 14:52:29 DEBUG software.amazon.awssdk.request  Received successful response: 200
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ: DefaultLastHttpContent(data: UnpooledSlicedByteBuf(ridx: 0, widx: 182, cap: 182/182, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf(ridx: 618, widx: 618, cap: 647)), decoderResult: success), 182B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 7b 22 6a 6f 62 41 72 6e 22 3a 22 61 72 6e 3a 61 |{"jobArn":"arn:a|
|00000010| 77 73 3a 62 61 74 63 68 3a 65 75 2d 77 65 73 74 |ws:batch:eu-west|
|00000020| 2d 31 3a 31 36 34 36 38 32 32 32 31 36 33 35 3a |-1:164682221635:|
|00000030| 6a 6f 62 2f 63 30 30 39 31 36 32 64 2d 32 38 31 |job/c009162d-281|
|00000040| 37 2d 34 39 33 64 2d 39 30 62 32 2d 32 35 38 62 |7-493d-90b2-258b|
|00000050| 38 32 33 66 34 66 39 64 22 2c 22 6a 6f 62 4e 61 |823f4f9d","jobNa|
|00000060| 6d 65 22 3a 22 66 65 74 63 68 2d 70 69 63 6e 69 |me":"fetch-picni|
|00000070| 63 2d 64 61 79 2d 32 30 32 31 30 39 31 33 2d 30 |c-day-20210913-0|
|00000080| 32 35 32 32 39 22 2c 22 6a 6f 62 49 64 22 3a 22 |25229","jobId":"|
|00000090| 63 30 30 39 31 36 32 64 2d 32 38 31 37 2d 34 39 |c009162d-2817-49|
|000000a0| 33 64 2d 39 30 62 32 2d 32 35 38 62 38 32 33 66 |3d-90b2-258b823f|
|000000b0| 34 66 39 64 22 7d                               |4f9d"}          |
+--------+-------------------------------------------------+----------------+
2021-09-13 14:52:29 DEBUG i.n.h.codec.http.LastHttpContent  Received LastHttpContent [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443]
2021-09-13 14:52:29 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 - R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] READ COMPLETE
2021-09-13 14:52:29 INFO  application  Served POST /submitJob/fetch with status 200 in 756ms
2021-09-13 14:52:34 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 ! R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] INACTIVE
2021-09-13 14:52:34 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x16b845cf, L:/192.168.100.22:53987 ! R:batch.eu-west-1.amazonaws.com/34.243.50.129:443] UNREGISTERED

Here is the log when I use the new suggested ResourceRequirement:

val cpuRequirement = ResourceRequirement.builder().`type`(ResourceType.VCPU).value(vcpu.toString).build()
val memoryRequirement = ResourceRequirement.builder().`type`(ResourceType.MEMORY).value((memoryInGB * 1024 * 9 / 10).toString).build()

val overrides = ContainerOverrides.builder()
      .command(command.filter(_.nonEmpty): _*)
      .resourceRequirements(cpuRequirement, memoryRequirement)
      .build()
...
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Binding validation: 118ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Static validation: 0ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Instance member validation: 0ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Provider verification: 1ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Delayed Binding initialization: 0ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Static member injection: 0ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Instance injection: 0ms
2021-09-13 14:57:09 DEBUG c.g.inject.internal.util.Stopwatch  Preloading singletons: 1ms
2021-09-13 14:57:09 INFO  akka.event.slf4j.Slf4jLogger  Slf4jLogger started
2021-09-13 14:57:09 DEBUG akka.event.EventStream EventStream(akka://application) logger log1-Slf4jLogger started
2021-09-13 14:57:09 DEBUG akka.event.EventStream EventStream(akka://application) Default Loggers started
2021-09-13 14:57:09 DEBUG a.s.Serialization(akka://application) akka.serialization.Serialization(akka://application) Replacing JavaSerializer with DisabledJavaSerializer, due to `akka.actor.allow-java-serialization = off`.
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaTypedJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [akka.serialization.jackson.AkkaStreamJacksonModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]
2021-09-13 14:57:09 DEBUG a.s.j.JacksonObjectMapperProvider$ JacksonObjectMapperProvider$ Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]
2021-09-13 14:57:09 INFO  play.api.Play  Application started (Dev) (no global state)
2021-09-13 14:57:09 DEBUG i.n.u.i.l.InternalLoggerFactory  Using SLF4J as the default logging framework
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  -Dio.netty.noUnsafe: false
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  Java version: 11
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  sun.misc.Unsafe.theUnsafe: available
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  sun.misc.Unsafe.copyMemory: available
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  java.nio.Buffer.address: available
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  java.nio.Bits.unaligned: available, true
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @149410d
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent0  java.nio.DirectByteBuffer.<init>(long, int): unavailabl
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  sun.misc.Unsafe: available
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  maxDirectMemory: 1073741824 bytes (maybe)
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.tmpdir: /var/folders/rz/j1sp0c394pn5sv5pttp2pkt40000gn/T (java.io.tmpdir)
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  Platform: MacOS
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.maxDirectMemory: -1 bytes
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-09-13 14:57:09 DEBUG io.netty.util.internal.CleanerJava9  java.nio.ByteBuffer.cleaner(): available
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  -Dio.netty.noPreferDirect: false
2021-09-13 14:57:09 DEBUG i.n.c.MultithreadEventLoopGroup  -Dio.netty.eventLoopThreads: 16
2021-09-13 14:57:09 DEBUG i.n.u.i.InternalThreadLocalMap  -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-09-13 14:57:09 DEBUG i.n.u.i.InternalThreadLocalMap  -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-09-13 14:57:09 DEBUG io.netty.channel.nio.NioEventLoop  -Dio.netty.noKeySetOptimization: false
2021-09-13 14:57:09 DEBUG io.netty.channel.nio.NioEventLoop  -Dio.netty.selectorAutoRebuildThreshold: 512
2021-09-13 14:57:09 DEBUG i.n.util.internal.PlatformDependent  org.jctools-core.MpscChunkedArrayQueue: available
2021-09-13 14:57:09 DEBUG io.netty.handler.ssl.OpenSsl  netty-tcnative not in the classpath; OpenSslEngine will be unavailable.
2021-09-13 14:57:09 DEBUG controllers.AssetsConfiguration  Using the following cache configuration for assets:
     enableCaching = false
     enableCacheControl = false
     defaultCacheControl = public, max-age=3600
     aggressiveCacheControl = public, max-age=31536000, immutable
     configuredCacheControl:
2021-09-13 14:57:09 DEBUG software.amazon.awssdk.request  Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=batch.eu-west-1.amazonaws.com, encodedPath=/v1/submitjob, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])
2021-09-13 14:57:09 DEBUG io.netty.handler.ssl.JdkSslContext  Default protocols (JDK): [TLSv1.3, TLSv1.2] 
2021-09-13 14:57:09 DEBUG io.netty.handler.ssl.JdkSslContext  Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]
2021-09-13 14:57:09 DEBUG io.netty.channel.DefaultChannelId  -Dio.netty.processId: 16583 (auto-detected)
2021-09-13 14:57:09 DEBUG io.netty.util.NetUtil  -Djava.net.preferIPv4Stack: false
2021-09-13 14:57:09 DEBUG io.netty.util.NetUtil  -Djava.net.preferIPv6Addresses: false
2021-09-13 14:57:09 DEBUG io.netty.util.NetUtilInitializations  Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1%lo0)
2021-09-13 14:57:09 DEBUG io.netty.util.NetUtil  Failed to get SOMAXCONN from sysctl and file /proc/sys/net/core/somaxconn. Default: 128
2021-09-13 14:57:09 DEBUG io.netty.channel.DefaultChannelId  -Dio.netty.machineId: a0:78:17:ff:fe:80:7c:99 (auto-detected)
2021-09-13 14:57:09 DEBUG io.netty.util.ResourceLeakDetector  -Dio.netty.leakDetection.level: simple
2021-09-13 14:57:09 DEBUG io.netty.util.ResourceLeakDetector  -Dio.netty.leakDetection.targetRecords: 4
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.numHeapArenas: 10
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.numDirectArenas: 10
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.pageSize: 8192
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxOrder: 11
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.chunkSize: 16777216
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.smallCacheSize: 256
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.normalCacheSize: 64
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.cacheTrimInterval: 8192
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.useCacheForAllThreads: true
2021-09-13 14:57:09 DEBUG i.n.buffer.PooledByteBufAllocator  -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2021-09-13 14:57:09 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.allocator.type: pooled
2021-09-13 14:57:09 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.threadLocalDirectBufferSize: 0
2021-09-13 14:57:09 DEBUG io.netty.buffer.ByteBufUtil  -Dio.netty.maxThreadLocalCharBufferSize: 16384
2021-09-13 14:57:09 DEBUG io.netty.buffer.AbstractByteBuf  -Dio.netty.buffer.checkAccessible: true
2021-09-13 14:57:09 DEBUG io.netty.buffer.AbstractByteBuf  -Dio.netty.buffer.checkBounds: true
2021-09-13 14:57:09 DEBUG i.n.util.ResourceLeakDetectorFactory  Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@26b78ad5
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6] REGISTERED
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6] CONNECT: batch.eu-west-1.amazonaws.com/34.249.190.228:443
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7] REGISTERED
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7] CONNECT: batch.eu-west-1.amazonaws.com/34.249.190.228:443
2021-09-13 14:57:10 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.maxCapacityPerThread: 4096
2021-09-13 14:57:10 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.maxSharedCapacityFactor: 2
2021-09-13 14:57:10 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.linkCapacity: 16
2021-09-13 14:57:10 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.ratio: 8
2021-09-13 14:57:10 DEBUG io.netty.util.Recycler  -Dio.netty.recycler.delayedQueue.ratio: 8
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6, L:/192.168.100.22:54028 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] ACTIVE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] ACTIVE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] WRITE: software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$StreamedRequest(DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /v1/submitjob HTTP/1.1
Host: batch.eu-west-1.amazonaws.com
amz-sdk-invocation-id: a89b558a-b59c-c0d8-a6e8-67fd61f61ded
amz-sdk-request: attempt=1; max=4
Authorization: AWS4-HMAC-SHA256 Credential=AKIASMV6TJBB3WPVSTDK/20210913/eu-west-1/batch/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=f5dea1dc436d9faa7a9c66ac3ae178a5f9ed6fed0ae85f5bb251a9a35fe7fe0a
Content-Length: 492
Content-Type: application/x-amz-json-1.1
User-Agent: aws-sdk-java/2.17.37 Mac_OS_X/11.5.2 OpenJDK_64-Bit_Server_VM/11.0.12+7-LTS Java/11.0.12 scala/2.13.6 vendor/Azul_Systems__Inc. io/async http/NettyNio cfg/retry-mode/legacy
X-Amz-Date: 20210913T115709Z)
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] FLUSH
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] FLUSH
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] WRITE: DefaultHttpContent(data: UnpooledHeapByteBuf(ridx: 0, widx: 492, cap: 492/492), decoderResult: success), 492B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 7b 22 6a 6f 62 4e 61 6d 65 22 3a 22 66 65 74 63 |{"jobName":"fetc|
|00000010| 68 2d 70 69 63 6e 69 63 2d 64 61 79 2d 32 30 32 |h-picnic-day-202|
|00000020| 31 30 39 31 33 2d 30 32 35 37 30 39 22 2c 22 6a |10913-025709","j|
|00000030| 6f 62 51 75 65 75 65 22 3a 22 73 69 6e 67 75 6c |obQueue":"singul|
|00000040| 61 72 69 74 79 2d 66 65 74 63 68 2d 71 75 65 75 |arity-fetch-queu|
|00000050| 65 2d 64 65 76 22 2c 22 6a 6f 62 44 65 66 69 6e |e-dev","jobDefin|
|00000060| 69 74 69 6f 6e 22 3a 22 70 72 6f 70 68 65 74 2d |ition":"prophet-|
|00000070| 6a 6f 62 2d 64 65 66 69 6e 69 74 69 6f 6e 2d 64 |job-definition-d|
|00000080| 65 76 22 2c 22 63 6f 6e 74 61 69 6e 65 72 4f 76 |ev","containerOv|
|00000090| 65 72 72 69 64 65 73 22 3a 7b 22 63 6f 6d 6d 61 |errides":{"comma|
|000000a0| 6e 64 22 3a 5b 22 66 65 74 63 68 5f 64 61 74 61 |nd":["fetch_data|
|000000b0| 2e 70 79 22 2c 22 2d 2d 63 75 73 74 6f 6d 65 72 |.py","--customer|
|000000c0| 3d 70 69 63 6e 69 63 22 2c 22 2d 2d 74 6f 6b 65 |=picnic","--toke|
|000000d0| 6e 3d 61 36 63 6e 33 77 70 30 36 6c 35 35 64 32 |n=a6cn3wp06l55d2|
|000000e0| 41 43 54 49 37 33 67 32 4d 54 52 69 72 61 4b 49 |ACTI73g2MTRiraKI|
|000000f0| 66 31 22 2c 22 2d 2d 66 72 65 71 3d 64 61 79 22 |f1","--freq=day"|
|00000100| 2c 22 2d 2d 63 75 72 72 65 6e 63 79 3d 47 42 50 |,"--currency=GBP|
|00000110| 22 2c 22 2d 2d 67 72 6f 75 70 69 6e 67 3d 73 74 |","--grouping=st|
|00000120| 6f 72 65 2c 63 61 74 65 67 6f 72 79 2c 73 75 62 |ore,category,sub|
|00000130| 63 61 74 65 67 6f 72 79 2c 70 72 6f 64 75 63 74 |category,product|
|00000140| 5f 63 6f 64 65 22 2c 22 2d 2d 6d 65 74 72 69 63 |_code","--metric|
|00000150| 73 3d 73 61 6c 65 73 5f 70 63 73 22 2c 22 2d 2d |s=sales_pcs","--|
|00000160| 69 73 6f 2d 77 65 65 6b 22 5d 2c 22 72 65 73 6f |iso-week"],"reso|
|00000170| 75 72 63 65 52 65 71 75 69 72 65 6d 65 6e 74 73 |urceRequirements|
|00000180| 22 3a 5b 7b 22 76 61 6c 75 65 22 3a 22 32 22 2c |":[{"value":"2",|
|00000190| 22 74 79 70 65 22 3a 22 56 43 50 55 22 7d 2c 7b |"type":"VCPU"},{|
|000001a0| 22 76 61 6c 75 65 22 3a 22 31 38 34 33 22 2c 22 |"value":"1843","|
|000001b0| 74 79 70 65 22 3a 22 4d 45 4d 4f 52 59 22 7d 5d |type":"MEMORY"}]|
|000001c0| 7d 2c 22 74 69 6d 65 6f 75 74 22 3a 7b 22 61 74 |},"timeout":{"at|
|000001d0| 74 65 6d 70 74 44 75 72 61 74 69 6f 6e 53 65 63 |temptDurationSec|
|000001e0| 6f 6e 64 73 22 3a 31 38 30 30 7d 7d             |onds":1800}}    |
+--------+-------------------------------------------------+----------------+
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] FLUSH
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6, L:/192.168.100.22:54028 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 DEBUG io.netty.handler.ssl.SslHandler  [id: 0x51ba24f6, L:/192.168.100.22:54028 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-09-13 14:57:10 DEBUG io.netty.handler.ssl.SslHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6, L:/192.168.100.22:54028 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x51ba24f6, L:/192.168.100.22:54028 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] WRITE, EmptyLastHttpContent, 0B
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] FLUSH
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ: DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Mon, 13 Sep 2021 11:57:10 GMT
Content-Type: application/json
Connection: keep-alive
x-amzn-RequestId: affeaee2-404a-411b-9b3a-9d9f5f6e243c
Access-Control-Allow-Origin: *
x-amz-apigw-id: FmZnhEitDoEF0JQ=
Access-Control-Expose-Headers: X-amzn-errortype,X-amzn-requestid,X-amzn-errormessage,X-amzn-trace-id,X-amz-apigw-id,dat
X-Amzn-Trace-Id: Root=1-613f3c96-228930c51c467d1850fd8e25
content-length: 182
2021-09-13 14:57:10 DEBUG software.amazon.awssdk.request  Received successful response: 200
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ: DefaultLastHttpContent(data: UnpooledSlicedByteBuf(ridx: 0, widx: 182, cap: 182/182, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf(ridx: 618, widx: 618, cap: 647)), decoderResult: success), 182B
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 7b 22 6a 6f 62 41 72 6e 22 3a 22 61 72 6e 3a 61 |{"jobArn":"arn:a|
|00000010| 77 73 3a 62 61 74 63 68 3a 65 75 2d 77 65 73 74 |ws:batch:eu-west|
|00000020| 2d 31 3a 31 36 34 36 38 32 32 32 31 36 33 35 3a |-1:164682221635:|
|00000030| 6a 6f 62 2f 36 38 61 66 65 38 30 31 2d 61 62 66 |job/68afe801-abf|
|00000040| 39 2d 34 66 66 37 2d 38 62 31 34 2d 62 30 34 37 |9-4ff7-8b14-b047|
|00000050| 31 62 32 65 65 34 66 65 22 2c 22 6a 6f 62 4e 61 |1b2ee4fe","jobNa|
|00000060| 6d 65 22 3a 22 66 65 74 63 68 2d 70 69 63 6e 69 |me":"fetch-picni|
|00000070| 63 2d 64 61 79 2d 32 30 32 31 30 39 31 33 2d 30 |c-day-20210913-0|
|00000080| 32 35 37 30 39 22 2c 22 6a 6f 62 49 64 22 3a 22 |25709","jobId":"|
|00000090| 36 38 61 66 65 38 30 31 2d 61 62 66 39 2d 34 66 |68afe801-abf9-4f|
|000000a0| 66 37 2d 38 62 31 34 2d 62 30 34 37 31 62 32 65 |f7-8b14-b0471b2e|
|000000b0| 65 34 66 65 22 7d                               |e4fe"}          |
+--------+-------------------------------------------------+----------------+
2021-09-13 14:57:10 DEBUG i.n.h.codec.http.LastHttpContent  Received LastHttpContent [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443]
2021-09-13 14:57:10 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 - R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] READ COMPLETE
2021-09-13 14:57:10 INFO  application  Served POST /submitJob/fetch with status 200 in 721ms
2021-09-13 14:57:15 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 ! R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] INACTIVE
2021-09-13 14:57:15 DEBUG i.n.handler.logging.LoggingHandler  [id: 0x3f0235c7, L:/192.168.100.22:54029 ! R:batch.eu-west-1.amazonaws.com/34.249.190.228:443] UNREGISTERED
khanetor commented 3 years ago

Apparently the VCPU and MEMORY are passed as resource requirement, similar to Fargate, in the second case.

Jno21 commented 2 years ago

Hi, did you resolve your issue ?

khanetor commented 2 years ago

Hi, did you resolve your issue ?

Nope. Using resource requirements still does not pass the VCPU and MEMORY override to the task/job.I am switching to creating multiple job definition with different VCPU and MEMORY, so this is no longer relevant to me, but it is still a bug in your side.