search

brooklyncentral / clocker

Apache Brooklyn cloud native infrastructure blueprints
Apache License 2.0
426 stars 66 forks source link

Exceptions collecting container.stats.memory.percent and container.stats.cpu.percent #379

Open aledsage opened 7 years ago

aledsage commented 7 years ago

In the logs, I see the two exceptions shown below:

2016-10-25 21:50:16,015 WARN  117 o.a.b.c.f.AttributePollHandler [r-NmGPUwnZ-30374] unable to compute VanillaSoftwareProcessImpl{id=cb3pwep5pc}->Sensor: container.stats.memory.percent (java.lang.Double); on val=org.apache.brooklyn.feed.ssh.SshPollValue@1df4a1b0
org.apache.brooklyn.util.javalang.coerce.ClassCoercionException: Cannot coerce "70   0.00%" to java.lang.Double (70   0.00%): adapting failed
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.stringToPrimitive(PrimitiveStringTypeCoercions.java:202)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.tryCoerce(PrimitiveStringTypeCoercions.java:47)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerceInternal(TypeCoercerExtensible.java:131)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerce(TypeCoercerExtensible.java:107)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:97)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:93)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.flags.TypeCoercions.coerce(TypeCoercions.java:79)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:130)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:127)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:211)[67:com.google.guava:16.0.1]
        at org.apache.brooklyn.core.feed.AttributePollHandler.transformValueOnSuccess(AttributePollHandler.java:114)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.AttributePollHandler.onSuccess(AttributePollHandler.java:101)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.DelegatingPollHandler.onSuccess(DelegatingPollHandler.java:51)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:78)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:156)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:149)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:436)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]

2016-11-10 19:06:31,400 WARN  117 o.a.b.c.f.AttributePollHandler [-NmGPUwnZ-229662] unable to compute VanillaSoftwareProcessImpl{id=q9wq6umn7m}->Sensor: container.stats.cpu.percent (java.lang.Double); on val=org.apache.brooklyn.feed.ssh.SshPollValue@66242ba5
org.apache.brooklyn.util.javalang.coerce.ClassCoercionException: Cannot coerce "205b7205497b8f7887f5" to java.lang.Double (205b7205497b8f7887f5): adapting failed
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.stringToPrimitive(PrimitiveStringTypeCoercions.java:202)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.PrimitiveStringTypeCoercions.tryCoerce(PrimitiveStringTypeCoercions.java:47)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerceInternal(TypeCoercerExtensible.java:131)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.tryCoerce(TypeCoercerExtensible.java:107)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:97)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.javalang.coerce.TypeCoercerExtensible.coerce(TypeCoercerExtensible.java:93)[140:org.apache.brooklyn.utils-common:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.flags.TypeCoercions.coerce(TypeCoercions.java:79)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:130)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.sensor.ssh.SshCommandSensor$3.apply(SshCommandSensor.java:127)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:211)[67:com.google.guava:16.0.1]
        at org.apache.brooklyn.core.feed.AttributePollHandler.transformValueOnSuccess(AttributePollHandler.java:114)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.AttributePollHandler.onSuccess(AttributePollHandler.java:101)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.DelegatingPollHandler.onSuccess(DelegatingPollHandler.java:51)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$PollJob$1.run(Poller.java:78)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:156)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.core.feed.Poller$1$1.call(Poller.java:149)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:436)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)[117:org.apache.brooklyn.core:0.10.0.20161011_2234]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_101]

This tells us that the bash below (from https://github.com/brooklyncentral/clocker/blob/master/common/catalog/docker/docker.bom#L543-L562) is incorrect - i.e. the cut commands are picking up the wrong characters. It looks really brittle. I'd have though we should split it on the separator, rather than looking for a specific character range.

        - type: org.apache.brooklyn.core.sensor.ssh.SshCommandSensor
          brooklyn.config:
            name: container.stats.cpu.percent
            period: 1m
            targetType: double
            command: |
              test -f pid.txt &&
                ( docker stats --no-stream $(cat pid.txt) |
                  tail -n +2 |
                  cut -c21-40 )
        - type: org.apache.brooklyn.core.sensor.ssh.SshCommandSensor
          brooklyn.config:
            name: container.stats.memory.percent
            period: 1m
            targetType: double
            command: |
              test -f pid.txt &&
                ( docker stats --no-stream $(cat pid.txt) |
                  tail -n +2 |
                  cut -c63-82 )
aledsage commented 6 years ago

My output from docker stats --no-stream $(cat pid.txt) is:

CONTAINER                                                          CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
bf29e117ac83803eb92dbec77d5a5e29d3453cc5c04a6a5ad98537160b77f90d   1.39%               685.9MiB / 3.518GiB   19.04%              0B / 0B             909kB / 760MB       0

We should be able to instead usedocker stats --no-stream --format "table {{.CPUPerc}}" $(cat pid.txt), which gives:

CPU %
1.46%

And docker stats --no-stream --format "table {{.MemPerc}}" $(cat pid.txt), which gives:

MEM %
19.04%

(But we'd still need to strip off the % for this to be coerced to a double).

geomacy commented 6 years ago

In that case you could do 

docker stats --no-stream --format "table {{.CPUPerc}}" $(cat pid.txt) | sed -n '2s/%//p'

to get just the 1.46 and similarly for the memory percent.

aledsage commented 6 years ago

See fix in https://github.com/brooklyncentral/clocker/pull/446