Error parsing stdout from checks of some containers

[elyulka]

I see in logs that some container checks result in runtime errors. I've enabled DEBUG mode though it dumps stdout only if exit code is non zero but not in case of exception while parsing it, so I can't determine what child container outputs.

Traceback (most recent call last):
  File "/code/orchestrator.py", line 309, in check_dns_in_cluster
    check_result = ContainerNetworkTableResult.schema().loads(res.stdout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/marshmallow/schema.py", line 755, in loads
    data = self.opts.render_module.loads(json_data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2024-04-15 13:34:53 [docker-service-dns-prometheus-exporter]: Checking Container 922da503e5ccad5be26c0e3f9cdeeef942bcbc33cac0511de7a53545551578ff (service_name=cron_dockerproxy, service_id=0mde47zxyy9o0879stupexbt1, container_id=922da503e5ccad5be26c0e3f9cdeeef942bcbc33cac0511de7a53545551578ff, node_id=ye1tqwlj7lag2sy839fce03ca)
Traceback (most recent call last):
  File "/code/orchestrator.py", line 309, in check_dns_in_cluster
    check_result = ContainerNetworkTableResult.schema().loads(res.stdout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/marshmallow/schema.py", line 755, in loads
    data = self.opts.render_module.loads(json_data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2024-04-15 13:34:54 [docker-service-dns-prometheus-exporter]: Checking Container ea7e28ff9a61b6b047694abcb428e11a03c36537c49efad739ad3f4d2185b962 (service_name=mon_cadvisor, service_id=57ikn63opexzvrdzo7xxx4zp4, container_id=ea7e28ff9a61b6b047694abcb428e11a03c36537c49efad739ad3f4d2185b962, node_id=l2k9ezxymne48hjuyswx02z6j)
Traceback (most recent call last):
  File "/code/orchestrator.py", line 309, in check_dns_in_cluster
    check_result = ContainerNetworkTableResult.schema().loads(res.stdout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/marshmallow/schema.py", line 755, in loads
    data = self.opts.render_module.loads(json_data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[s4ke]

Please try again with 0.1.22

[elyulka]

@s4ke Thank you for quick response. Seems like problem is that my docker daemon log driver is fluentd and that results in empty stdout when running child container. It's better to execute it like docker run -it --log-driver local ... to make it more error-prone in different environments.

[s4ke]

I just minted 0.1.23 - try to use that once it is done building.

[elyulka]

Thanks! Unfortunately it didn't help. Still catches empty child stdout. I supposed it could be linked to moby issue 45689 - i enforce cpu resources limit, but i removed limits and it also didn't help. Very strange issue

[elyulka]

Maybe somehow linked to Standard output and Standard error are not being written unless docker is invoked with -i