Prints not appearing when using ExternalTaskWorker

[wmeijer221]

I've implemented a simple external task client in Python and docker-compose. However, none of my print() statements seem to appear in the console.

The code I wrote is rather straightforward:

import os

from camunda.external_task.external_task_worker import ExternalTaskWorker
from camunda.external_task.external_task import ExternalTask, TaskResult
from camunda.client.engine_client import EngineClient

ENDPOINT_KEY = "ENDPOINT"
TOPIC_KEY = "TOPIC"

def task_handler(task: ExternalTask) -> TaskResult:
    return task.complete()

if __name__ == "__main__":
    ENDPOINT = os.getenv(ENDPOINT_KEY)
    BASE_URL = f'http://{ENDPOINT}/engine-rest'
    TOPIC_NAME = os.getenv(TOPIC_KEY)

    print(f'{BASE_URL=}\n{TOPIC_NAME=}')

    client = EngineClient(BASE_URL)
    worker = ExternalTaskWorker(worker_id=0, base_url=BASE_URL)
    worker.subscribe(TOPIC_NAME, task_handler)

I learned that whenever I remove the last two lines (i.e. the ones creating/using worker), the prints show up just fine. Additionally, whenever I attach myself to the client's docker instance (using docker attach) and then cause a KeyboardInterrupt, the logs do show up. So they are printed at some point, just not at runtime.

Does anyone know how I can use print() in combination with ExternalTaskWorker? (Thanks in advance!)

Note that the code works just fine; whenever I create a task the client is interested in, it completes it.

In case it's relevant, this is the docker-compose I'm using:

version: "3"

services:
  postgres:
    image: postgres:11
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=app
      - POSTGRES_PASSWORD=password
    ports:
      - "5432:5432"

  camunda-workflow:
    image: camunda/camunda-bpm-platform:7.18.0
    container_name: camunda-workflow
    depends_on:
      - postgres
    ports:
      - "8080:8080"

  camunda-ext-client:
    build: .
    container_name: camunda-ext-client
    depends_on:
      - camunda-workflow
    environment:
      - ENDPOINT=camunda-workflow:8080
      - TOPIC=topicName

volumes:
  pgdata:
    driver: local

[wmeijer221]

I explored the problem a little further. When using other logging tools (e.g. logging), the same thing happens.

When launching the worker in a separate thread, the prints in the main thread seem to work, but the prints in the worker threads don't.

I'm starting to think this issue is more of a bug in ExternalTaskWorker than anything else. Maybe the workers are somehow blocking stdout?

Edit: After a little more exploration, I figured out that the following works:

import logging
logging.root.setLevel(logging.NOTSET)
logger = logging.getLogger(__name__)

So overwriting the logging level and then using the logging package works. I'm not too familiar with the logging package, so I'm not too sure whether this is by design or not.

It might be worth explicitly noting down somewhere in the README that you should use logging or log_with_context for your logs, though.