Using python-on-whales installed via Poetry throws Pydantic errors

[ivanabrkic]

Hello everyone!

I have an issue with using python-on-whales installed via poetry.

poetry version: 1.8.2 (installed via pipx) python version: 3.9.6 python-on-whales version: 0.68.0

When running my python script, I am getting this error:

pydantic_core._pydantic_core.ValidationError: 10 validation errors for ComposeConfig services.anvil.entrypoint Input should be a valid list [type=list_type, input_value='anvil', input_type=str] For further information visit https://errors.pydantic.dev/2.5/v/list_type services.api_redis_pubsub_handler.command Input should be a valid list [type=list_type, input_value='python project_root/api...redis_pubsub_handler.py', input_type=str]

Error source:

docker.compose.config()

I did manage to bypass this error when I used return_json = True parameter, but then I got other errors, also regarding python-on-whales commands. These errors are not that much relevant because this setup was working before and then I had to reinstall my system - and now I'm blocked.

Do you have any idea why I have these issues?

[gabrieldemarmiesse]

Maybe Python-on-whales encountered a new syntax for compose? Can you give the docker-compose.yml? You can remove stuff that you don't want to share.

I'm interested in the syntax use to declare the command of the service "services.api_redis_pubsub_handler"

[ivanabrkic]

I would need to change a lot to give you whole docker-compose, but I can give you part by part and then edit it. Version of docker-compose is 3.5

api_redis_pubsub_handler:
    <<: *some_template
    ports: [ ]
    command: python project_root/api/redis_pubsub_handler.py
    depends_on:
      - some_other_service
    network_mode: "host"

Thanks for helping!

[ivanabrkic]

I've manage to solve it by separating command arguments with '-'.

Thanks again!

[rwuthric]

I have same issue. Here a minimal code to reproduce the error:

from python_on_whales import DockerClient
docker = DockerClient(compose_files=['docker-compose.yml'])
docker.compose.config()

with this docker-compose.yml file:

version: '3.9'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    container_name: zookeeper
    networks:
      - factory-net
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:7.4.0
    container_name: broker
    expose:
      - 29092
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    networks:
      - factory-net
    depends_on:
      - zookeeper
    healthcheck:
      test: nc -z localhost 9092 || exit -1
      start_period: 15s
      interval: 5s
      timeout: 10s
      retries: 10
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092     
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.29.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      broker:
         condition: service_healthy
    healthcheck:
      test: nc -z localhost 8088 || exit -1
      start_period: 15s
      interval: 5s
      timeout: 10s
      retries: 10
    ports:
      - "8088:8088"
    networks:
      - factory-net
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

  ksqldb-cli:
    # To run interactively:
    # docker exec -it ksqldb-cli ksql http://ksqldb-server:8088
    image: confluentinc/ksqldb-cli:0.29.0
    container_name: ksqldb-cli
    environment:
      - KSQL_START_SCRIPTS=mtcdevices.sql
    volumes:
      - "./scripts:/home/appuser/scripts"
    networks:
      - factory-net
    depends_on:
      broker:
         condition: service_healthy
      ksqldb-server:
         condition: service_healthy
    entrypoint: bash -c "source ./scripts/start_ksql.sh; tail -F anything"
    tty: true

networks:

  factory-net:
    name: factory-net
    external: true

And here the pydantic ValidationError

pydantic_core._pydantic_core.ValidationError: 8 validation errors for ComposeConfig
services.broker.environment.KAFKA_BROKER_ID
  Input should be a valid string [type=string_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
  Input should be a valid string [type=string_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
  Input should be a valid string [type=string_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
  Input should be a valid string [type=string_type, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.ksqldb-cli.environment
  Input should be a valid dictionary [type=dict_type, input_value=['KSQL_START_SCRIPTS=mtcdevices.sql'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.6/v/dict_type
services.ksqldb-cli.entrypoint
  Input should be a valid list [type=list_type, input_value='bash -c "source ./script...l.sh; tail -F anything"', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/list_type
services.zookeeper.environment.ZOOKEEPER_CLIENT_PORT
  Input should be a valid string [type=string_type, input_value=2181, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.zookeeper.environment.ZOOKEEPER_TICK_TIME
  Input should be a valid string [type=string_type, input_value=2000, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/string_type

However, the docker-compose.yml file is perfectly valid and the command

docker compose config

doesn't throw any error. The compose project can be spin up without any problem with

docker compose up

I am using python-on-whales==0.69.0 and Docker version 25.0.4.

[gabrieldemarmiesse]

Many thanks for the reproducible example! I will be able to fix it fast, I think it's just a small change to do in the pydantic model :)

[gabrieldemarmiesse]

So after a bit of research: docker compose v2.24.6 works totally fine. But in docker compose v2.24.7 the pydantic parsing crashes left and right.

When I compare the output of docker compose config --format json, it's changing quite a bit, at least with the example you provided:

# v2.24.6
...
            "environment": {
                "KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092",
                "KAFKA_BROKER_ID": "1",
                "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP": "PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT",
                "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR": "1",
                "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR": "1",
                "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR": "1",
                "KAFKA_ZOOKEEPER_CONNECT": "zookeeper:2181"
            },
            "expose": [
                "29092"
            ],
            "healthcheck": {
                "test": [
                    "CMD-SHELL",
                    "nc -z localhost 9092 || exit -1"
                ],
                "timeout": "10s",
                "interval": "5s",
                "retries": 10,
                "start_period": "15s"
            },
...

# v2.24.7
      "environment": {
          "KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092",
          "KAFKA_BROKER_ID": 1,
          "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP": "PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT",
          "KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR": 1,
          "KAFKA_TRANSACTION_STATE_LOG_MIN_ISR": 1,
          "KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR": 1,
          "KAFKA_ZOOKEEPER_CONNECT": "zookeeper:2181"
        },
        "expose": [
          29092
        ],
        "healthcheck": {
          "interval": "5s",
          "retries": 10,
          "start_period": "15s",
          "test": "nc -z localhost 9092 || exit -1",
          "timeout": "10s"
        },

As you can see, a lot of stuff changed. Some list were converted to string, some strings to int, etc...

We can't, with python-on-whales, try to keep backward compatibility, when docker compose is breaking backward compatibility, especially if they decide to not fix it in the future. The only thing we can do here is to make the pydantic model more flexible and to let the users handle the type changes in their code (or ensure they only work with one version of docker compose in their team).

Fix is coming soon, but be aware that some types will change.

Also I would like to say that nothing in the compose release notes indicate this change: https://github.com/docker/compose/releases/tag/v2.24.7

[gabrieldemarmiesse]

I wouldn't be surprised if there were other parsing bug with compose v2.24.7. Do not hesitate to report those bugs (or do pull requests), and in the meantime, you can always install compose v2.24.6 from the release page of docker compose. The installation instruction to change the version are here: https://github.com/docker/compose. It's quite easy to do, it's just a wget and chmod +x