Closed ivanabrkic closed 5 months ago
Maybe Python-on-whales encountered a new syntax for compose? Can you give the docker-compose.yml? You can remove stuff that you don't want to share.
I'm interested in the syntax use to declare the command of the service "services.api_redis_pubsub_handler"
I would need to change a lot to give you whole docker-compose, but I can give you part by part and then edit it. Version of docker-compose is 3.5
api_redis_pubsub_handler:
<<: *some_template
ports: [ ]
command: python project_root/api/redis_pubsub_handler.py
depends_on:
- some_other_service
network_mode: "host"
Thanks for helping!
I've manage to solve it by separating command arguments with '-'.
Thanks again!
I have same issue. Here a minimal code to reproduce the error:
from python_on_whales import DockerClient
docker = DockerClient(compose_files=['docker-compose.yml'])
docker.compose.config()
with this docker-compose.yml
file:
version: '3.9'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.4.0
container_name: zookeeper
networks:
- factory-net
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
broker:
image: confluentinc/cp-kafka:7.4.0
container_name: broker
expose:
- 29092
ports:
# To learn about configuring Kafka for access across networks see
# https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
- "9092:9092"
networks:
- factory-net
depends_on:
- zookeeper
healthcheck:
test: nc -z localhost 9092 || exit -1
start_period: 15s
interval: 5s
timeout: 10s
retries: 10
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
ksqldb-server:
image: confluentinc/ksqldb-server:0.29.0
hostname: ksqldb-server
container_name: ksqldb-server
depends_on:
broker:
condition: service_healthy
healthcheck:
test: nc -z localhost 8088 || exit -1
start_period: 15s
interval: 5s
timeout: 10s
retries: 10
ports:
- "8088:8088"
networks:
- factory-net
environment:
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: broker:29092
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
ksqldb-cli:
# To run interactively:
# docker exec -it ksqldb-cli ksql http://ksqldb-server:8088
image: confluentinc/ksqldb-cli:0.29.0
container_name: ksqldb-cli
environment:
- KSQL_START_SCRIPTS=mtcdevices.sql
volumes:
- "./scripts:/home/appuser/scripts"
networks:
- factory-net
depends_on:
broker:
condition: service_healthy
ksqldb-server:
condition: service_healthy
entrypoint: bash -c "source ./scripts/start_ksql.sh; tail -F anything"
tty: true
networks:
factory-net:
name: factory-net
external: true
And here the pydantic
ValidationError
pydantic_core._pydantic_core.ValidationError: 8 validation errors for ComposeConfig
services.broker.environment.KAFKA_BROKER_ID
Input should be a valid string [type=string_type, input_value=1, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
Input should be a valid string [type=string_type, input_value=1, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
Input should be a valid string [type=string_type, input_value=1, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.broker.environment.KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
Input should be a valid string [type=string_type, input_value=1, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.ksqldb-cli.environment
Input should be a valid dictionary [type=dict_type, input_value=['KSQL_START_SCRIPTS=mtcdevices.sql'], input_type=list]
For further information visit https://errors.pydantic.dev/2.6/v/dict_type
services.ksqldb-cli.entrypoint
Input should be a valid list [type=list_type, input_value='bash -c "source ./script...l.sh; tail -F anything"', input_type=str]
For further information visit https://errors.pydantic.dev/2.6/v/list_type
services.zookeeper.environment.ZOOKEEPER_CLIENT_PORT
Input should be a valid string [type=string_type, input_value=2181, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
services.zookeeper.environment.ZOOKEEPER_TICK_TIME
Input should be a valid string [type=string_type, input_value=2000, input_type=int]
For further information visit https://errors.pydantic.dev/2.6/v/string_type
However, the docker-compose.yml
file is perfectly valid and the command
docker compose config
doesn't throw any error. The compose project can be spin up without any problem with
docker compose up
I am using python-on-whales==0.69.0
and Docker version 25.0.4
.
Many thanks for the reproducible example! I will be able to fix it fast, I think it's just a small change to do in the pydantic model :)
So after a bit of research: docker compose v2.24.6 works totally fine. But in docker compose v2.24.7 the pydantic parsing crashes left and right.
When I compare the output of docker compose config --format json
, it's changing quite a bit, at least with the example you provided:
# v2.24.6
...
"environment": {
"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092",
"KAFKA_BROKER_ID": "1",
"KAFKA_LISTENER_SECURITY_PROTOCOL_MAP": "PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT",
"KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR": "1",
"KAFKA_TRANSACTION_STATE_LOG_MIN_ISR": "1",
"KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR": "1",
"KAFKA_ZOOKEEPER_CONNECT": "zookeeper:2181"
},
"expose": [
"29092"
],
"healthcheck": {
"test": [
"CMD-SHELL",
"nc -z localhost 9092 || exit -1"
],
"timeout": "10s",
"interval": "5s",
"retries": 10,
"start_period": "15s"
},
...
# v2.24.7
"environment": {
"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092",
"KAFKA_BROKER_ID": 1,
"KAFKA_LISTENER_SECURITY_PROTOCOL_MAP": "PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT",
"KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR": 1,
"KAFKA_TRANSACTION_STATE_LOG_MIN_ISR": 1,
"KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR": 1,
"KAFKA_ZOOKEEPER_CONNECT": "zookeeper:2181"
},
"expose": [
29092
],
"healthcheck": {
"interval": "5s",
"retries": 10,
"start_period": "15s",
"test": "nc -z localhost 9092 || exit -1",
"timeout": "10s"
},
As you can see, a lot of stuff changed. Some list were converted to string, some strings to int, etc...
We can't, with python-on-whales, try to keep backward compatibility, when docker compose is breaking backward compatibility, especially if they decide to not fix it in the future. The only thing we can do here is to make the pydantic model more flexible and to let the users handle the type changes in their code (or ensure they only work with one version of docker compose in their team).
Fix is coming soon, but be aware that some types will change.
Also I would like to say that nothing in the compose release notes indicate this change: https://github.com/docker/compose/releases/tag/v2.24.7
I wouldn't be surprised if there were other parsing bug with compose v2.24.7. Do not hesitate to report those bugs (or do pull requests), and in the meantime, you can always install compose v2.24.6 from the release page of docker compose. The installation instruction to change the version are here: https://github.com/docker/compose. It's quite easy to do, it's just a wget
and chmod +x
Hello everyone!
I have an issue with using python-on-whales installed via poetry.
poetry version: 1.8.2 (installed via pipx) python version: 3.9.6 python-on-whales version: 0.68.0
When running my python script, I am getting this error:
pydantic_core._pydantic_core.ValidationError: 10 validation errors for ComposeConfig services.anvil.entrypoint Input should be a valid list [type=list_type, input_value='anvil', input_type=str] For further information visit https://errors.pydantic.dev/2.5/v/list_type services.api_redis_pubsub_handler.command Input should be a valid list [type=list_type, input_value='python project_root/api...redis_pubsub_handler.py', input_type=str]
Error source:
docker.compose.config()
I did manage to bypass this error when I used
return_json = True
parameter, but then I got other errors, also regarding python-on-whales commands. These errors are not that much relevant because this setup was working before and then I had to reinstall my system - and now I'm blocked.Do you have any idea why I have these issues?