[Bug]: Data Flushed/Deleted in local Docker setup after restarting the containers

ikharsha commented 2 weeks ago

What component(s) are affected?

[X] Python SDK
[ ] Opik UI
[X] Opik Server
[ ] Documentation

Opik version

Opik version: x.x.x

Describe the problem

While i setup Opik locally and use it ,the Traces ,Logs and all are being stored and shown up in the Dashboard but after stopping the Opik running containers and after restarting them the Data is been Deleted/flushed

please sort this this out is there any command or code predefined to delete them or i need to do some modification to solve this persistent memory issue!!

Reproduction steps

No response

jverre commented 2 weeks ago

@ikharsha Let me take a look, this shouldn't be the case but we did make some changes to the docker compose deployment recently so will try to reproduce

jverre commented 2 weeks ago

@ikharsha I've run the following and wasn't able to reproduce:

cd deployment/docker-compose

# Start Opik
docker compose up -d

# Logged some traces and prompts

# Stopped Opik
docker compose down

# Restarted Opik
docker compose up -d

# -> Data was still available

Can you share a bit more about your setup (Windows vs Mac machine for example) and some code to reproduce using the latest code on the main branch ? The data should definitely be persisted so we might have a bug depending on the setup

ikharsha commented 1 week ago

Hey @jverre, I deployed Opik on an Azure VM instance and can access it through the VM's URL on PORT:5173. I’ve configured the Opik URL the same way on my personal laptop. I’ve logged some traces, and everything is visible on the dashboard, working perfectly.

However, my VM automatically shuts down at midnight (12:00 AM). When I restart the VM in the morning, I manually restart the Opik dependency containers (Redis, MySQL, Clickhouse, backend, frontend) that were stopped due to the system shutdown. But now, I see nothing on my dashboard—everything appears new, and I’ve lost all the traces/logged data.

cd deployment/docker-compose
#Start Opik:
docker compose up -d

#Log some traces and prompts:
#Verified that traces and prompts were visible on the dashboard.
#VM automatically shuts down at midnight (12:00 AM):

#All Opik dependency containers (Redis, MySQL, Clickhouse, backend, frontend) stop due to the system shutdown.
#Restart the VM in the morning:

#Manually restart the containers:

docker start <container-ID1> <container-ID2> <container-ID3> <container-ID4> <container-ID5>

#Dashboard appears empty, and all previously logged traces and data are lost.

I hope this scenario makes sense. Should I run docker compose down to stop the containers before the shutdown and then start them again after the restart? Will this prevent the data loss?

liyaka commented 1 week ago

Hi @ikharsha, all data is configured to stay on the local volume. Can you please print here docker-compose.yaml that you use?

ikharsha commented 1 week ago

`name: opik

services: mysql: image: mysql:8.4.2 hostname: mysql environment: MYSQL_ROOT_PASSWORD: opik MYSQL_DATABASE: opik MYSQL_USER: opik MYSQL_PASSWORD: opik healthcheck: test: [ "CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "--silent" ] timeout: 1s interval: 1s retries: 300 ports:

"3306" volumes:
Mounted on your $HOME folder
~/opik/mysql:/var/lib/mysql/

redis: image: redis:7.2.4-alpine3.19 hostname: redis command: redis-server --requirepass opik ports:
'6379' healthcheck: test: [ "CMD", "nc", "-z", "localhost", "6379" ] interval: 2s timeout: 4s retries: 20 start_period: 30s

clickhouse: image: clickhouse/clickhouse-server:24.3.6.48-alpine hostname: clickhouse environment: CLICKHOUSE_DB: opik CLICKHOUSE_USER: opik CLICKHOUSE_PASSWORD: opik

Enables SQL-driven Access Control and Account Management:

https://clickhouse.com/docs/en/operations/access-rights#enabling-access-control

CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1 ports:
"8123" # HTTP default port
"9000" # Native Protocol port volumes:
Mounted on your $HOME folder
~/opik/clickhouse/data:/var/lib/clickhouse/
~/opik/clickhouse/logs:/var/log/clickhouse-server/ healthcheck: test: [ "CMD", "wget", "--spider", "-q", "http://127.0.0.1:8123/ping" ] interval: 1s timeout: 1s retries: 300

backend: image: ghcr.io/comet-ml/opik/opik-backend:${OPIK_VERSION:-latest} build: context: ../../apps/opik-backend dockerfile: Dockerfile args: OPIK_VERSION: ${OPIK_VERSION:-latest} platform: linux/amd64 hostname: backend command: [ "bash", "-c", "./run_db_migrations.sh && ./entrypoint.sh" ] environment: DOCKER_BUILDKIT: 1 STATE_DB_PROTOCOL: "jdbc:mysql://" STATE_DB_URL: "mysql:3306/opik?createDatabaseIfNotExist=true&rewriteBatchedStatements=true" STATE_DB_DATABASE_NAME: opik STATE_DB_USER: opik STATE_DB_PASS: opik ANALYTICS_DB_MIGRATIONS_URL: "jdbc:clickhouse://clickhouse:8123" ANALYTICS_DB_MIGRATIONS_USER: opik ANALYTICS_DB_MIGRATIONS_PASS: opik ANALYTICS_DB_PROTOCOL: "HTTP" ANALYTICS_DB_HOST: "clickhouse" ANALYTICS_DB_PORT: 8123 ANALYTICS_DB_USERNAME: opik ANALYTICS_DB_DATABASE_NAME: opik JAVA_OPTS: "-Dliquibase.propertySubstitutionEnabled=true" REDIS_URL: redis://:opik@redis:6379/ ANALYTICS_DB_PASS: opik OPIK_OTEL_SDK_ENABLED: false OTEL_VERSION: 2.9.0 OTEL_PROPAGATORS: "tracecontext,baggage,b3" OTEL_EXPERIMENTAL_EXPORTER_OTLP_RETRY_ENABLED: true OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION: BASE2_EXPONENTIAL_BUCKET_HISTOGRAM OTEL_EXPERIMENTAL_RESOURCE_DISABLED_KEYS: process.command_args OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE: delta OPIK_USAGE_REPORT_ENABLED: ${OPIK_USAGE_REPORT_ENABLED:-true} ports:
"8080"
"3003" depends_on: mysql: condition: service_healthy clickhouse: condition: service_healthy

frontend: image: ghcr.io/comet-ml/opik/opik-frontend:${OPIK_VERSION:-latest} build: context: ../../apps/opik-frontend dockerfile: Dockerfile platform: linux/amd64 hostname: frontend ports:
"5173:5173" extra_hosts:
"apihost:host-gateway" volumes:
./nginx_default_local.conf:/etc/nginx/conf.d/default.conf depends_on: backend: condition: service_started

networks: default: `

volumes are not reciding at the home folder location specified in the yaml file

~/opik/mysql:/var/lib/mysql/

but they found to recide in the parent folder of Opik repo clone

ikharsha commented 1 week ago

My machine gets shut down everyday so every morning i am starting my containers manually with the following command

docker start

and i am not using the below command

docker compose down

the containers are stopping with the machine being auto shut down by EOD

Could you check out the above comment and help out with the folders - ~/opik/mysql:/var/lib/mysql/

andrescrz commented 4 days ago

Hi @ikharsha

Thanks for all the details. We implemented the following changes in relation to Docker compose volumes:

https://github.com/comet-ml/opik/pull/626

From your docker-compose print out, I believe you aren't using the latest version. Could you please update your repository and try again? There a some upgrades that affect data storage.

In the meantime, we'd need the following information about you Azure VM:

Operating system and version.
What type of storage you're using: -- Ephemeral instance storage? -- Attached volumes?

Please provide as much details as you can. Thank you very much.

comet-ml / opik