Open RaduStoicescu opened 4 years ago
@RaduStoicescu : modeldb backend is unable to connect to postgres , can you verify it postgres container is up?
I am not sure how this is done on windows but this is a list of container which needs to be up for modeldb to work correctly.
(⎈) ~/w/c/o/k/l/e/p/m/backend> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d8ad539fcf0 vertaaiofficial/modeldb-frontend:2.0.7 "docker-entrypoint.s…" 8 seconds ago Up 7 seconds 0.0.0.0:3000->3000/tcp modeldb-frontend
cecb257725c5 postgres "docker-entrypoint.s…" 11 seconds ago Up 10 seconds 5432/tcp modeldb-postgres
1d02914e811a vertaaiofficial/modeldb-proxy:2.0.7 "/app/bin/proxy" 11 days ago Up 8 seconds 0.0.0.0:8080->8080/tcp modeldb-proxy
84abdfb07132 vertaaiofficial/modeldb-graphql:2.0.7 "/app/bin/graphql" 11 days ago Up 8 seconds 0.0.0.0:4000->4000/tcp modeldb-graphql
271c82d3f177 vertaaiofficial/modeldb-backend:2.0.7 "java -jar modeldb-1…" 11 days ago Up 9 seconds 0.0.0.0:8085-8086->8085-8086/tcp modeldb-backend
If postgres had problems coming up it should also show up earlier in the logs you shared.
Looks like it has trouble having access to "/var/lib/postgresql/data/pgdata". I'm not sure how to create or how to set permissions for that directory inside the Docker
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
2020-07-14 19:43:17.163 UTC [82] FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
2020-07-14 19:43:17.163 UTC [82] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```
Hey @RaduStoicescu, I am a contributor to modeldb backend and have docker-compose working in my setup on windows. My set up is
Windows 10 Home single language
I have used docker toolbox (https://docs.docker.com/toolbox/toolbox_install_windows/) setup for the dockerization of applications.
this is my docker-compose-all.yaml file
version: "3.5"
services:
modeldb-backend:
image: vertaaiofficial/modeldb-backend:2.0.7
container_name: "modeldb-backend"
expose:
- "8085"
- "8086"
ports:
- "8085:8085"
- "8086:8086"
networks:
- modeldb_network
environment:
- VERTA_MODELDB_CONFIG:/config/config.yaml
volumes:
- /c/Users/config:/config/
- /c/Users/artifact-store:/artifact-store/
modeldb-proxy:
image: vertaaiofficial/modeldb-proxy:2.0.7
container_name: "modeldb-proxy"
environment:
MDB_ADDRESS: "modeldb-backend:8085"
SERVER_HTTP_PORT: "8080"
expose:
- "8080"
ports:
- "8080:8080"
depends_on:
- modeldb-backend
networks:
- modeldb_network
modeldb-graphql:
image: vertaaiofficial/modeldb-graphql:2.0.7
container_name: "modeldb-graphql"
environment:
MDB_ADDRESS: "modeldb-backend:8085"
SERVER_HTTP_PORT: "4000"
QUERY_PATH: "/api/v1/graphql/query"
expose:
- "4000"
ports:
- "4000:4000"
depends_on:
- modeldb-backend
networks:
- modeldb_network
modeldb-frontend:
image: vertaaiofficial/modeldb-frontend:2.0.7
container_name: "modeldb-frontend"
expose:
- "3000"
ports:
- "3000:3000"
depends_on:
- modeldb-proxy
- modeldb-graphql
environment:
DEPLOYED: "yes"
BACKEND_API_PROTOCOL: "http"
BACKEND_API_DOMAIN: "modeldb-webapp:3000"
MDB_ADDRESS: http://modeldb-proxy:8080
ARTIFACTORY_ADDRESS: http://modeldb-backend:8086
GQL_ADDRESS: http://modeldb-graphql:4000
networks:
- modeldb_network
modeldb-postgres:
image: postgres
container_name: "modeldb-postgres"
restart: always
environment:
POSTGRES_PASSWORD: root
PGDATA: /var/lib/postgresql/data/pgdata
networks:
- modeldb_network
volumes:
- /e/Users/data:/var/lib/postgresql/data/pgdata
networks:
modeldb_network:
where I have used shared /c/Users
volume for config & artifact-store and for the Postgres data I am using other drive location like /e/Users/data
.
I could not use /c/Users
as mapped volume for the Postgres because it is by default read-only access by the windows and you can not do any write operation on it. Postgres will create backup volume on shared volume so you needs write permission on the shared volume.
Can you try to use other drive location instead of c:
as a mapped volume for Postgres as shown in docker-compose-all.yaml
above.
I hope you can replicate my setup and that unblocks you. If not, then let me know your Windows version and about the docker setup, and then I can help you better.
I changed it like this and I have the same problem
I have Windows 10 Business, version 1903
I have found this on docker forums, it looks like a common problem: https://forums.docker.com/t/data-directory-var-lib-postgresql-data-pgdata-has-wrong-ownership/17963
image: postgres
container_name: "modeldb-postgres"
restart: always
environment:
POSTGRES_PASSWORD: root
PGDATA: /var/lib/postgresql/data/pgdata
networks:
- modeldb_network
volumes:
- /d/users/data:/var/lib/postgresql/data/pgdata
This is the error:
modeldb-postgres | This user must also own the server process.
modeldb-postgres |
modeldb-postgres | The database cluster will be initialized with locale "en_US.utf8".
modeldb-postgres | The default database encoding has accordingly been set to "UTF8".
modeldb-postgres | The default text search configuration will be set to "english".
modeldb-postgres |
modeldb-postgres | Data page checksums are disabled.
modeldb-postgres |
modeldb-postgres | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
modeldb-postgres | creating subdirectories ... ok
modeldb-postgres | selecting dynamic shared memory implementation ... posix
modeldb-postgres | selecting default max_connections ... 20
modeldb-postgres | selecting default shared_buffers ... 400kB
modeldb-postgres | selecting default time zone ... Etc/UTC
modeldb-postgres | creating configuration files ... ok
modeldb-postgres | running bootstrap script ... 2020-07-16 17:56:19.080 UTC [82] FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
modeldb-postgres | 2020-07-16 17:56:19.080 UTC [82] HINT: The server must be started by the user that owns the data directory.
modeldb-postgres | child process exited with exit code 1
modeldb-postgres | initdb: removing contents of data directory "/var/lib/postgresql/data/pgdata"```
@RaduStoicescu : were you able to bring up Postgres? if so could you share the high-level steps so that we could add it to the modeldb documentation.
If you can point us to the steps which unblocked you at a high level, someone from our team could update the documents.
If you rather prefer to contribute a PR you could add a section for Windows setup
under https://github.com/VertaAI/modeldb/blob/master/DEPLOY.md#Deploy-pre-published-images.
Basically found a similar issue on Ubuntu 22.04 with latest docker installed
docker compose -f docker-compose-all.yaml up
modeldb-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization
modeldb-postgres |
modeldb-postgres | 2024-04-17 06:35:33.234 UTC [1] LOG: starting PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
modeldb-postgres | 2024-04-17 06:35:33.235 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
modeldb-postgres | 2024-04-17 06:35:33.235 UTC [1] LOG: listening on IPv6 address "::", port 5432
modeldb-postgres | 2024-04-17 06:35:33.236 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
modeldb-postgres | 2024-04-17 06:35:33.240 UTC [29] LOG: database system was shut down at 2024-04-17 06:24:56 UTC
modeldb-postgres | 2024-04-17 06:35:33.248 UTC [1] LOG: database system is ready to accept connections
modeldb-graphql | 2024/04/17 06:35:33 SERVER_HTTP_PORT : 4000
modeldb-proxy | 2024/04/17 06:35:33 MDB_ADDRESS : modeldb-backend:8085
modeldb-proxy | 2024/04/17 06:35:33 SERVER_HTTP_PORT : 8080
modeldb-proxy | 2024/04/17 06:35:33 Starting verta-backend proxy on port : 8080
modeldb-frontend | yarn run v1.22.4
modeldb-frontend | warning package.json: No license field
modeldb-frontend | $ nodemon server.js
modeldb-frontend | [nodemon] 1.19.4
modeldb-frontend | [nodemon] to restart at any time, enter `rs`
modeldb-frontend | [nodemon] watching dir(s): *.*
modeldb-frontend | [nodemon] watching extensions: js,mjs,json
modeldb-frontend | [nodemon] starting `node server.js`
modeldb-frontend | [HPM] Proxy created: / -> http://modeldb-proxy:8080
modeldb-frontend | [HPM] Proxy rewrite rule created: "^/api/v1/modeldb" ~> "/v1"
modeldb-frontend | [HPM] Proxy created: / -> http://modeldb-backend:8086
modeldb-frontend | [HPM] Proxy created: / -> http://modeldb-graphql:4000
modeldb-frontend | [HPM] Proxy rewrite rule created: "^/api/v1/graphql/" ~> "/"
Repeating...
modeldb-backend | {"thread":"main","level":"DEBUG","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil getSessionFactory() retrying for DB connection after 2560 millisecond ","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1713335805,"nanoOfSecond":539000000},"threadId":1,"threadPriority":5,"hostName":"9c2449b11b85","kubernetes.podIP":""}
modeldb-backend | {"thread":"main","level":"WARN","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil checkDBConnection() got error ","thrown":{"commonElementCount":0,"localizedMessage":"org.mariadb.jdbc.Driver","message":"org.mariadb.jdbc.Driver","name":"java.lang.ClassNotFoundException","extendedStackTrace":"java.lang.ClassNotFoundException: org.mariadb.jdbc.Driver\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:435) ~[?:?]\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]\n\tat org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:89) ~[modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]\n\tat java.lang.Class.forName0(Native Method) ~[?:?]\n\tat java.lang.Class.forName(Class.java:340) ~[?:?]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnection(ModelDBHibernateUtil.java:477) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnectionInLoop(ModelDBHibernateUtil.java:325) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.createOrGetSessionFactory(ModelDBHibernateUtil.java:240) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.initializeServicesBaseOnDataBase(App.java:368) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.main(App.java:263) [classes!/:1.0-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1713335805,"nanoOfSecond":541000000},"threadId":1,"threadPriority":5,"hostName":"9c2449b11b85","kubernetes.podIP":""}
Try adding a Project on UI
modeldb-frontend | /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend | Requesting /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend | /static/media/Verta_logo.b2461278.svg
modeldb-frontend | Returning 503 Service Unavailable; 261b sent modeldb-frontend | /favicon.png modeldb-frontend | /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend | Requesting /api/v1/modeldb/hydratedData/findHydratedProjects
modeldb-frontend | Returning 503 Service Unavailable; 261b sent
On Windows 10 when running
docker-compose -f docker-compose-all.yaml up
I get the following:modeldb-backend | {"thread":"main","level":"DEBUG","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil getSessionFactory() retrying for DB connection after 2560 millisecond ","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737612,"nanoOfSecond":72000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""} modeldb-backend | {"thread":"main","level":"WARN","loggerName":"ai.verta.modeldb.utils.ModelDBHibernateUtil","message":"ModelDBHibernateUtil checkDBConnection() got error ","thrown":{"commonElementCount":0,"localizedMessage":"The connection attempt failed.","message":"The connection attempt failed.","name":"org.postgresql.util.PSQLException","cause":{"commonElementCount":19,"localizedMessage":"Connect timed out","message":"Connect timed out","name":"java.net.SocketTimeoutException","extendedStackTrace":"java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n"},"extendedStackTrace":"org.postgresql.util.PSQLException: The connection attempt failed.\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:292) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.makeConnection(Driver.java:458) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.Driver.connect(Driver.java:260) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:677) ~[java.sql:?]\n\tat java.sql.DriverManager.getConnection(DriverManager.java:228) ~[java.sql:?]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnection(ModelDBHibernateUtil.java:483) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.checkDBConnectionInLoop(ModelDBHibernateUtil.java:325) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.utils.ModelDBHibernateUtil.createOrGetSessionFactory(ModelDBHibernateUtil.java:240) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.initializeServicesBaseOnDataBase(App.java:363) [classes!/:1.0-SNAPSHOT]\n\tat ai.verta.modeldb.App.main(App.java:260) [classes!/:1.0-SNAPSHOT]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]\n\tat jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [modeldb-1.0-SNAPSHOT-client-build.jar:1.0-SNAPSHOT]\nCaused by: java.net.SocketTimeoutException: Connect timed out\n\tat sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]\n\tat sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:333) ~[?:?]\n\tat java.net.Socket.connect(Socket.java:648) ~[?:?]\n\tat org.postgresql.core.PGStream.<init>(PGStream.java:75) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.6.jar!/:42.2.6]\n\tat org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.6.jar!/:42.2.6]\n\t... 19 more\n"},"endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","instant":{"epochSecond":1594737622,"nanoOfSecond":77000000},"threadId":1,"threadPriority":5,"hostName":"03fd2e8e7248","kubernetes.podIP":""}