Cannot create database in a k8s deployment

brent-artesion commented 6 months ago

ArcadeDB Version:

v.24.5.1-SNAPSHOT (build 6841fe32aab0676e3bdd8388d41a19882adb25e7/1716312977604/main)

OS and JDK Version:

Linux 5.15.0-105-generic - OpenJDK 64-Bit Server VM 11.0.23 (Temurin-11.0.23+9)

Expected behavior

Database created

Actual behavior

Error message: "Internal error Cannot create directory '/mnt/data0/databases/test'

Steps to reproduce

Kubernetes deployment

Access the admin ui
Login as root
Navigate to the database screen
Click "+ Create" button
Create New Database dialog screen
Enter database name "test"
Click "Send"

My cluster is k3s installed on a three node on-prem cluster. The above error occurs when I use a Longhorn storage class.

I get different error when I use local storage classes of local-path or openebs-hostpath. In those cases the error is:

"Internal error class com.arcadedb.database.LocalDatabase cannot be cast to class com.arcadedb.server.ha.ReplicatedDatabase (com.arcadedb.database.LocalDatabase and com.arcadedb.server.ha.ReplicatedDatabase are in unnamed module of loader 'app')"

lvca commented 6 months ago

Do you have the full stack trace of the exception?

brent-artesion commented 6 months ago

There seems to be two different issues and perhaps you'd want to split them into two separate issues.

1) This is the exception when using a longhorn storage class for the pvc

here's what's in the logs when the pods start up for the first time

│ arcadedb-0 2024-05-24 18:06:52.773 INFO  [ArcadeDBServer] <arcadedb-0> Creating default database 'Universe'...Exception in thread "main" com.arcadedb.exception.DatabaseOperationE │
│ xception: Cannot create directory '/mnt/data0/databases/Universe'                                                                                                                  │
│ arcadedb-0     at com.arcadedb.database.LocalDatabase.create(LocalDatabase.java:205)                                                                                               │
│ arcadedb-0     at com.arcadedb.database.DatabaseFactory.create(DatabaseFactory.java:90)                                                                                            │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.createDatabase(ArcadeDBServer.java:355)                                                                                       │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.loadDefaultDatabases(ArcadeDBServer.java:577)                                                                                 │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:158)                                                                                                │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:106)

Also, when I exec into the pod/container only /mnt/data0 exists. i.e. databases directory is not there

2) This is the exception creating the database when using a local-path or a openebs-hostpath

2024-05-24 18:19:44.040 INFO  [ArcadeDBServer] <arcadedb-1> Studio web tool available at http://arcadedb-1.arcadedb.default.svc.cluster.local:2480 <arcadedb-1> Error on command execution (PostServerCommandHandler)
java.lang.ClassCastException: class com.arcadedb.database.LocalDatabase cannot be cast to class com.arcadedb.server.ha.ReplicatedDatabase (com.arcadedb.database.LocalDatabase and com.arcadedb.server.ha.ReplicatedDatabase are in unnamed module of loader 'app')
    at com.arcadedb.server.http.handler.PostServerCommandHandler.createDatabase(PostServerCommandHandler.java:170)
    at com.arcadedb.server.http.handler.PostServerCommandHandler.execute(PostServerCommandHandler.java:93)
    at com.arcadedb.server.http.handler.AbstractServerHttpHandler.handleRequest(AbstractServerHttpHandler.java:127)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
    at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
    at java.base/java.lang.Thread.run(Thread.java:829)

What also shows up in the logs and IDK if it's related or not is an error electing a leader, maybe?

2024-05-24 18:19:48.641 INFO  [HAServer] Starting election of local server asking for votes from [arcadedb-0.arcadedb.default.svc.cluster.local:2424] (turn=1 retry=0 lastReplicationMessage=-1 configuredServers=1 majorityOfVotes=1)Error contacting server arcadedb-0.arcadedb.default.svc.cluster.local:2424 for election
java.net.UnknownHostException: arcadedb-0.arcadedb.default.svc.cluster.local
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at com.arcadedb.network.binary.ChannelBinaryClient.<init>(ChannelBinaryClient.java:46)
    at com.arcadedb.server.ha.HAServer.createNetworkConnection(HAServer.java:1008)
    at com.arcadedb.server.ha.HAServer.startElection(HAServer.java:1116)
    at java.base/java.lang.Thread.run(Thread.java:829)

To get this to deploy on a local, on-prem, cluster I did change the arcadedb-http service type to ClusterIP because LoadBalancer doesn't work in such a deploy (k3s) where we use a reverse proxy to expose services.

yurymkomarov commented 5 months ago

@brent-artesion

you need to set spec.template.spec.securityContext.fsGroup: 1000
pass all flags as ARCADEDB_SETTINGS variable, cause for some reasons service ignores them from command (you can see that in init log with list of all params)

ArcadeData / arcadedb