quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.85k stars 2.7k forks source link

Track which dev services work using kubedock #34051

Open iocanel opened 1 year ago

iocanel commented 1 year ago

Description

There have been discussion about running dev services over kubedock, to support dockerless environments. We should first check which dev services work with kubedock and which are not.

Failing Dev Services

Here's a list of Dev Services that do not work

See more details & logs in the comments below

Working Dev Services

Here's a list of Dev Services that do not work

Core extensions with DevService support

quarkus-bot[bot] commented 1 year ago

/cc @geoand (devservices), @stuartwdouglas (devservices)

iocanel commented 1 year ago

cc @maxandersen

iocanel commented 1 year ago

Most of the quick starts I tried failed.

From what I understand in most cases the issue has to do with file permissions. The root cause is the that images used created their own user (e.g. mysql, mariadb etc) and use that to setup directories while in Openshift containers run as random users causing all those permission issues.

There are specific guidelines on how to fix those images but I am not sure if we want to get down to that road.

jdbc-postgres (hibernate-orm-quickstart)

Status: FAILED

│ chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted                                                                            │
│ chmod: changing permissions of '/var/run/postgresql': Operation not permitted                                                                                 │
│ The files belonging to this database system will be owned by user "1011300000".                                                                               │
│ This user must also own the server process.                                                                                                                   │
│                                                                                                                                                               │
│ The database cluster will be initialized with locale "en_US.utf8".                                                                                            │
│ The default database encoding has accordingly been set to "UTF8".                                                                                             │
│ The default text search configuration will be set to "english".                                                                                               │
│                                                                                                                                                               │
│ Data page checksums are disabled.                                                                                                                             │
│                                                                                                                                                               │
│ fixing permissions on existing directory /var/lib/postgresql/data ... initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Op │
│ Stream closed EOF for ikanello1-dev/kubedock-967355e7b5b8 (main)

jdbc-mysql (hibernate-orm-quickstart)

Status: FAILED

MySQL container starts, logs seems clear but schema never gets created and tests eventually fail with Status 500: {"message":"command terminated with exit code 2"}

2023-09-06 07:59:27+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.34-1.el8 started.                                                       │
│ 2023-09-06 07:59:27+00:00 [Note] [Entrypoint]: Initializing database files                                                                                    │
│ 2023-09-06T07:59:27.586582Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use │
│ 2023-09-06T07:59:27.586687Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.34) initializing of server in progress as process 42                 │
│ 2023-09-06T07:59:27.588958Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root                                            │
│ 2023-09-06T07:59:27.592312Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.                                                                │
│ 2023-09-06T07:59:27.723833Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.                                                                  │
│ 2023-09-06T07:59:28.826775Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initializ │
│ 2023-09-06 07:59:30+00:00 [Note] [Entrypoint]: Database files initialized                                                                                     │
│ 2023-09-06 07:59:30+00:00 [Note] [Entrypoint]: Starting temporary server                                                                                      │
│ 2023-09-06T07:59:30.782806Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use │
│ 2023-09-06T07:59:30.785017Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.34) starting as process 86                                           │
│ 2023-09-06T07:59:30.796542Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root                                            │
│ 2023-09-06T07:59:30.802120Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.                                                                │
│ 2023-09-06T07:59:30.919266Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.                                                                  │
│ 2023-09-06T07:59:31.132007Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.                                                            │
│ 2023-09-06T07:59:31.132050Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this ch │
│ 2023-09-06T07:59:31.133202Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to a │
│ 2023-09-06T07:59:31.155639Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock                               │
│ 2023-09-06T07:59:31.157651Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.34'  socket: '/var/run/mysqld/mysqld.sock' │
│ 2023-09-06 07:59:31+00:00 [Note] [Entrypoint]: Temporary server started.                                                                                      │
│ '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'                                                                                                  │
│ Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.                                                                          │
│ Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.                                                                    │
│ Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.                                                                          │
│ Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.                                                                            │
│ Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.                                                                             │
│ Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.                                                                         │
│ 2023-09-06 07:59:33+00:00 [Note] [Entrypoint]: Creating database quarkus                                                                                      │
│ 2023-09-06 07:59:33+00:00 [Note] [Entrypoint]: Creating user quarkus                                                                                          │
│ 2023-09-06 07:59:33+00:00 [Note] [Entrypoint]: Giving user quarkus access to schema quarkus                                                                   │
│                                                                                                                                                               │
│ 2023-09-06 07:59:33+00:00 [Note] [Entrypoint]: Stopping temporary server                                                                                      │
│ 2023-09-06T07:59:33.765450Z 13 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.34).       

jdbc-oracle (hibernate-orm-quickstart)

Status: FAILED Container does not even get created.

jdbc-mariadb (hibernate-orm-quickstart)

Status: FAILED

2023-09-06 08:29:37+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.5+maria~ubu2204 started.                                          │
│ 2023-09-06 08:29:37+00:00 [Note] [Entrypoint]: Initializing database files                                                                                    │
│ 2023-09-06  8:29:37 0 [Warning] Can't create test file /var/lib/mysql/kubedock-b1e480f4fd1b.lower-test                                                        │
│ 2023-09-06  8:29:37 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")                                     │
│ 2023-09-06  8:29:37 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log                                                                      │
│ 2023-09-06  8:29:37 0 [ERROR] Aborting                                                                                                                        │
│                                                                                                                                                               │
│ Installation of system tables failed!  Examine the logs in                                                                                                    │
│ /var/lib/mysql/ for more information.                                                                                                                         │
│                                                                                                                                                               │
│ The problem could be conflicting information in an external                                                                                                   │
│ my.cnf files. You can ignore these by doing:                                                                                                                  │
│                                                                                                                                                               │
│     shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf                                                                                              │
│                                                                                                                                                               │
│ You can also try to start the mariadbd daemon with:                                                                                                           │
│                                                                                                                                                               │
│     shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &                                                                                             │
│                                                                                                                                                               │
│ and use the command line tool /usr/bin/mariadb                                                                                                                │
│ to connect to the mysql database and look at the grant tables:                                                                                                │
│                                                                                                                                                               │
│     shell> /usr/bin/mariadb -u root mysql                                                                                                                     │
│     MariaDB> show tables;                                                                                                                                     │
│                                                                                                                                                               │
│ Try '/usr/sbin/mariadbd --help' if you have problems with paths.  Using                                                                                       │
│ --general-log gives you a log in /var/lib/mysql/ that may be helpful.                                                                                         │
│                                                                                                                                                               │
│ The latest information about mysql_install_db is available at                                                                                                 │
│ https://mariadb.com/kb/en/installing-system-tables-mysql_install_db                                                                                           │
│ You can find the latest source at https://downloads.mariadb.org and                                                                                           │
│ the maria-discuss email list at https://launchpad.net/~maria-discuss                                                                                          │
│                                                                                                                                                               │
│ Please check all of the above before submitting a bug report   

jdbc-db2 (hibernate-orm-quickstart)

Status: FAILED

│ chmod: changing permissions of '/database': Operation not permitted                                                                                           │
│ mkdir: cannot create directory ‘/database/config’: Permission denied                                                                                          │
│ touch: cannot touch '/database/config/instance.cfg': No such file or directory                                                                                │
│ awk: cmd. line:1: fatal: cannot open file `/database/config/instance.cfg' for reading (No such file or directory)                                             │
│ /opt/ibm/db2/V11.5/bin/db2fupdt: line 254: /database/config/instance.cfg: No such file or directory                                                           │
│ /var/db2_setup/include/db2_common_functions: line 304: (!) Unable to backup the IP address ...: command not found                                             │
│ touch: cannot touch '/database/config/instance.cfg': No such file or directory                                                                                │
│ awk: cmd. line:1: fatal: cannot open file `/database/config/instance.cfg' for reading (No such file or directory)                                             │
│ /opt/ibm/db2/V11.5/bin/db2fupdt: line 254: /database/config/instance.cfg: No such file or directory                                                           │
│ /var/db2_setup/include/db2_common_functions: line 309: (!) Unable to backup the database port number ...: command not found                                   │
│ touch: cannot touch '/database/config/instance.cfg': No such file or directory                                                                                │
│ awk: cmd. line:1: fatal: cannot open file `/database/config/instance.cfg' for reading (No such file or directory)                                             │
│ /opt/ibm/db2/V11.5/bin/db2fupdt: line 254: /database/config/instance.cfg: No such file or directory                                                           │
│ /var/db2_setup/include/db2_common_functions: line 314: (!) Unable to backup the instance name ...: command not found                                          │
│ (*) Previous setup has not been detected. Creating the users...                                                                                               │
│ (*) Creating users ...                                                                                                                                        │
│ mkdir: cannot create directory ‘/database/config’: Permission denied                                                                                          │
│ groupadd: Permission denied.                                                                                                                                  │
│ groupadd: cannot lock /etc/group; try again later.                                                                                                            │
│ groupadd: Permission denied.                                                                                                                                  │
│ groupadd: cannot lock /etc/group; try again later.                                                                                                            │
│ (*) Creating instance ...                                                                                                                                     │
│ /var/db2_setup/include/db2_common_functions: line 50: /opt/ibm/db2/V11.5/instance/db2icrt: Permission denied                                                  │
│ (!) There was a problem configuring the instance. Copying over logs and traces to /database/config ...                                                        │
│ mkdir: cannot create directory ‘/database/config/db2icrt’: No such file or directory                                                                          │
│ cp: cannot stat '/tmp/*db2*': No such file or directory                                                                                                       │
│                                                                                                                                                               │
│ Directory "/database/config/db2icrt" does not exist or is not a fully qualified path name.                                                                    │
│ Stream closed EOF for ikanello1-dev/kubedock-15501ca22d88 (main)

mongodb (mongodb-quickstart)

Status: FAILED

                                              Autoscroll:On      FullScreen:Off     Timestamps:Off     Wrap:Off                                               │
│ {"t":{"$date":"2023-09-06T09:02:56.649+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable T │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port": │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"W",  "c":"CONTROL",  "id":20720,   "ctx":"initandlisten","msg":"Available memory is less than system memo │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"U │
│ {"t":{"$date":"2023-09-06T09:02:56.652+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"opt │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"E",  "c":"STORAGE",  "id":20557,   "ctx":"initandlisten","msg":"DBException in initAndListen, terminating │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator  │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}         │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService" │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"initandlisten","msg":"Shutdown: going to close listening socket │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool" │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"STORAGE",  "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acqu │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"REPL",     "id":4784907, "ctx":"initandlisten","msg":"Shutting down the replica set node execut │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}     │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"} │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}           │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}             │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}  │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}   │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"FTDC",     "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}    │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}                             │
│ {"t":{"$date":"2023-09-06T09:02:56.653+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}   │
│ Stream closed EOF for ikanello1-dev/kubedock-1ad4dd8cf123 (main)                                                                                              │

smallrye-reactive-messaging-kafka (kafka-quickstart)

Status: FAILED

redis-client (redis-quickstart)

Status: OK

keycloak-authorization (security-keycloak-authorization)

Status: OK

n1hility commented 1 year ago

Thanks for diving into this @iocanel!

Most of the quick starts I tried failed.

From what I understand in most cases the issue has to do with file permissions. The root cause is the that images used created their own user (e.g. mysql, mariadb etc) and use that to setup directories while in Openshift containers run as random users causing all those permission issues.

+1 Many popular images are designed around running as root, and fall apart when passed a UID. Or they expected a fixed non-root UID, which is then overridden by kube/openshift to a random one. Another common failure case is a pre-created image path with files owned by root get squashed by the storage provider to a different UID than the one running the container. Once userns support lands in Kube/Openshift (coming soon), the situation should eventually improve since fixed UIDs are fine (get remapped to a different UID).

There are specific guidelines on how to fix those images but I am not sure if we want to get down to that road.

IMO it is worth it to work with upstream image providers to improve their images. Aside from using our own layers to workaround we might be able to patch kubedock to support (if it doesnt already) init container usage to fix up file related issues.

iocanel commented 1 year ago

What I find interesting is that some of the failing images, work if they are used with oc new-app. For examples the centos and rhel flavors of postgress fall into that category. I can even manually portforward and use them in the quickstarts, but still fail when used via kubedock. The only differnence between kubedock and oc I observe is that oc automatically adds an empty dir volume to the pod when its declared in the image. It might worth implementing something like this in kubedock to see if it makes any difference.

joyrex2001 commented 1 year ago

This does not apply to all issues, but nevertheless might help getting more insight.

Since volume mounts are difficult to get correctly configured in ci, and may not even possible if testcontainers is using a remote host (eg testcontainers cloud), a lot of the modules are copying config instead of using volumes.

One of the gaps when simulating the docker api on kubernetes is that in docker, you can copy files to an unstarted container, while in kubernetes you can’t since there is no unstarted state. So kubedock will “always” start a container when files are being copied to it. (“always” because there is a —pre-archive flag which works around this, using configmaps).

This approach however, can lead to race exceptions (e.g. copying the config after the container proces requires it), or to permission issues. Usually this is overcome by implemeting a script that waits until the config is present in the container. However, this script might have a destination that is not writable for non-root users, like /.

The kafka devservice has its own implementation, and implements it in a pattern that can fit both worlds. See this pr as well: https://github.com/quarkusio/quarkus/pull/19736

This is another interesting pr, tackling a similar issue: https://github.com/testcontainers/testcontainers-java/pull/7524

tqvarnst commented 1 year ago

Another thing that worries me about using a shared cluster like this with testcontainers is that most (if not all) containers are hosted on DockerHub, and a shared cluster is likely to run into rate limits. We need to take this into consideration as well and see if we can find a workaround by caching images in the local OpenShift cluster.

Blarc commented 1 year ago

I am trying to run test containers as kubernetes pods in GitLab pipeline using GitLab Kubernetes Executor (as runner). I added kubedock as GitLab service.

This is my current devservices configuration:

'%test':
  quarkus:
    devservices:
      enabled: true
      timeout: 300
    datasource:
      devservices:
        image-name: mariadb:10.11
    elasticsearch:
      devservices:
        image-name: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
# Works with opensearch distribution
#        distribution: opensearch
#        image-name: docker.io/opensearchproject/opensearch:2.9.0
    kafka:
      devservices:
        image-name: docker.io/vectorized/redpanda:v22.3.4

This works when using opensearch as the distribution of elasticsearch, but fails when I use default (elastic) distribution. The other two containers (kafka and mariadb) are started up correctly.

java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.elasticsearch.restclient.common.deployment.DevServicesElasticsearchProcessor#startElasticsearchDevService threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    at io.quarkus.elasticsearch.restclient.common.deployment.DevServicesElasticsearchProcessor.startElasticsearchDevService(DevServicesElasticsearchProcessor.java:107)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
    at java.base/java.lang.Thread.run(Thread.java:840)
    at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:361)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:334)
    at io.quarkus.elasticsearch.restclient.common.deployment.DevServicesElasticsearchProcessor.lambda$startElasticsearch$1(DevServicesElasticsearchProcessor.java:229)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at io.quarkus.elasticsearch.restclient.common.deployment.DevServicesElasticsearchProcessor.startElasticsearch(DevServicesElasticsearchProcessor.java:243)
    at io.quarkus.elasticsearch.restclient.common.deployment.DevServicesElasticsearchProcessor.startElasticsearchDevService(DevServicesElasticsearchProcessor.java:97)
    ... 11 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:346)
    ... 16 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:565)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:356)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    ... 17 more
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"command terminated with exit code 2"}
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:30)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:13)
    at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
    at org.testcontainers.shaded.com.github.dockerjava.core.command.CopyArchiveFromContainerCmdImpl.exec(CopyArchiveFromContainerCmdImpl.java:66)
    at org.testcontainers.containers.ContainerState.copyFileFromContainer(ContainerState.java:378)
    at org.testcontainers.elasticsearch.ElasticsearchContainer.containerIsStarted(ElasticsearchContainer.java:136)
    at org.testcontainers.containers.GenericContainer.containerIsStarted(GenericContainer.java:724)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:544)
    ... 19 more
    at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:639)
    at io.quarkus.test.junit.QuarkusTestExtension.interceptTestClassConstructor(QuarkusTestExtension.java:723)
    at java.base/java.util.Optional.orElseGet(Optional.java:364)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

The log from the elasticsearch pod does not contain any errors. When observing the whole process, it looks to me like the pod is killed before elasticsearch finishes initializing.

elastic-no-error-log.txt

Blarc commented 1 year ago

After some debugging, I found out that elasticsearch fails, because testcontainers make request for retrieving cert /usr/share/elasticsearch/config/certs/http_ca.crt from container but the request fails with 500 ~is missing and kubedock returns 500 instead of 404~.

image

Edit: It seems the request for retrieving cert is too early. Running the request a bit later after the container starts, successfully downloads the cert.

Update: Solved with https://github.com/joyrex2001/kubedock/pull/60#event-10969641193.

xstefank commented 7 months ago

We did the superhero workshop in Dev Spaces yesterday, so I can confirm that with Quarkus 3.9.2

jdbc-postgresql works without any additional configuration messaging-kafka works with Redpanda, and also we fixed kafka-native

ahussey-redhat commented 2 months ago

Hello, I'm not sure if this issue is still active, but Keycloak doesn't appear to be working with kubedock

Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2024-09-02 06:28:56,117 INFO [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 10602ms
Server configuration updated and persisted. Run the following command to review the configuration:
kc.sh show-config
Next time you run the server, just run:
kc.sh start --http-enabled=true --hostname-strict=false --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --optimized
ERROR: Unexpected error when starting the server in (production) mode
ERROR: Failed to start quarkus
ERROR: Failed to reaad default user profile configuration: /opt/keycloak/upconfig.json
ERROR: /opt/keycloak/upconfig.json (No such file or directory)
For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.