search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.58k stars 2.63k forks source link

Remote debug not working #23765

Closed edeandrea closed 2 years ago

edeandrea commented 2 years ago

I used the openshift extension to deploy my app to OpenShift via an S2I binary build. The app is up and running and quarkus:remote-dev connects to it fine and any local changes are pushed/reflected as expected.

My challenge is I can't seem to connect a remote debugger to it. I can't seem to get the pod to run in debug.

I've tried lots of combinations of environment variables/arguments, but nothing works. Here's what I've tried:

The Quarkus OpenShift config says that the base jvm image is registry.access.redhat.com/ubi8/openjdk-11. When looking at the documentation for the image I should just be able to set JAVA_DEBUG=true and it should work, but it doesn't seem to be.

Is this a Quarkus issue? An issue with the openjdk-11 image?

Originally posted by @edeandrea in https://github.com/quarkusio/quarkus/discussions/23243 - See there & https://github.com/quarkusio/quarkus/discussions/23243#discussioncomment-2064464 for more details

quarkus-bot[bot] commented 2 years ago

/cc @geoand, @iocanel

geoand commented 2 years ago

Is this a Quarkus issue?

It can't be as Quarkus has nothing to do with JVM debugging.

set env var JAVA_DEBUG=true

How were you setting this?

In any case, you'll need to see what the configuration of the running Pod is to see what environment variables and/or arguments are set. Another thing to check is to see exec into the the running container and find the actual java ... command that launched the application.

edeandrea commented 2 years ago

It can't be as Quarkus has nothing to do with JVM debugging.

But the Dockerfile.jvm that was used to create the image (https://github.com/quarkusio/quarkus-super-heroes/blob/main/rest-villains/src/main/docker/Dockerfile.jvm) came from code.quarkus.io. That Dockerfile defines its own ENTRYPOINT which does not include any debug "stuff".

set env var JAVA_DEBUG=true

How were you setting this?

By adding the environment variable to the DeploymentConfig and then triggering a rollout

In any case, you'll need to see what the configuration of the running Pod is to see what environment variables and/or arguments are set. Another thing to check is to see exec into the the running container and find the actual java ... command that launched the application.

It seems the image used (ubi8/openjdk-11-runtime) doesn't have any kind of cli tools installed (no ps nor lsof), so I'm not sure how to see what the actual java process looks like, although looking at the ENTRYPOINT I would suspect it is simply java -jar.

If I then push the app directly to OpenShift using the Quarkus openshift extension it seems to use the ubi8/openjdk-11 image. If you look at the documentation it mentions the whole JAVA_DEBUG thing.

I do not see the Listening for transport dt_socket at address message in the pod startup.

When I follow the instructions in that guide for the OpenJDK image with the example it uses everything works fine, so something in the ./mvnw package -Dquarkus.kubernetes.deploy=true isn't working the way it should.

Not sure if its helpful, but here's the s2i build log:

Receiving source from STDIN as archive ...
time="2022-02-17T13:31:26Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
I0217 13:31:26.256173       1 defaults.go:102] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
Caching blobs under "/var/cache/blobs".
Trying to pull registry.access.redhat.com/ubi8/openjdk-11@sha256:bef8578609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4...
Getting image source signatures
Copying blob sha256:8a53271b80c8bb6c008ffd7bc2a1c244014a8f5e576a4015dca62a7d6c9c9858
Copying blob sha256:131f1a26eef08c7274f7a115f461689819a4218e246ccacacc50ebcfd5e1c758
Copying blob sha256:16b78ed2e82205ceca54aa3a4205f029096a354d346b3bd2434a56b7f66e21c1
Copying config sha256:106bcc7299ebfc09cb1cf049181d9796c7c2faabdf8ea486c3d904704aec4a33
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image registry.access.redhat.com/ubi8/openjdk-11@sha256:bef8578609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4
Adding transient rw bind mount for /run/secrets/rhsm
STEP 1/9: FROM registry.access.redhat.com/ubi8/openjdk-11@sha256:bef8578609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4
STEP 2/9: LABEL "io.openshift.build.image"="registry.access.redhat.com/ubi8/openjdk-11@sha256:bef8578609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4"       "io.openshift.build.source-location"="/tmp/build/inputs"       "io.openshift.s2i.destination"="/tmp"
STEP 3/9: ENV OPENSHIFT_BUILD_NAME="rest-villains-1"     OPENSHIFT_BUILD_NAMESPACE="eric-deandrea-dev"
STEP 4/9: USER root
STEP 5/9: COPY upload/src /tmp/src
STEP 6/9: RUN chown -R 185:0 /tmp/src
STEP 7/9: USER 185
STEP 8/9: RUN /usr/local/s2i/assemble
INFO S2I source build with plain binaries detected
INFO Copying binaries from /tmp/src to /deployments ...
quarkus-app-dependencies.txt
quarkus-run.jar
app/
app/rest-villains-1.0.jar
lib/
lib/boot/
lib/boot/io.quarkus.quarkus-bootstrap-runner-2.7.0.Final.jar
lib/boot/io.quarkus.quarkus-development-mode-spi-2.7.0.Final.jar
lib/boot/io.smallrye.common.smallrye-common-io-1.8.0.jar
lib/boot/org.graalvm.sdk.graal-sdk-21.3.0.jar
lib/boot/org.jboss.logging.jboss-logging-3.4.3.Final.jar
lib/boot/org.jboss.logmanager.jboss-logmanager-embedded-1.0.9.jar
lib/boot/org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar
lib/main/
lib/main/antlr.antlr-2.7.7.jar
lib/main/com.fasterxml.classmate-1.5.1.jar
lib/main/com.fasterxml.jackson.core.jackson-annotations-2.13.1.jar
lib/main/com.fasterxml.jackson.core.jackson-core-2.13.1.jar
lib/main/com.fasterxml.jackson.core.jackson-databind-2.13.1.jar
lib/main/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.13.1.jar
lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jdk8-2.13.1.jar
lib/main/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar
lib/main/com.fasterxml.jackson.module.jackson-module-parameter-names-2.13.1.jar
lib/main/com.github.ben-manes.caffeine.caffeine-2.9.3.jar
lib/main/com.google.errorprone.error_prone_annotations-2.10.0.jar
lib/main/com.sun.activation.jakarta.activation-1.2.1.jar
lib/main/com.sun.istack.istack-commons-runtime-3.0.10.jar
lib/main/io.agroal.agroal-api-1.14.jar
lib/main/io.agroal.agroal-narayana-1.14.jar
lib/main/io.agroal.agroal-pool-1.14.jar
lib/main/io.micrometer.micrometer-core-1.8.2.jar
lib/main/io.micrometer.micrometer-registry-prometheus-1.8.2.jar
lib/main/io.netty.netty-buffer-4.1.73.Final.jar
lib/main/io.netty.netty-codec-4.1.73.Final.jar
lib/main/io.netty.netty-codec-dns-4.1.73.Final.jar
lib/main/io.netty.netty-codec-haproxy-4.1.73.Final.jar
lib/main/io.netty.netty-codec-http-4.1.73.Final.jar
lib/main/io.netty.netty-codec-http2-4.1.73.Final.jar
lib/main/io.netty.netty-codec-socks-4.1.73.Final.jar
lib/main/io.netty.netty-common-4.1.73.Final.jar
lib/main/io.netty.netty-handler-4.1.73.Final.jar
lib/main/io.netty.netty-handler-proxy-4.1.73.Final.jar
lib/main/io.netty.netty-resolver-4.1.73.Final.jar
lib/main/io.netty.netty-resolver-dns-4.1.73.Final.jar
lib/main/io.netty.netty-tcnative-classes-2.0.46.Final.jar
lib/main/io.netty.netty-transport-4.1.73.Final.jar
lib/main/io.prometheus.simpleclient-0.12.0.jar
lib/main/io.prometheus.simpleclient_common-0.12.0.jar
lib/main/io.prometheus.simpleclient_tracer_common-0.12.0.jar
lib/main/io.prometheus.simpleclient_tracer_otel-0.12.0.jar
lib/main/io.prometheus.simpleclient_tracer_otel_agent-0.12.0.jar
lib/main/io.quarkus.arc.arc-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-agroal-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-arc-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-caffeine-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-container-image-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-container-image-docker-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-container-image-openshift-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-core-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-credentials-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-datasource-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-datasource-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-fs-util-0.0.8.jar
lib/main/io.quarkus.quarkus-hibernate-orm-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-hibernate-orm-panache-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-hibernate-orm-panache-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-hibernate-validator-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-jackson-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-jacoco-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-jdbc-postgresql-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-jsonp-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-kubernetes-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-kubernetes-client-internal-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-micrometer-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-micrometer-registry-prometheus-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-minikube-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-mutiny-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-narayana-jta-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-netty-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-openshift-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-panache-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-panache-hibernate-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-resteasy-reactive-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-resteasy-reactive-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-resteasy-reactive-jackson-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-resteasy-reactive-jackson-common-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-security-runtime-spi-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-smallrye-context-propagation-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-smallrye-health-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-smallrye-openapi-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-swagger-ui-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-vertx-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-vertx-http-2.7.0.Final.jar
lib/main/io.quarkus.quarkus-vertx-http-dev-console-runtime-spi-2.7.0.Final.jar
lib/main/io.quarkus.resteasy.reactive.resteasy-reactive-2.7.0.Final.jar
lib/main/io.quarkus.resteasy.reactive.resteasy-reactive-common-2.7.0.Final.jar
lib/main/io.quarkus.resteasy.reactive.resteasy-reactive-jackson-2.7.0.Final.jar
lib/main/io.quarkus.resteasy.reactive.resteasy-reactive-vertx-2.7.0.Final.jar
lib/main/io.quarkus.security.quarkus-security-1.1.4.Final.jar
lib/main/io.smallrye.common.smallrye-common-annotation-1.8.0.jar
lib/main/io.smallrye.common.smallrye-common-classloader-1.8.0.jar
lib/main/io.smallrye.common.smallrye-common-constraint-1.8.0.jar
lib/main/io.smallrye.common.smallrye-common-expression-1.8.0.jar
lib/main/io.smallrye.common.smallrye-common-function-1.8.0.jar
lib/main/io.smallrye.config.smallrye-config-2.8.2.jar
lib/main/io.smallrye.config.smallrye-config-common-2.8.2.jar
lib/main/io.smallrye.config.smallrye-config-core-2.8.2.jar
lib/main/io.smallrye.config.smallrye-config-validator-2.8.2.jar
lib/main/io.smallrye.reactive.mutiny-1.3.1.jar
lib/main/io.smallrye.reactive.mutiny-smallrye-context-propagation-1.3.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-auth-common-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-bridge-common-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-core-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-runtime-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-mutiny-vertx-web-common-2.18.1.jar
lib/main/io.smallrye.reactive.smallrye-reactive-converter-api-2.6.0.jar
lib/main/io.smallrye.reactive.smallrye-reactive-converter-mutiny-2.6.0.jar
lib/main/io.smallrye.reactive.vertx-mutiny-generator-2.18.1.jar
lib/main/io.smallrye.smallrye-context-propagation-1.2.2.jar
lib/main/io.smallrye.smallrye-context-propagation-api-1.2.2.jar
lib/main/io.smallrye.smallrye-context-propagation-jta-1.2.2.jar
lib/main/io.smallrye.smallrye-context-propagation-storage-1.2.2.jar
lib/main/io.smallrye.smallrye-fault-tolerance-vertx-5.2.1.jar
lib/main/io.smallrye.smallrye-health-3.1.2.jar
lib/main/io.smallrye.smallrye-health-api-3.1.2.jar
lib/main/io.smallrye.smallrye-health-provided-checks-3.1.2.jar
lib/main/io.smallrye.smallrye-open-api-core-2.1.17.jar
lib/main/io.vertx.vertx-auth-common-4.2.4.jar
lib/main/io.vertx.vertx-bridge-common-4.2.4.jar
lib/main/io.vertx.vertx-codegen-4.2.4.jar
lib/main/io.vertx.vertx-core-4.2.4.jar
lib/main/io.vertx.vertx-web-4.2.4.jar
lib/main/io.vertx.vertx-web-common-4.2.4.jar
lib/main/jakarta.annotation.jakarta.annotation-api-1.3.5.jar
lib/main/jakarta.el.jakarta.el-api-3.0.3.jar
lib/main/jakarta.enterprise.jakarta.enterprise.cdi-api-2.0.2.jar
lib/main/jakarta.inject.jakarta.inject-api-1.0.jar
lib/main/jakarta.interceptor.jakarta.interceptor-api-1.2.5.jar
lib/main/jakarta.persistence.jakarta.persistence-api-2.2.3.jar
lib/main/jakarta.transaction.jakarta.transaction-api-1.3.3.jar
lib/main/jakarta.validation.jakarta.validation-api-2.0.2.jar
lib/main/net.bytebuddy.byte-buddy-1.12.7.jar
lib/main/org.eclipse.microprofile.config.microprofile-config-api-2.0.jar
lib/main/org.eclipse.microprofile.context-propagation.microprofile-context-propagation-api-1.2.jar
lib/main/org.eclipse.microprofile.health.microprofile-health-api-3.1.jar
lib/main/org.eclipse.microprofile.openapi.microprofile-openapi-api-2.0.1-RC1.jar
lib/main/org.glassfish.jakarta.el-3.0.4.jar
lib/main/org.glassfish.jakarta.json-1.1.6.jar
lib/main/org.glassfish.jaxb.jaxb-runtime-2.3.3-b02.jar
lib/main/org.glassfish.jaxb.txw2-2.3.3-b02.jar
lib/main/org.hdrhistogram.HdrHistogram-2.1.12.jar
lib/main/org.hibernate.common.hibernate-commons-annotations-5.1.2.Final.jar
lib/main/org.hibernate.hibernate-core-5.6.5.Final.jar
lib/main/org.hibernate.hibernate-graalvm-5.6.5.Final.jar
lib/main/org.hibernate.quarkus-local-cache-0.1.0.jar
lib/main/org.hibernate.validator.hibernate-validator-6.2.1.Final.jar
lib/main/org.jacoco.org.jacoco.agent-0.8.7-runtime.jar
lib/main/org.jacoco.org.jacoco.agent-0.8.7.jar
lib/main/org.jacoco.org.jacoco.core-0.8.7.jar
lib/main/org.jacoco.org.jacoco.report-0.8.7.jar
lib/main/org.jboss.jandex-2.4.2.Final.jar
lib/main/org.jboss.jboss-transaction-spi-7.6.0.Final.jar
lib/main/org.jboss.logging.commons-logging-jboss-logging-1.0.0.Final.jar
lib/main/org.jboss.logging.jboss-logging-annotations-2.2.1.Final.jar
lib/main/org.jboss.narayana.jta.narayana-jta-5.12.4.Final.jar
lib/main/org.jboss.narayana.jts.narayana-jts-integration-5.12.4.Final.jar
lib/main/org.jboss.slf4j.slf4j-jboss-logmanager-1.1.0.Final.jar
lib/main/org.jboss.spec.javax.resource.jboss-connector-api_1.7_spec-1.0.0.Final.jar
lib/main/org.jboss.spec.javax.ws.rs.jboss-jaxrs-api_2.1_spec-2.0.1.Final.jar
lib/main/org.jboss.spec.javax.xml.bind.jboss-jaxb-api_2.3_spec-2.0.0.Final.jar
lib/main/org.jboss.threads.jboss-threads-3.4.2.Final.jar
lib/main/org.latencyutils.LatencyUtils-2.0.3.jar
lib/main/org.mapstruct.mapstruct-1.4.2.Final.jar
lib/main/org.ow2.asm.asm-9.2.jar
lib/main/org.ow2.asm.asm-analysis-9.2.jar
lib/main/org.ow2.asm.asm-commons-9.2.jar
lib/main/org.ow2.asm.asm-tree-9.2.jar
lib/main/org.postgresql.postgresql-42.3.1.jar
lib/main/org.reactivestreams.reactive-streams-1.0.3.jar
lib/main/org.slf4j.slf4j-api-1.7.33.jar
lib/main/org.yaml.snakeyaml-1.30.jar
quarkus/
quarkus/generated-bytecode.jar
quarkus/quarkus-application.dat
quarkus/transformed-bytecode.jar
INFO Cleaning up source directory (/tmp/src)
STEP 9/9: CMD /usr/local/s2i/run
COMMIT temp.builder.openshift.io/eric-deandrea-dev/rest-villains-1:3d5e96cb
time="2022-02-17T13:31:50Z" level=warning msg="Adding metacopy option, configured globally"
Getting image source signatures
Copying blob sha256:e8228e50fe1805fb75cf22886a451b94982b6beea5c04ddcf24d1d2302200449
Copying blob sha256:4716779a2c027017c4bb38402df4224b49f7a65264b9072ee70b2e44de85b806
Copying blob sha256:2a8a3b359e47a83286a737d4df66063c4ac7cce5c5e9d6ba94d0872b79619ab3
Copying blob sha256:8fb287b280201d2c80541303b289fa0974643abb9012513997e17c14b81295c5
Copying config sha256:8b82ec9d2be54588dc5d184a45eaead9903806e1f42c6211e643785aa087ea35
Writing manifest to image destination
Storing signatures
--> 8b82ec9d2be
Successfully tagged temp.builder.openshift.io/eric-deandrea-dev/rest-villains-1:3d5e96cb
8b82ec9d2be54588dc5d184a45eaead9903806e1f42c6211e643785aa087ea35

Pushing image image-registry.openshift-image-registry.svc:5000/eric-deandrea-dev/rest-villains:1.0 ...
Getting image source signatures
Copying blob sha256:8fb287b280201d2c80541303b289fa0974643abb9012513997e17c14b81295c5
Copying blob sha256:8a53271b80c8bb6c008ffd7bc2a1c244014a8f5e576a4015dca62a7d6c9c9858
Copying blob sha256:131f1a26eef08c7274f7a115f461689819a4218e246ccacacc50ebcfd5e1c758
Copying blob sha256:16b78ed2e82205ceca54aa3a4205f029096a354d346b3bd2434a56b7f66e21c1
Copying config sha256:8b82ec9d2be54588dc5d184a45eaead9903806e1f42c6211e643785aa087ea35
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/eric-deandrea-dev/rest-villains@sha256:b9c8d2058f4acc39ae1153f739fd7b6d882d76cfa5722fcda7de45ac403ad626
Push successful

and here is the DeploymentConfig that gets generated/deployed (I removed some non-important noise from it). Take note of the command - it completely skips/ignores/bypasses the run-java.sh script that is included in the base image (which respects the JAVA_DEBUG flag):

kind: DeploymentConfig
apiVersion: apps.openshift.io/v1
metadata:
  annotations:
    app.openshift.io/connects-to: villains-db
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/build-timestamp: '2022-02-17 - 13:30:45 +0000'
    app.quarkus.io/commit-id: c7f99c3016ef468c142d56d5178fd79754526c49
    prometheus.io/path: /q/metrics
    prometheus.io/port: '8084'
    prometheus.io/scheme: http
    prometheus.io/scrape: 'true'
  name: rest-villains
  namespace: eric-deandrea-dev
  labels:
    app: rest-villains
    app.kubernetes.io/name: rest-villains
    app.kubernetes.io/part-of: villains-service
    app.kubernetes.io/version: '1.0'
    app.openshift.io/runtime: quarkus
    application: villains-service
    system: quarkus-super-heroes
spec:
  strategy:
    type: Rolling
    rollingParams:
      updatePeriodSeconds: 1
      intervalSeconds: 1
      timeoutSeconds: 600
      maxUnavailable: 25%
      maxSurge: 25%
    resources: {}
    activeDeadlineSeconds: 21600
  triggers:
    - type: ImageChange
      imageChangeParams:
        automatic: true
        containerNames:
          - rest-villains
        from:
          kind: ImageStreamTag
          namespace: eric-deandrea-dev
          name: 'rest-villains:1.0'
        lastTriggeredImage: >-
          image-registry.openshift-image-registry.svc:5000/eric-deandrea-dev/rest-villains@sha256:b9c8d2058f4acc39ae1153f739fd7b6d882d76cfa5722fcda7de45ac403ad626
  replicas: 1
  revisionHistoryLimit: 10
  test: false
  selector:
    app.kubernetes.io/name: rest-villains
    app.kubernetes.io/part-of: villains-service
    app.kubernetes.io/version: '1.0'
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rest-villains
        app.kubernetes.io/name: rest-villains
        app.kubernetes.io/part-of: villains-service
        app.kubernetes.io/version: '1.0'
        app.openshift.io/runtime: quarkus
        application: villains-service
        system: quarkus-super-heroes
      annotations:
        app.openshift.io/connects-to: villains-db
        app.openshift.io/vcs-url: <<unknown>>
        app.quarkus.io/build-timestamp: '2022-02-17 - 13:30:45 +0000'
        app.quarkus.io/commit-id: c7f99c3016ef468c142d56d5178fd79754526c49
        prometheus.io/path: /q/metrics
        prometheus.io/port: '8084'
        prometheus.io/scheme: http
        prometheus.io/scrape: 'true'
    spec:
      containers:
        - resources:
            limits:
              memory: 768Mi
            requests:
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /q/health/ready
              port: 8084
              scheme: HTTP
            timeoutSeconds: 10
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: rest-villains
          command:
            - java
            - '-Dquarkus.http.host=0.0.0.0'
            - '-Djava.util.logging.manager=org.jboss.logmanager.LogManager'
            - '-jar'
            - /deployments/quarkus-run.jar
          livenessProbe:
            httpGet:
              path: /q/health/live
              port: 8084
              scheme: HTTP
            timeoutSeconds: 10
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: JAVA_DEBUG
              value: 'true'
            - name: JAVA_APP_JAR
              value: /deployments/quarkus-run.jar
          ports:
            - name: http
              containerPort: 8084
              protocol: TCP
          imagePullPolicy: IfNotPresent
          terminationMessagePolicy: File
          envFrom:
            - configMapRef:
                name: rest-villains-config
          image: >-
            image-registry.openshift-image-registry.svc:5000/eric-deandrea-dev/rest-villains@sha256:b9c8d2058f4acc39ae1153f739fd7b6d882d76cfa5722fcda7de45ac403ad626
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
geoand commented 2 years ago

But the Dockerfile.jvm that was used to create the image (https://github.com/quarkusio/quarkus-super-heroes/blob/main/rest-villains/src/main/docker/Dockerfile.jvm) came from code.quarkus.io. That Dockerfile defines its own ENTRYPOINT which does not include any debug "stuff".

You never mentioned you were using the Dockerfile...

It seems the image used (ubi8/openjdk-11-runtime) doesn't have any kind of cli tools installed (no ps nor lsof), so I'm not sure how to see what the actual java process looks like, although looking at the ENTRYPOINT I would suspect it is simply java -jar.

Does it comes with jcmd in the directory where java is found? If so, you can do things like:

jcmd <pid> VM.flags
jcmd <pid> VM.command_line
edeandrea commented 2 years ago

You never mentioned you were using the Dockerfile...

That's what I tried first, just deploying the image that was built with that Dockerfile. When that didn't work I switched back to the s2i binary build, deploying from my local machine.

Does it comes with jcmd in the directory where java is found? If so, you can do things like:

It does have jcmd but I don't know what the pid is....

If you look at the DeplopymentConfig that was generated by the openshift extension (see the end of my previous comment), take note of the command - it completely skips/ignores/bypasses the run-java.sh script that is included in the base image (which respects the JAVA_DEBUG flag)

edeandrea commented 2 years ago

Note - if I manually go into the DeploymentConfig and alter the command by adding -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 and trigger a rollout, it then works.

Doing -Dquarkus.openshift.jvm-arguments=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 when running the ./mvnw package ... does not work though. I would have expected it to.

geoand commented 2 years ago

It does have jcmd but I don't know what the pid is....

jps should give it you

geoand commented 2 years ago

One question, where does quarkus.openshift.jvm-arguments come from? Is it mentioned somewhere in the docs? Have you tried quarkus.openshift.arguments=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 instead?

edeandrea commented 2 years ago

In fact, if I run ./mvnw clean package -DskipTests -Dquarkus.kubernetes.deployment-target=openshift -Dquarkus.container-image.builder=openshift -Dquarkus.openshift.resources.limits.memory=768Mi -Dquarkus.openshift.resources.requests.memory=256Mi -Dquarkus.kubernetes.deploy=true -Dquarkus.openshift.env.vars.JAVA_DEBUG=true -Dquarkus.container-image.build=true -Dquarkus.openshift.jvm-arguments="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" the Pod fails to even start. It gives a message

ERROR: JDWP Non-server transport dt_socket must have a connection address specified through the 'address=' option
ERROR: JDWP invalid option: -agentlib:jdwp=transport=dt_socket

If I look. at the command in the DeploymentConfig that was deployed, I see

command:
  - java 
  - '-agentlib:jdwp=transport=dt_socket'
  - server=y
  - suspend=n
  - 'address=*:5005'
  - '-jar'
  - /deployments/quarkus-run.jar

It looks like because the argument has commas in it, its getting separated into separate arguments. If I (manually) change the command to be

command:
  - java 
  - '-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005'
  - '-jar'
  - /deployments/quarkus-run.jar

then it works

One question, where does quarkus.openshift.jvm-arguments come from?

https://quarkus.io/guides/all-config#quarkus-container-image-openshift_quarkus.openshift.jvm-arguments

Have you tried quarkus.openshift.arguments=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 instead?

Has no effect whatsoever

geoand commented 2 years ago

@iocanel when I set quarkus.openshift.arguments=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005, then 

        args:
            - -agentlib:jdwp=transport=dt_socket
            - server=y
            - suspend=n
            - address=*:5005

is added to the container instead of appended to the command (the code that is responsible is in KubernetesCommonHelder#createArgsDecorator). That seems incorrect, no? If it's not, I can fix it

edeandrea commented 2 years ago

I guess in addition was not using the provided run script (which handles the presence of the JAVA_DEBUG environment variable) a conscious decision by Quarkus? It seems when the openshift extension generates the DeploymentConfig it is overriding the command from the base image.

Was that a conscious choice? If so, why?

geoand commented 2 years ago

Was that a conscious choice? If so, why?

I am not privy to this, it's up to @iocanel to answer

edeandrea commented 2 years ago

This looks like its been resolved now.

The DeploymentConfig no longer specified the command.

The latest code.quarkus.io also generates a different Dockerfile - one that's based on the OpenJDK image and does not override the ENTRYPOINT.

geoand commented 2 years ago

Good to know!

edeandrea commented 2 years ago

Actually I'm going to re-open this as only part of the issue seems to have been resolved. Now that theDockerfile.jvm references the OpenJDK image and doesn't override the ENTRYPOINT, deploying an image I build using the docker extension and setting JAVA_DEBUG=true works.

What seems broken is if I am generating resources and/or deploying via the openshift extension.

In https://github.com/quarkusio/quarkus-super-heroes/tree/main/rest-villains If I run

./mvnw clean package -DskipTests \
  -Dquarkus.kubernetes.deployment-target=openshift \
  -Dquarkus.container-image.builder=openshift \
  -Dquarkus.openshift.resources.limits.memory=768Mi \
  -Dquarkus.openshift.resources.requests.memory=256Mi \
  -Dquarkus.openshift.env.vars.JAVA_DEBUG=true

and then inspect the resulting target/kubernetes/openshift.yml I see that the app's DeploymentConfig looks like the following and the command is overridden.

In this scenario it is almost impossible to turn on remote debug unless I then go in and manually tweak the resulting DeploymentConfig. The DeploymentConfig already references the image (which is already set up for remote debug), so why the need to also specify a command?

Also, the imagePullPolicy doesn't seem to be respected. The docs say that the default value for quarkus.openshift.image-pull-policy (or quarkus.kubernetes.image-pull-policy) is Always, which doesn't seem to be the case, as below it shows IfNotPresent.

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    app.openshift.io/connects-to: villains-db
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/commit-id: 87ecfd80d2c94edf67f93cae5c8a424e49a9ebc7
    app.quarkus.io/build-timestamp: 2022-02-23 - 13:26:04 +0000
    prometheus.io/scrape: "true"
    prometheus.io/path: /q/metrics
    prometheus.io/port: "8084"
    prometheus.io/scheme: http
  labels:
    app.kubernetes.io/name: rest-villains
    app.kubernetes.io/part-of: villains-service
    app.kubernetes.io/version: "1.0"
    app: rest-villains
    application: villains-service
    system: quarkus-super-heroes
    app.openshift.io/runtime: quarkus
  name: rest-villains
spec:
  replicas: 1
  selector:
    app.kubernetes.io/part-of: villains-service
    app.kubernetes.io/name: rest-villains
    app.kubernetes.io/version: "1.0"
  template:
    metadata:
      annotations:
        app.openshift.io/connects-to: villains-db
        app.openshift.io/vcs-url: <<unknown>>
        app.quarkus.io/commit-id: 87ecfd80d2c94edf67f93cae5c8a424e49a9ebc7
        app.quarkus.io/build-timestamp: 2022-02-23 - 13:26:04 +0000
        prometheus.io/scrape: "true"
        prometheus.io/path: /q/metrics
        prometheus.io/port: "8084"
        prometheus.io/scheme: http
      labels:
        app.kubernetes.io/name: rest-villains
        app.kubernetes.io/version: "1.0"
        app: rest-villains
        application: villains-service
        system: quarkus-super-heroes
        app.openshift.io/runtime: quarkus
        app.kubernetes.io/part-of: villains-service
    spec:
      containers:
        - command:
            - java
            - -Dquarkus.http.host=0.0.0.0
            - -Djava.util.logging.manager=org.jboss.logmanager.LogManager
            - -jar
            - /deployments/quarkus-run.jar
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: JAVA_DEBUG
              value: "true"
            - name: JAVA_APP_JAR
              value: /deployments/quarkus-run.jar
          envFrom:
            - configMapRef:
                name: rest-villains-config
            - secretRef:
                name: rest-villains-config-creds
          image: quay.io/quarkus-super-heroes/rest-villains:1.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/live
              port: 8084
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
          name: rest-villains
          ports:
            - containerPort: 8084
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/ready
              port: 8084
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
          resources:
            limits:
              memory: 768Mi
            requests:
              memory: 256Mi
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - rest-villains
        from:
          kind: ImageStreamTag
          name: rest-villains:1.0
      type: ImageChange
florentgarin commented 2 years ago

Default port env variable seems not correct. On my side I had to specify JAVA_DEBUG_PORT=*:5005 instead of default 5005 to make it works.

Sgitario commented 2 years ago

@florentgarin do you mean that using the configuration in https://github.com/gsmet/quarkus/commit/67505dc3d5c32bcc64e3c92121c5bedccae4e3ff#diff-352a260cfb79854020efe849e44cedb10355aa4bd7b5eed1db676b77d8870a86R14, didn't work for you? For me, it's working fine tho. But it's also working fine with "*:5005".

florentgarin commented 2 years ago

No I mean that if you simply use the generated Dockerfile.jvm, build it with the docker command exactly as mentioned in the comments and run it (as explained, plain basic Docker style). Then you will not be able to remote debug your container without -e JAVA_DEBUG_PORT=*:5005