[SOLVED]: The Application Agent isn't working on WildFly 20

[Utk98]

The Application Agent isn't working on WildFly 20

Description

I had followed instructions given in documentation , but not able to connect with drill4j

Details

1. Attach full log files of your Application Under Test. Make sure it includes Drill4J Agent ASCII logo followed by version

   These are the logs -->

   JBoss Bootstrap Environment

   JBOSS_HOME: /opt/wildfly20

   JAVA: /opt/jdk1.8/bin/java

   JAVA_OPTS: -server -Xms2048m -Xmx2048m -XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n

=========================================================================

Picked up JAVA_TOOL_OPTIONS: -agentpath:/data/agent/libdrill_agent.so

---

| "\U | "\ u |"| |"| | ||"| U |"| u
/| | | || |) |/ |"| U | | u U | | u | || | | |/
U| || || < | | | |/ | |/ | | | || |,-. |____/ u|| _\ U/| |\u || || /|_|\ _/-(/
||| // \.-,|_|,-.// \ // \ u|||u //
(_)) () ()_)-' '-(/(")(")(")("_) ()) (__)
Java Agent (v0.9.1) 2024-09-09 07:03:09.000 INFO [com.epam.drill.agent.Agent] agentOnLoad: Java Agent has been loaded. Pid is: 167 2024-09-09 12:33:10.285 INFO [com.epam.drill.test2code.Test2Code] Scanning classes, package prefixes: [com/digite/app/platform]... Listening for transport dt_socket at address: 8787 2024-09-09 12:33:12.361 INFO [com.epam.drill.test2code.Test2Code] Scanned 0 classes with 0 methods 12:33:13,384 INFO [org.jboss.modules] (main) JBoss Modules version 1.10.1.Final 12:33:14,236 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final 12:33:14,283 INFO [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final 12:33:14,950 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 20.0.1.Final (WildFly Core 12.0.3.Final) starting 12:33:16,988 WARN [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0456: System property "jboss.node.name" is already set in the section of the configuration file. The value set in the command line will be overridden by that value. 12:33:19,499 INFO [org.wildfly.security] (ServerService Thread Pool -- 26) ELY00001: WildFly Elytron version 1.12.1.Final 12:33:22,893 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 12:33:23,005 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 24) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation. 12:33:26,923 INFO [org.jboss.as.repository] (ServerService Thread Pool -- 30) WFLYDR0001: Content added at location /opt/wildfly20/standalone/data/content/14/9f7993fc63c8ea5c8f2d4796539b7c7daa84fb/content 12:33:27,013 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http) 12:33:27,070 INFO [org.xnio] (MSC service thread 1-7) XNIO version 3.8.1.Final 12:33:27,094 INFO [org.xnio.nio] (MSC service thread 1-7) XNIO NIO Implementation Version 3.8.1.Final 12:33:27,489 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 44) WFLYCLINF0001: Activating Infinispan subsystem.

2. Describe your tech stack (for both application and tests)
   • OS:
   • Tech stack: java

Screenshots

Additional Notes

docker-compose-app.yml

version: '3.8' services:

skapp: image: 014936247795.dkr.ecr.us-east-1.amazonaws.com/digite/sk-wildfly20-app-server:SK_10.17_2637 container_name: skapp ports:

• "8080:80" environment:
• TZ=Asia/Kolkata
• DBNAME=swiftkandan
• DBHOST=skdb
• DBPORT=3306
• DBUSER=root1
• DBPASSWORD=pass@000
• RAM=
• CLUSTER_MODE_ON=No
• IS_UPGRADE_DB=No
• "ACCESS_CONTROL_ALLOW_ORIGIN=http://localhost:${REALWORLD_FRONTEND_PORT}"
• NODE=node1
• "JAVA_TOOL_OPTIONS=-agentpath:/data/agent/libdrill_agent.so"
• "DRILL_API_URL=${DRILL_ADMIN_ADDRESS}"
• "DRILL_API_KEY=${DRILL_API_KEY}"
• "DRILL_APP_ID=${DRILL_API_AGENT_ID}"
• "DRILL_GROUP_ID=${DRILL_GROUP_ID}"
• "DRILL_BUILD_VERSION=0.1.0"
• "DRILL_PACKAGE_PREFIXES=com/digite/app/platform" volumes:
• agent-files:/data entrypoint: /opt/entrypoint.sh networks:
• new_default extra_hosts:
• "host.docker.internal:host-gateway"

networks: new_default:

volumes: agent-files:

docker-compose-admin.yml version: '3.8'

services:

drill-ui: image: ghcr.io/drill4j/drill4j-ui:${DRILL_UI_VERSION} environment: API_HOST: host.docker.internal API_PORT: 8090 extra_hosts:

• "host.docker.internal:host-gateway" ports:
• "8091:8080" networks:

• new_default

  drill-admin: image: ghcr.io/drill4j/admin:${DRILL_ADMIN_BACKEND_VERSION} environment:

• DRILL_METRICS_UI_BASE_URL=${DRILL_UI_BASE_URL}
• DRILL_DB_PORT=5432
• DRILL_AGENTS_SOCKET_TIMEOUT=6000
• DRILL_DB_HOST=${POSTGRES_HOST}
• DRILL_DB_NAME=${POSTGRES_DB}
• DRILL_DB_USER_NAME=${POSTGRES_USER}
• DRILL_DB_PASSWORD=${POSTGRES_PASSWORD}
• DRILL_DB_MAX_POOL_SIZE=${DRILL_ADMIN_BACKEND_DB_MAX_POOL_SIZE}
• LOG_LEVEL=info healthcheck: test: [ "CMD", "curl", "http://localhost:8090" ] interval: 10s timeout: 3s retries: 30 ports:
• 8090:8090 networks:

• new_default depends_on: postgres: condition: service_healthy

  postgres: image: postgres:14.1 ports:

• '5432:5432' environment: POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} POSTGRES_USER: ${POSTGRES_USER} POSTGRES_DB: ${POSTGRES_DB} volumes:
• ./db-init:/docker-entrypoint-initdb.d
• drill-data-pg:/var/lib/postgresql/data healthcheck: test: ['CMD-SHELL', 'pg_isready -U postgres'] interval: 10s timeout: 5s retries: 5 networks:

• new_default command: ["postgres", "-c", "log_statement=all", "-c", "log_min_duration_statement=0"]

  devtools-proxy: image: drill4j/devtools-proxy:${DRILL_DEVTOOLS_PROXY_VERSION} ports:

• "8093:8080" environment: API_HOST: host.docker.internal API_PORT: 8090
  DEBUG: 'drill:*' DEBUG_COLORS: 'true' FORCE_COLOR: '3' DEBUG_LOG_LEVEL: '4' extra_hosts:
• "host.docker.internal:host-gateway" networks:

• new_default

  metabase: image: metabase/metabase@sha256:96416c11371d400aecc2073b8d0e8335cc7c413b236ef724e51dda095b2580ea container_name: metabase hostname: metabase volumes:

• /dev/urandom:/dev/random:ro ports:
• 8095:3000 environment: MB_DB_TYPE: postgres MB_DB_HOST: postgres MB_DB_PORT: 5432 MB_DB_USER: ${POSTGRES_USER} MB_DB_PASS: ${POSTGRES_PASSWORD} MB_DB_DBNAME: ${POSTGRES_DB} networks:

• new_default healthcheck: test: curl --fail -I http://localhost:3000/api/health || exit 1 interval: 15s timeout: 5s retries: 5

  agent-files: image: drill4j/java-agent:0.8.0-38 environment:

• AGENT_VERSION=${JAVA_AGENT_VERSION} volumes:
• agent-files:/data healthcheck: test: ["CMD", "sh", "-c", "test -f /data/agent/libdrill_agent.so || exit 1"] interval: 30s retries: 6 start_period: 10s timeout: 10s networks:
• new_default

volumes: drill-data-pg: agent-files:

networks: new_default:

.env

REALWORLD_API_ADDRESS=/api REALWORLD_FRONTEND_PORT=8080 REALWORLD_API_PORT=8081 REALWORLD_FRONTEND_2_PORT=8082 REALWORLD_API_2_PORT=8083

DRILL_ADMIN_BACKEND_VERSION=0.9.0 DRILL_UI_VERSION=0.1.1 DRILL_DEVTOOLS_PROXY_VERSION=0.1.0 DRILL_API_KEY=4_11100cc748abe3c15c5644ea638da47155d15af6eacb62af9c1cfc744e3a0c52 DRILL_ADMIN_BACKEND_DB_MAX_POOL_SIZE=50 DRILL_UI_BASE_URL=http://localhost:8095

JAVA_AGENT_VERSION=0.9.1 DRILL_API_AGENT_ID=skapp DRILL_GROUP_ID=skap DRILL_ADMIN_ADDRESS=http://host.docker.internal:8090/api

POSTGRES_HOST=postgres POSTGRES_DB=postgres POSTGRES_USER=postgres POSTGRES_PASSWORD=mysecretpassword

docker-compose-DB.yml

version: '3.8' services: skdb: image: 014936247795.dkr.ecr.us-east-1.amazonaws.com/digite/mysql:8.0.22 container_name: skdb environment:

• TZ=Asia/Kolkata

• MYSQL_DATABASE=swiftkandan

• MYSQL_USER=root1

• MYSQL_PASSWORD=pass@000

• MYSQL_ROOT_PASSWORD=pass@000

  volumes:

• ./data:/var/lib/mysql

• ./backup:/app/dataDump

• ./conf:/opt

  ports:

• 3306:3306

  restart: always

  networks: new_default:

networks: new_default:

build.gradle

plugins { id 'java' id("com.epam.drill.integration.cicd") version "0.1.1" }

group 'com.epam' version '1.0-SNAPSHOT'

allprojects {

repositories {
    mavenCentral()
}

apply plugin: 'java'
apply plugin: 'com.epam.drill.integration.cicd'

dependencies {
    implementation 'io.rest-assured:rest-assured:4.3.3'
    implementation 'org.junit.jupiter:junit-jupiter-api:5.6.0'
    runtimeOnly 'org.junit.jupiter:junit-jupiter-engine'
    implementation 'com.codeborne:selenide:5.19.0'
    implementation 'org.slf4j:slf4j-simple:1.7.32'
}

drill {
    groupId = "skap"
    appId = "skapp"
    packagePrefixes = ["com/digite/app/platform"]
    enableTestAgent {
        version="0.23.1"
        logLevel = "TRACE"
        logFile = "./drill-log.log"
        additionalParams = [
            "devToolsProxyAddress": "http://localhost:8093",
            "withJsCoverage": "false",
            "devtoolsAddressReplaceLocalhost": "host.docker.internal",
        ]
    }
}

}

[RomanDavlyatshin]

1. Please use "attach files" function to submit files, including logs.

   When you are submitting logs in messages they are incomplete, and almost of no use.

2. The issue you are describing is likely related to the way Drill4J Agent scans .war files. My guess is that scanning process completes before your app loads Try setting DRILL_CLASS_SCAN_DELAY=10000 env variable. It delays scanning process a bit, that could help. If 10000 (10s) does not work try setting it to larger value (20000, 30000)

[Utk98]

@RomanDavlyatshin problem is that using drill4j in my application (docker-compose-app), my application is not working and by commenting these lines , it is working . docker-compose-appp.docx

[Utk98]

logs.docx these are the logs

[Utk98]

build.gradle.docx doxker-compose-admin.docx

[RomanDavlyatshin]

@RomanDavlyatshin problem is that using drill4j in my application (docker-compose-app), my application is not working and by commenting these lines , it is working . docker-compose-appp.docx

From the messages you have provided I am unable to understand to which line you are referring to.

Looking at your docker-compose-appp.docx and build.gradle.docx I have noticed two things:

1. You copy-pasted the "ACCESS_CONTROL_ALLOW_ORIGINS" and other env variables from realworld demo. "ACCESS_CONTROL_ALLOW_ORIGIN" is not related to Drill4J agent is simply a env variable for realworld app. You don't have to copy it.

2. I don't see environment variable values for the following:

     - "DRILL_API_URL=${DRILL_ADMIN_ADDRESS}"
     - "DRILL_API_KEY=${DRILL_API_KEY}"
     - "DRILL_APP_ID=${DRILL_API_AGENT_ID}"
     - "DRILL_GROUP_ID=${DRILL_GROUP_ID}"

Possible solution steps:

1. Please refer to documentation in order to setup java agent for your application instead. Do not use realworld demo as instruction.

   Here are relevant docs: how to set env variables and agent parameters reference

2. Set the "DRILL_LOG_LEVEL=TRACE" for your application and provide the complete log files

3. Provide the complete log file for Drill4J Admin service

P.S. You can send .yml, .gradle, .log files as ordinary text files saved in any editor, such as Notepad or VSCode. There is no need to save these in .docx - it makes harder to open and read these.

[Utk98]

I think you didn't understand the problem , problem is that sk app is not working when integration with drill4j , and when i am removing drill4j lines from docker-compose-app.yml- environment lines , it is working perfectly fine , after delay also same problem is there , i have attached the updated logs , and docker -compose-app sklogs.txt docker-copose-app.txt

[RomanDavlyatshin]

According to your application logs you are not setting DRILL_CLASS_SCAN_DELAY:

1. Parameters log does not have classScanDelay (starting with initializeJvm: Found input parameters) :

   ____    ____                 _       _          _  _                _      
   |  _"\U |  _"\ u     ___     |"|     |"|        | ||"|            U |"| u   
   /| | | |\| |_) |/    |_"_|  U | | u U | | u      | || |_          _ \| |/    
   U| |_| |\|  _ <       | |    \| |/__ \| |/__     |__   _|        | |_| |_,-. 
   |____/ u|_| \_\    U/| |\u   |_____| |_____|      /|_|\          \___/-(_/  
   |||_   //   \\_.-,_|___|_,-.//  \\  //  \\      u_|||_u          _//       
   (__)_) (__)  (__)\_)-' '-(_/(_")("_)(_")("_)     (__)__)         (__)  
   Java Agent (v0.9.1)
   2024-09-10 09:57:39.000 INFO  [com.epam.drill.agent.Agent] agentOnLoad: Java Agent has been loaded. Pid is: 166
   2024-09-10 15:27:39.274 DEBUG [com.epam.drill.agent.configuration.Configuration] initializeJvm: Found input parameters: 
   // I have formatted this line for easier reading:
   {instanceId=5cba9039-bdf9-48d7-adc4-7946ee6f43f6,
   logLevel=TRACE,
   drillInstallationDir=/data/agent,
   groupId=skap,
   apiUrl=http://host.docker.internal:8090/api,
   packagePrefixes=com/digite/app/platform,
   appId=skapp,
   buildVersion=0.1.1,
   apiKey=5_8d989df62cd33ca4972e703a05cf1a134eca7543de8b20349acf0b2b959e9e48}
   // as you can see, there is no classScanDelay parameter present

2. According to timestamps class scan starts almost immediately:

   // this should not start _this_ early (same second as parameter log above).
   // If you'd had DRILL_CLASS_SCAN_DELAY set, scan process would've started after the delay
   2024-09-10 15:27:39.541 INFO  [com.epam.drill.test2code.Test2Code] Scanning classes, package prefixes: [com/digite/app/platform]... 

If for some reason environment variable is not working for you, you can try setting it through agent options string:

      - "JAVA_TOOL_OPTIONS=-agentpath:/data/agent/libdrill_agent.so=classScanDelay=30000"

[Utk98]

@RomanDavlyatshin logs.txt still facing same problem , sk app is not opening

[RomanDavlyatshin]

@RomanDavlyatshin logs.txt still facing same problem , sk app is not opening

The logs indicate that our class scanning algorithm is having difficulty finding your application packages (com/digite/app/platform).

This issue may be related to the use of JBoss with .ear files, which is a complex setup (in terms of app structure) and one we haven’t extensively tested.

To assist you better, please provide a minimal reproducible example. This will help us address the issue more effectively.

EDIT: possible workaround is to increase classScanDelay even more (lets say 120000 that is equal to 2 minutes) and wait for 5 - 10 minutes just to see if its going to work. If you try these, please report back whether it worked or not

[Utk98]

@RomanDavlyatshin . I had tried for 30000 ms and 1200000 ms , but it is not working. Matter of concern is why it is stopping skapp to run . I had attach log files . logss.txt logss1.txt

[Utk98]

In 0.8.5 version , drill4j not stopping skapp to run , but it is not able to give coverage. @RomanDavlyatshin

[RomanDavlyatshin]

@RomanDavlyatshin . I had tried for 30000 ms and 1200000 ms , but it is not working. Matter of concern is why it is stopping skapp to run . I had attach log files . logss.txt logss1.txt

The value I suggested above is 120000 (2 minutes). But according to your logs you added an extra 0, so instead of 2 minutes you have instructed Drill4J agent to wait for 20 minutes. Other log indicates that you set delay to 300000 which amounts to 30 minutes. In both cases, because of these big delay values, class scanning process did not started to work when you captured logs.

To make it more clear:

1. The workaround with delay is based on the idea that your application classes aren't loaded immediately and Drill4J scanning algorithm completes work before they become available. Based on this 30000 (30 seconds) may not be enough for your application classes to load, depending on its size. So I, once again, suggest to set value that is moderately bigger - e.g. 120000ms. After waiting for around 5 minutes try capturing logs.

2. If the issue is caused by something else - e.g. Drill4J is completely unable to locate your application packages in .ear - the delay workaround won't work. That is the reason I have asked for minimal reproducible example with any basic JBoss-based application. We could do it ourselves, but it will take time - so you submitting it will speed up the process.

I cannot answer on why your application does not work with Drill4J agent based on available information.

In 0.8.5 version , drill4j not stopping skapp to run , but it is not able to give coverage

We don't provide support for 0.8.5, as its heavily outdated.

[Utk98]

1.I have tried for 120000(2 min) also but facing same problem .(logs attached) logss2.txt

2. You may conduct your testing using the following resource: https://hub.docker.com/r/bitnami/wildfly/ . Unfortunately, we are unable to provide detailed information about our product at this time. However, if feasible, we would be happy to connect and test the features and problems of Drill4J via a Google Meet session.

[Utk98]

With the latest version of drill4j and by following the updated documentaion for integrating with wildfly , I faced problem that matrices is not showing in metabase . .I have attached the logs for

docker-compose-admin logs adminlogs.txt

docker-compose-app logs logs.txt

docker-compose-app file

docker-compose-app (1).txt

stanalone.conf

standalone.txt

[RomanDavlyatshin]

1. According to your application logs and configuration you actually supply Java process with Drill4J Application Agent twice. You set -agenpath both in standalone.sh and in docker-compose-app.yml. This is not correct and can lead to issues. Please decide on the one way or the other.

2. Make sure scanClassPath=/opt/wildfly20/standalone/deployments/digite.ear points to the correct location and it is accessible from the directory you are launching Wildfly from. E.g. perform ls /opt/wildfly20/standalone/deployments before launching standalone.sh to ensure you see digite.ear in the output

3. To confirm application agent has found your classes look for ClassPathScanner: scanning class entry: com/digite/app/platform/ log entries (there should be one entry for each of your application classes in com/digite/app/platform/

[Utk98]

I had noted the above changes, and implemented them in my code , i discovered some things --> metabase has taken some time to fetch and timer also started in tab (photos attached)

but metrices still not showned --> logs attached skapp_logs.txt

[RomanDavlyatshin]

1. The first issue I see - you did not set the packagePrefixes parameter.

   • it is empty according to JAVA_OPTS log - ...appId=skapp,packagePrefixes=,buildVersion=0.1.1

   • it is also confirmed by 2024-09-30 11:26:27.450 INFO [com.epam.drill.test2code.Test2Code] Scanning classes, package prefixes: []... log entry.

   • "Still waiting" on metabase dashboards indicates Application Agent submitted all packages (including 3rd-party packages from dependencies) to Drill4J Admin Backend because packagePrefixes were empty. It's likely amounts to hundreds of thousands of methods (100k, 200k, 300k) and takes a lot of time to load.

   • Solution:

     1. Make sure to specify packagePrefixes according to Application Agent reference.
     2. Reset the DB to cleanup the unnecessary 3rd-party packages (we will provide API delete method in the near future)
     3. Start application again

2. Please also set logLevel=TRACE parameter to application agent to ensure all the necessary debug information is written to logs. After that try to restart the application and capture logs again.

3. Does your application work as expected when Application Agent is enabled?

[Utk98]

1.Yes Roman ,after using the latest updates application is working fine when agent is enabled 2.I had updated the package prefixes and with and this time I tried with new version(0.1.2) , this is the console view-- I had attached the logs-- logs2.txt,still metrices doesn't appear

[RomanDavlyatshin]

I believe you set your package prefixes to the wrong string for your project. You set com/digite/app/platform but your packages are com/digite/app and com/digite/platform there is no /app/platform

Solution:

1. To capture all digite packages set packagePrefixes=com/digite. This will include both com/digite/app and com/digite/platform OR
2. Specify packages separately using ; as separator packagePrefixes=com/digite/app;com/digite/platform
   • That is handy if there are some packages you don't want from com/digite

The reason I think so is:

1. Your set packages prefixes filter to

   2024-09-30 22:03:40.002 INFO  [com.epam.drill.test2code.Test2Code] Scanning classes, package prefixes: 
   [com/digite/app/platform]... 

2. ClassPathScanner logged 769 entries for ClassPathScanner: scanning class entry: com/digite/platform/ (you can check it by performing CTRL+F in log file) - there is no /app before /platform

3. This log entry indicates Applciation Agent couldn't find classes:

   22:04:19,646 INFO  [stdout] (Thread-0) 2024-09-30 22:04:19.646 INFO  [com.epam.drill.test2code.Test2Code] 
   Scanned 0 classes with 0 methods

4. There are also 565 classes for ClassPathScanner: scanning class entry: com/digite/app/ (including /com/digite/app/kanban), for example:

   2024-09-30 22:03:46.989 TRACE [com.epam.drill.test2code.classloading.ClassPathScanner] ClassPathScanner: scanning class 
   entry: com/digite/app/WebVersionInfo.class
   2024-09-30 22:03:46.991 TRACE [com.epam.drill.test2code.classloading.ClassPathScanner] ClassPathScanner: scanning class 
   entry: com/digite/app/kanban/board/web/action/BTTest.class

⚠️⚠️⚠️Another thing I've noticed - you submitted logs for buildVersion=0.1.1⚠️⚠️⚠️, but you open Metabase dashboard for buildVersion=0.1.2. Make sure to open dashboard for the version you have specified in Application Agent parameters. Also as I've stated above, you better cleanup the Drill4J database before submitting new data, to ensure there are no extra packages from 3rd party libraries left from the previous launch with empty packagePrefixes

[Utk98]

@RomanDavlyatshin When i try to explore the probes and coverage covered it is showing 'Drill-through doesn’t work on SQL questions.'.Help regarding this

[RomanDavlyatshin]

You have to use it a bit differently. Packages table always displays all your packages and is not affected by filters. To see the detailed info:

1. Click on the package name in the corresponding column
   • This will update the "package" filter at the top of the page
2. Then scroll down to classes table - it should now display classes only for this particular package
3. Now click on the class name in the classes table. It sets "Class" filter
4. Then scroll down to methods table to see the methods for the particular class

To reset package or class filter enter asterisk * as the filter value

This is described in our docs too - refer to this section https://drill4j.github.io/docs/getting-started/metabase#code-coverage-dashboard to see the step-by-step instruction with screenshots

I'm closing the issue since the resolution is reached.