hawtio3.x not able to connect remote jolokia

[debasishdx08]

Hi,

I tried connecting jolokia application deployed on another server with the hawtio3.0.1.war and it loops infinitely.

The main reason for this is that while discovering local mbeans, the response returned is has the following format: domains:{mbeans...} however, when the response is fetched from external jolokia agent the response returned has the following format {mbeans}

Hence the unwindListResponse method fails to return any response for the remote response and the UI loops infinitely.

Is there any specific configuration required or is there any issue with remote jolokia connectivity for this version ?

[tadayosi]

Hi,

We need a bit more info for troubleshooting.

• Which web container are you using?
• What is the external JVM that you are trying to connect to?
• Which version of Jolokia are you using for the external JVM?
• Can you provide us with more concrete response of domains:{mbeans...}?

[grgrzybek]

I agree with @tadayosi, we need more information...

I tried connecting jolokia application deployed on another server with the hawtio3.0.1.war and it loops infinitely.

Please be more specific - I read it several times, but I still don't know whether the WAR is the application you're connecting from or connecting to. And this another server is what kind of server? What do you mean by "loop" - a Java StackOverflowException? Or blank page (which may seem like waiting for something)...

Can you attach the JSON fragment with domains:{mbeans...}?

[debasishdx08]

Both source and target are JBoss wildfly with the following deployment - source has hawtio.war along with fileloading.app.ear deployed, target has jolokia.war along with fileloading.app.ear deployed. Source server deployment screenshot:

Following is the mbeancontent that comes withing domains: {mbeans...} when hawito is doing list() on source server itself.

Chrome debugger screenshot for response of source JVM:

Target server deployment screenshot:

The same value is returned from the target server when jolokia list is called on the remote server but without wrapped within domains and the screen keeps loading, screenshot below:

Chrome debugger screenshot for response of target JVM:

The jolokia.war(1.7.2) deployed on the target server has jokolia-core-1.7.2.jar and the hawtio3.0.1 war deployed on the source has the same jolokia-core-1.7.2.jar.

Following error is thrown while trying to extract mbean information withing the unwindListResponse method because it expects the response needs to be wrapped with domains: {mbeans}

Please let me know if any further details are needed. Thanks in advance !

[grgrzybek]

because it expects the response needs to be wrapped with domains: {mbeans}

Not necessarily - this is one of potential responses. Such value (with domains and cache) are possible only for Karaf deployments with optimization for Camel and ActiveMQ beans (and you don't seem to have any).

You can export the network tab content to a HAR file (from Firefox or Chrome) - is it possible to get such dump?

[debasishdx08]

Dump for local(source_dump.har) and remote(target_dump.har) Mbean listing are attached here: dumps.zip

[grgrzybek]

Which Wildfly version are you using? Are you simply connecting from one hawtio to another using /connect/remote ("Connect" tab)? What did you provide in URL of the remote agent:

[debasishdx08]

The hawtio is deployed on WildFly Full 26.1.3.Final. Yes, I am trying to connect to the remote jolokia (not hawtio) using the connect option from "Connect" tab. The screenshot for the connect settings along with the rest response by hitting jolokia directly are shown below -

Connect Settings:

Jolokia response:

[grgrzybek]

With hawtio-3.0.1.war dropped into Wildfly 26.1.3's standalone/deployments, everything works. Also connection to self (http://localhost:8080/hawtio-war-3.0.1/jolokia) works fine.

I thought you have two WildFlies running, each with hawtio-3.0.1 (that was my first impression). But you're connecting to standalone Jolokia agent - how did you configure this agent (startup command) and what does this command return?:

curl -i -XPOST -H'Content-Type: application/json' -H'Accept: application/json' -d '{"type":"list"}' http://host:8750/jolokia/list?max_depth=7

[debasishdx08]

Yes, you are correct. If I deploy the hawtio.war in target as well, then it works fine. It does not work when the target has just the jolokia.war deployed instead of the hawtio.war. The target WildFly standalone/deployment looks like below:

The response of the curl command is shown below:

The jolokia-unsecured.war version 1.7.2 (renamed as jolokia.war) is deployed on the target WildFly for basic testing and evaluation without any specific jolokia option on the JVM arguments. The web.xml security/auth section is removed.

[grgrzybek]

Because I'm not that fluent with wildfly configuration - where did you configure the port 8750? socket-binding-group in standalone/configuration/standalone.xml? Any other changes there?

[debasishdx08]

Yes, the changes are needed in the socket-binding-group section. The default value are 8080 for http and 8443 for https. The section port-offset="${jboss.socket.binding.port-offset:0}" defines the offset value needs to be added where we have passed 670 as the offset hence the http is listening on 8080 + 670 = 8750.

With the default settings, you can try with 8080/ 8443. Depending on if you have one or both http/https enabled, you should be able to get the response.

[debasishdx08]

Did you manage to replicate the issue ?

[grgrzybek]

Not yet... Everything worked fine. I had some theory about number of MBeans, but couldn't verify it. Is fileloading.app.ear something I could install? Or is it your private application?

[debasishdx08]

It's private application. You want me to try something out ?

[grgrzybek]

no - for now I'll try to simulate. thanks!

[grgrzybek]

As expected - works with jolokia.war running in the other Wildfly instance...

I'll try to deploy an app with simulated MBeans registered (from your HAR).

[debasishdx08]

We are running on open-jdk 17, can that potentially cause any issues ? Also the jolokia war in you case is listening on /jolokia context not /hawtio/jolokia correct ?

[grgrzybek]

JDK shouldn't matter. Also standalone jolokia can run without problems on /jolokia when accessed from another Hawtio.

Please remind me - you see a problem in main Hawtio or in the Hawtio opened with "Connect" button?

[grgrzybek]

For now, I've extracted 2764 MBean names which I'm going to create in both Wildflies. Perhaps some of these MBean names cause problems?

[grgrzybek]

I've registered all the MBeans from your HAR files (source-dump.har), but without those in jboss.as domain (you can't do it). I got your MBeans (these are fake objects without any attributes/methods/notifications), but it still works...

[debasishdx08]

I tried without deploying the fileloading.ear I am able to connect, but when I deploy the fileloading.ear then it fails to connect. isJmxDomains(response) returns true without the fileloading and returns false with my ear. Not sure what to check now

const isJmxDomain = (value: unknown): value is OptimisedJmxDomain =>
      is(value, record(string(), define('MBeanInfo', isMBeanInfo)))
    const isJmxDomains = (value: unknown): value is OptimisedJmxDomains =>
      is(value, record(string(), define('JmxDomain', isJmxDomain)))

    if (isOptimisedListResponse(response)) {
      // Post process cached MBean info
      const { cache, domains } = response
      Object.entries(domains).forEach(([_, domain]) => {
        Object.entries(domain).forEach(([mbeanName, mbeanOrCache]) => {
          if (isString(mbeanOrCache)) {
            domain[mbeanName] = cache[mbeanOrCache] as OptimisedMBeanInfo
          }
        })
      })
      return domains as OptimisedJmxDomains
    }

    if (isJmxDomains(response)) {
      return response
    }

[grgrzybek]

Oh, good hint about domains true/false!

And the good news is that I've reproduced it! Taking your HAR, I've extracted Jolokia's type: list response and I've tweaked Jolokia 1.7.2 to simply return this JSON instead of searching current MBeanServer and I got it!:

[grgrzybek]

For the record - it also fails on latest Hawtio 4.

[debasishdx08]

For the record - it also fails on latest Hawtio 4.

It was working in hawtio 2.x with jolokia 1.7.2

[grgrzybek]

hawtio 2 was based on Angular and hawtio is refactored version.

But I have good news. What is failing during validation of JMX tree is isMBeanInfo() method using this check from https://docs.superstructjs.org/:

is(
  value,
  type({
    desc: string(),
    class: optional(string()),
    attr: optional(record(string(), object())),
    op: optional(record(string(), object())),
    notif: optional(record(string(), object())),
  }),
)

proper MBean Info is a type with single mandatory field - desc. But In your failing HAR file you have FOUR such MBeans:

"j2eeType=ResourceAdapterModule,name=goldensource.jboss.workmanager.rar,J2EEServer=default,J2EEApplication=fileloading.app.ear": {
  "error": "javax.management.InstanceNotFoundException: WFLYEEMGMT0007: Could not find ResourceAdapterModule"
},
"j2eeType=ResourceAdapterModule,name=goldensource.kafka.connector.rar,J2EEServer=default": {
  "error": "javax.management.InstanceNotFoundException: WFLYEEMGMT0007: Could not find ResourceAdapterModule"
},
"j2eeType=ResourceAdapterModule,name=goldensource.filesystem.rar,J2EEServer=default,J2EEApplication=fileloading.app.ear": {
  "error": "javax.management.InstanceNotFoundException: WFLYEEMGMT0007: Could not find ResourceAdapterModule"
},
"j2eeType=ResourceAdapterModule,name=goldensource.engine.rar,J2EEServer=default,J2EEApplication=fileloading.app.ear": {
  "error": "javax.management.InstanceNotFoundException: WFLYEEMGMT0007: Could not find ResourceAdapterModule"
}

Now we can fix it at Hawtio side, because Jolokia simply returns error information (org.jolokia.handler.list.MBeanInfoData#addException()) instead of full javax.management.MBeanInfo.

[debasishdx08]

Thanks a lot !

[grgrzybek]

Now it works!