Enhancement: JMS RMI Broker port offset is weird /FISH-1420

[andreasdijkman]

Description

---

Put all the ports of a payara instance in the actual assigned port pool

Expected Outcome

If you create an instance with a port pool or port base, all instances should be in that range of 100, as expected, our should be specified in some property

Current Outcome

If you start your instance with the embedded OpenMQ-broker enabled (default setting), the instance gets started with an OpenMQ-broker-port of JMS_PROVIDER_PORT + 100. That port is not expected or inside an expected range of 100. Instance1 with portbase 5100 starts the JMS-broker with port 5176 and RMI-Broker-port 5276. Instance2 with portbase 5200 cannot start because the JMS-broker can't start because the configured JMS_PROVIDER_PORT 5276 is already in use by the RMI-broker-port of instance1.

Alternatives

Create an extra instance variable or offset-parameter, like JMS_BROKER_PORT that reflects this port or offset of this port. A broker port offset of 100 is unexpected and not configured or very hard to define. I had to look in the source to find the property com.sun.enterprise.connectors.system.mq.rmiport. Alternative is to lower the RMIBROKEROFFSET to something sensible like 1 or 2, but not 100.

https://github.com/payara/Payara/blob/master/appserver/jms/jms-core/src/main/java/com/sun/enterprise/connectors/jms/system/ActiveJmsResourceAdapter.java#L203

Context

Payara 5.2021.9 OpenJDK 8u312

[fturizo]

@andreasdijkman, I take it by your explanation that if there are port clashes across the 2 instances in your example is due to them running on the same host system, correct? If that is the case, wouldn't it be better to define manually set up the JMS provider ports to prevent collisions instead of relying on the offsets like this?

Also, is there any particular reason for letting the 2 instances initialize separate embedded brokers? This is not recommended in a production environment, and if the intention is to create an OpenMQ cluster to make it distributable, then it is much better to run a separate cluster or OpenMQ instances on separate hosts and connect to it from Payara Server remotely.

We'd like to understand better this use case before considering your proposal as the use case is not clear enough.

[andreasdijkman]

Yes, I'm running 2 instances on 1 host. I'm not aware of actively using OpenMQ on the instances but as I said, it's default behavior.

I'm creating instances like this:

asadmin create-local-instance --portbase 5100 instance1
asadmin create-local-instance --portbase 5200 instance2

During creation of the instances, they report JMS_PROVIDER_PORT=5176 and JMS_PROVIDER_PORT=5276.

First start instance1 and wait until it's fully started, OpenMQ and all, then start instance2. Instance2 won't start because of the default config from instance1 of OpenMQ-broker.

If OpenMQ can safely be disabled without touching EJB-containers, EJB-timers and that stuff, then that is the better route. But then the default behavior should be that OpenMQ would be DISABLED by default, not EMBEDDED.

I now fixed it by creating JVM-options

asadmin create-jvm-options --target=instance1-config "-Dcom.sun.enterprise.connectors.system.mq.rmiport\=5177"
asadmin create-jvm-options --target=instance2-config "-Dcom.sun.enterprise.connectors.system.mq.rmiport\=5277"

I also saw that it get's worked around in the following order:

1. Create instance1 with portbase 5100
2. Start instance1
3. Create instance2 with portbase 5200
4. Start instance2

In step 3, the JMS_PROVIDER_PORT is given an offset of 77, not 76 as the intended port 5276 is already in use by the RMI-broker of instace1.

But if I'm understanding this right, If you only need 1 OpenMQ-broker in the domain or no broker at all, why start an OpenMQ-broker on every created instance by default? I'm aware of a blog-post where OpenMQ get's disabled but if it's not needed for core-components like batch or EJB, DISABLED should be the default.

But if you actually do need more then 1 OpenMQ-broker, the offset of the JMS-RMI-broker-port should have a more sane value as it could give you problems starting containers with an adjacent portbase. And the broker-port only gets mentioned in the JVM Report (and netstat), nowhere else. Not in a single config or logfile. From my point of view ALL used ports should be reflected in some config like the instance values of JVM_PROVIDER_PORT and all those parameters.

Just trying to help creating a better product :-)

[fturizo]

Thanks for the extensive update, @andreasdijkman. I agree with you in that this issue is caused by the OpenMQ broker being enabled by default, which in a production environment would be unneeded in most cases. Let me bring this issue to our product management team and decide what would be the best way to proceed on this issue

[fturizo]

Greetings @andreasdijkman. Our Product Management team has weighted their thoughts on this issue, and they agree that the --portbase usage needs some review and fixing in general for specific cases like yours. To this effect, we have added your scenario to the internal issue FISH-1420, which has been in our backlog for some time, so that the feature is fixed all across the board.

Due to the way this feature works, and its very specific use cases, we have categorized it as low priority for the moment, and it may take some considerable time to be fixed, perhaps after Payara Platforrm 6 is out in the open.