Pade 1.6.2 Connections Closing after 17 seconds.

[rcampion]

Using Openfire 4.7.1, Pade 1.6.2

Bridge failure at 17 seconds.

Using WebSockets

Any ideas?

[rcampion]

[gjaekel]

Please provide us a log of the full startup of OpenFire with Pàdé. Do you have NAT between the Internet and the outer interface? Then you probably step into an (probably fixed, #385) regression of release 1.6.2. You might try to use the latest 1.6.3-snapshot on https://www.igniterealtime.org/projects/openfire/plugin-archive.jsp?plugin=pade

[rcampion]

I will try 1.6,3-snapshot

openfire.log stderror.log stdoutt.log

https://www.zdslogic.com/live/ZdsLogic

username:test.user password:Testing123

[rcampion]

Changed router setting NAT filtering Open

Still same problem

[rcampion]

Does anyone have a working Nginx config for WebSocket?

[gjaekel]

Concerning your logs:

2022.02.24 00:44:06 ^[[1;31mERROR^[[m [pool-6-thread-2]: org.jivesoftware.openfire.container.PluginManager - An exception occurred while loading plugin 'ofmeet':
java.lang.IllegalStateException: OFFocus plugin detected. This version of The OFMeet plugin cannot run next to the OFFocus plugin (OFFocus is no longer needed). Please remove OFFocus and reinstall OFMeet!
        at org.jivesoftware.openfire.plugin.ofmeet.OfMeetPlugin.initializePlugin(OfMeetPlugin.java:111) ~[ofmeet-0.9.5.jar:?]
        at org.jivesoftware.openfire.container.PluginManager.loadPlugin(PluginManager.java:637) [xmppserver-4.7.1.jar:4.7.1]
        at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:375) [xmppserver-4.7.1.jar:4.7.1]
        at org.jivesoftware.openfire.container.PluginMonitor$MonitorTask$4.call(PluginMonitor.java:363) [xmppserver-4.7.1.jar:4.7.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022.02.24 00:44:16 ^[[32mINFO ^[[m [pool-6-thread-4]: org.jivesoftware.openfire.container.PluginManager - Successfully loaded plugin 'offocus'.

• It seems that you have also installed the completely outdated OFmeet- and OFFocus-Plugins. You have to remove them before you may use any Version of Pàdé
• There are also some SQL-Statement Errors. Did you use the "right" DDLscript to setup or update the Database?

• Starting with V1.6, Dèle changed the loglevel for the output of some detailed setup information for the external JiCoFo and JVB components from INFO to DEBUG, but the global loglevel is INFO. For that, this is now hidden as intended but I forgot to advise you to re-enalbe it. Please add the following to the .../conf/log4j2.xml near bottom:

  [...]
      <!-- <<< 20220217/gj    Re-Enable JVB/JiCoFo logging -->
      <Logger name="org.jivesoftware.openfire.plugin.ofmeet.JitsiJvbWrapper" level="debug"/>
      <Logger name="org.jivesoftware.openfire.plugin.ofmeet.JitsiJicofoWrapper" level="debug"/>
      <!-- >>> -->
  
      <Root level="info">
  [...]

  Please fix the issues mentioned, enable the logging for the Jitsi comonents and re-record the startup process (in an empty logfile).

If you're able to load and start the WebClient within a browser, the first step is done. If the Application can't build-up the XMPP controll connection and/or the A/V streams, it's probably some configuration issue concerning the (comparable complex) network parameters. Maybe https://github.com/igniterealtime/openfire-pade-plugin/wiki/OFmeet-Network-Scheme will help you if it fits to your network schema.

[gjaekel]

I connect to your site and inspect the Javascript browser console: As expected, the so-called ICE-handling for the A/V-streams fail, it just offers some private network address (192.168.1.100:10000/udp(host)). Probably the IP-Address announced by the ICE-harvester component are not the "right one" (not the visible public internet address) or the used UDP-port isn't "open" or traversed by the firewall.

[rcampion]

I must have sent you the wrong logs. This is a completely new 4.7.1 server.

[gjaekel]

Do you use Openfire/Pàdé (or OFMeet) before (with success?)

[rcampion]

openfire.log stdoutt.log

[rcampion]

4.5.1 worked with old plugins

[gjaekel]

4.5.1 worked with old plugins

Good to know. Then, we don't have an "external" problem.

"This" opefire.log looks much better, e.g. cleaner. Now, you have to enable the DEBUG logging level. FYI: The log4j.xmlis even loaded dynamical, but we need the info just logged at startup.

[rcampion]

Thanks for you help. How do I set the DEBUG logging level?

[gjaekel]

You're welcome! BTW: Timezone difference is +6h; it's about 19 o'clock here at Frankfurt/Main (Germany). I have to prepare evening dinner soon, but probably get time later in (my) evening.

For logging, see above at https://github.com/igniterealtime/openfire-pade-plugin/issues/392#issuecomment-1058310142

[gjaekel]

Does anyone have a working Nginx config for WebSocket?

Sorry, me is using good-old Apache HTTPd :smile: Maybe you got hints from https://www.unixe.de/jitsi-websockets-und-nginx/ or https://github.com/jitsi/docker-jitsi-meet/issues/774#issuecomment-771123152

[rcampion]

Can;t find log4j.xml

[gjaekel]

Sorry, it's named log4j2.xml, should be at /_/servers/xmpp/openfire/4.7.1/conf/at your site (which looks well-organized to me).

[rcampion]

Thanks. There is no log4j2.xml in that folder.

[rcampion]

[gjaekel]

Very strange, because it must read some Log4J config file to get this output IMHO. I'm very sorry, I have to break now.

[rcampion]

Enjoy your dinner!

[gjaekel]

You be right, I can't find it by hurry in the distribution archive! Maybe the Openfire-Team have shifted it to some other location. I don't update this folder by intention while switching to Openfire-4.7.1

[gjaekel]

Maybe now it's sub-packed in some JAR; that's possible. I'll send you my version lateron and you may place it there.

[gjaekel]

Located it at .../lib/log4j2.xml. You may insert the loglevel changing statements there.

[rcampion]

Added the statements deleted the existing logs restarted the server

[rcampion]

I just replaced the log4j2.xml file with the one you sent me.

[rcampion]

Did a test call. Here are the new logs. openfire.log stderror.log stdoutt.log

[rcampion]

JAVAHOME=//java/64/jdk1.8.0_311

[rcampion]

restarted server with new JVM

[rcampion]

/_/java/64/jdk1.8.0311/jre/bin/java -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dconfig.file=//servers/xmpp/openfire/4.7.1/plugins/pade/classes/jvb/application.conf -Dnet.java.sip.communicator.SC_HOME_DIRLOCATION=//servers/xmpp/openfire/4.7.1/plugins/pade/classes/jvb -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=config -Djava.util.logging.config.file=./logging.properties -Djdk.tls.ephemeralDHKeySize=2048 -cp ./jitsi-videobridge-2.1-SNAPSHOT.jar:./jitsi-videobridge-2.1-SNAPSHOT-jar-with-dependencies.jar org.jitsi.videobridge.MainKt --apis=rest 20220303-205804.429 DEBUG [pool-6-thread-2] [o.j.o.p.o.JitsiJvbWrapper] JVB config. videobridge {

http-servers {
    private {
        port = 8188
        host = 192.168.1.100
    }
    public {
        port = 8180
        host = 192.168.1.100
    }
}

websockets {
    server-id = zdslogic.com
    enabled = true
    domain = "zdslogic.com:7443"
    tls = true
}

ice {
    tcp {
        enabled = false
        port = "4443"
        mapped-port = "4443"
    }
    udp {
        port = "10000"
        local-address = 192.168.1.100
        public-address = 192.168.1.100
    }
}

sctp {
    # Whether SCTP data channels are enabled
    enabled = false
}

apis {
    xmpp-client {
        configs {
            # Connect to the first XMPP server
            shard {
                hostname= "zdslogic.com"
                domain = "zdslogic.com"
                username = "jvb"
                password = "G6Qjnu6Dl9gd3Ch46CpNwAjSKpNRhyMbKCS30dj5"
                muc_jids = "ofmeet@conference.zdslogic.com"
                muc_nickname = "jvb"
                disable_certificate_verification = true
            }
        }
    }
    rest {
        enabled = true
    }
}

health {
    interval = 300 seconds
}

stats {
    # Enable broadcasting stats/presence in a MUC
    enabled = true
    transports = [
        { type = "muc" }
    ]
}

rest {
    debug {
        enabled = true
    }
    health {
        enabled = true
    }
    shutdown {
        enabled = true
    }
    version {
        enabled = true
     }
}

}

[rcampion]

openfire.log stderror.log stdoutt.log

[DevynCJohnson]

4.5.1 worked with old plugins

What specific version of Pade worked with Openfire 4.5.1? My team is also having the exact same issue with Openfire and Pade as you are having. We started with the newest versions and are needing something that works.

[gjaekel]

You posted the content of .../plugins/pade/classes/jvb/application.conf. What's the content .../plugins/pade/classes/jvb/config/sip-communicator.properties, please. In the creating code Dèle did changes to introduce clustering support what causes the regression. I "nailed" a hand-edited version with the correct parameters by setting it r/o for the Openfire user. This will work but you must not restart the plugin via the AdminUI afterward, because this action will (try to and fail) completely erase a plugins subdirectory at .../plugins/foo and expand it from the corresponding JAR.

Here the content added to my current version of the file; this will make V1.6.2 and recent others of the regression working:

root@prodfire0 /opt/openfire # grep harvest ./plugins/pade/classes/jvb/config/sip-communicator.properties
org.ice4j.ice.harvest.USE_DYNAMIC_HOST_HARVESTER=false
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=<local ip>
org.ice4j.ice.harvest.USE_DYNAMIC_HOST_HARVESTER=false
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=<public ip>

This entries may be missing or wrong.

Also, the section http-servers of the file .../plugins/pade/classes/jvb/config/application.conf must contain the line host = <local ip> in both subsections private and public. The term public is misleading here, because at the plugin we have an wss-to-ws proxy implemented in front of to be reachable from outside.

videobridge {

    http-servers {
        private {
            port = 8188
            host = <local ip>
        }
        public {
            port = 8180
            host = <local ip>
        }
    }

[rcampion]

From sip-communicator.properties

Jisti Video Bridge

Thu Mar 03 20:58:04 EST 2022

org.jitsi.videobridge.TCP_HARVESTER_PORT=4443 org.jitsi.videobridge.TRUST_BWE=true org.jitsi.videobridge.TCP_HARVESTER_MAPPED_PORT=4443 org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true org.ice4j.ice.harvest.USE_DYNAMIC_HOST_HARVESTER=false org.jitsi.videobridge.octo.PUBLIC_ADDRESS=192.168.1.100 org.jitsi.videobridge.REGION=region1 org.jitsi.videobridge.SINGLE_PORT_HARVESTER_PORT=10000 org.jitsi.videobridge.octo.BIND_ADDRESS=192.168.1.100 org.jitsi.videobridge.octo.BIND_PORT=4096 org.jitsi.videobridge.TCP_HARVESTER_SSLTCP=false

From application.conf (one level up)

videobridge {

http-servers {
    private {
        port = 8188
        host = 192.168.1.100
    }
    public {
        port = 8180
        host = 192.168.1.100
    }
}

websockets {
    server-id = zdslogic.com
    enabled = true
    domain = "zdslogic.com:7443"
    tls = true
}

ice {
    tcp {
        enabled = false
        port = "4443"
        mapped-port = "4443"
    }
    udp {
        port = "10000"
        local-address = 192.168.1.100
        public-address = 192.168.1.100
    }
}

sctp {
    # Whether SCTP data channels are enabled
    enabled = false
}

apis {
    xmpp-client {
        configs {
            # Connect to the first XMPP server
            shard {
                hostname= "zdslogic.com"
                domain = "zdslogic.com"
                username = "jvb"
                password = "G6Qjnu6Dl9gd3Ch46CpNwAjSKpNRhyMbKCS30dj5"
                muc_jids = "ofmeet@conference.zdslogic.com"
                muc_nickname = "jvb"
                disable_certificate_verification = true
            }
        }
    }
    rest {
        enabled = true
    }
}

health {
    interval = 300 seconds
}

stats {
    # Enable broadcasting stats/presence in a MUC
    enabled = true
    transports = [
        { type = "muc" }
    ]
}

rest {
    debug {
        enabled = true
    }
    health {
        enabled = true
    }
    shutdown {
        enabled = true
    }
    version {
        enabled = true
     }
}

}

[rcampion]

Any chance you can test this?

https://www.zdslogic.com/live/ZdsLogic Username: test.user Password: Testing123

[gjaekel]

Yes, but you have to wait some minutes because i'm in an meeting at our instance at DNB. :)

[rcampion]

Thank you in advance Dr. Guido. Hope the pizza was good last night, lol

[gjaekel]

Yes, because it was a self-made home-brew one.

[rcampion]

Should my public IP be my external IP?

174.79.160.56

[gjaekel]

Should my public IP be my external IP?

Yes.

# host www.zdslogic.com
www.zdslogic.com is an alias for zdslogic.com.
zdslogic.com has address 174.79.160.56

Did you already add the missing haverster configuration to the sip-communicator.properties?

org.ice4j.ice.harvest.USE_DYNAMIC_HOST_HARVESTER=false
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=192.168.1.100
org.ice4j.ice.harvest.USE_DYNAMIC_HOST_HARVESTER=false
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=174.79.160.56

and also changed application.conf?

ice {
    tcp {
        enabled = false
        port = "4443"
        mapped-port = "4443"
    }
    udp {
        port = "10000"
        local-address = 192.168.1.100
-       public-address = 192.168.1.100
+       public-address = 174.79.160.56
    }
}

[gjaekel]

In the AdminUI, it should look like this:

[rcampion]

FANTASTIC. Thank you so much Dr. Guido

[gjaekel]

Thank you also for the nice test meeting on your instance.

@deleolajide: You should publish the upcomming release 1.6.3 containing a fix for this issue soon because it will hit by everyone using a NATed connection to the Internet, which will be the case in most enterprise-likel environments.

[deleolajide]

You should publish the upcoming release 1.6.3 containing a fix for this issue soon because it will hit by everyone using a NATed connection to the Internet, which will be the case in most enterprise-like environments.

This NAT issue should be been fixed in 1.6.2. I waited for confirmation before I made the release. See https://github.com/igniterealtime/openfire-pade-plugin/commit/a9a48ca47e26469e801a92a2dcb96813c2b8f1b1

The fix in 1.6.3 was to tidy up the clustering logic and make sure that if no values are specified, then clustering should NOT override with the XMP properties. XML properties should only override any specified explicit DB values. This will only affect clustering users and I don't think there are any users right now.

[DevynCJohnson]

4.5.1 worked with old plugins

@rcampion , you did not reply to my question. I will re-post it below.

What specific version of Pade worked with Openfire 4.5.1? My team is also having the exact same issue with Openfire and Pade as you are having. We started with the newest versions and are needing something that works.

[gjaekel]

@deleolajide Now I got the time to evaluate Pade-1.6.2 on Openfire-4.7.1 by my own: IMHO, this fixes the regression in writing config files. @rcampion Maybe the only problem was that you don't enter the right NAT settings in the Admin UI (https://github.com/igniterealtime/openfire-pade-plugin/issues/392#issuecomment-1059047150)

@DevynCJohnson The Releases page (https://github.com/igniterealtime/openfire-pade-plugin/releases) states:

This version ["1.6.2"] is for Openfire servers at version 4.7.x For Openfire servers at version 4.6.4, 4.6.5, 4.6.6 and 4.6.7, use version 1.5.6 For Openfire 4.6.3 and below, use version 1.5.2

Any special reason not to switch to OpenFire-4.7.1 ? No problem to look at your installation as well if you like. In that case please provide "clean" logs with the debugging for the JVM-Wrappers enabled as described.

[rcampion]

@gjaekel Yes, I think you are right. I saved the settings just in case. Thanks again for all your help.

[DevynCJohnson]

@gjaekel , thanks. I will certainly try OpenFire-4.7.1. In the next day or two, I will be sure to report back the findings and I will reach out to you if needed.

[gjaekel]

@DevynCJohnson Thank you for the logfile you send by email. But you didn't enable JiCoFi and JVB debug logging as discussed above. Please provide such a logfile because it contain the actual effective parameter settings for JiCoFo and JVB, especially for the ICE NAT harvester. This is the part of communication which probably fails.

From an exception one can see nevertheless that the JVB's public interface don't listen to the public ip:

2022.03.06 14:59:22 ERROR [socket_c2s-thread-2]: org.jivesoftware.openfire.plugin.ofmeet.JitsiJvbWrapper - getConferenceStats 24.107.19.245

In the other hand, some TURN server is told do use a local IP on a parameter for an public ip.

2022.03.06 14:55:31 INFO  [pool-6-thread-1]: org.ifsoft.turn.openfire.PionTurn - PionTurn enabled D:\WEBSERVER\Openfire\plugins\pionturn\classes\win-64\turn-server-log.exe -public-ip 192.168.117.62 -port 3478 -realm superiorsoft.com

Note that in the common case the A/V streams use UDP transport for reasons of performance and you need a STUN server just in the rare case where you must support clients placed on networks that don't transport UDP.

[DevynCJohnson]

@gjaekel , are you suggesting that the JVB wrapper not listen to a public IP? As for the PionTurn server, its name is confusing as that plugin provides a STUN server. Should that plugin be configured another way?