Connection not possible if a client and Calimero Server on same Docker host

[aschamberger]

I tried running Calimero as planned on my production home assistant server. However its not working. I already opened a issue at https://github.com/XKNX/xknx/issues/815 for the Home Assistant part. However it seems the problem lies on the Calimero side if the client application is running on the same Docker host. When I run Calimero in the same setup on another device at least Routing works. Tunneling also fails. You can find all the logs at https://github.com/XKNX/xknx/issues/815

[bmalinowsky]

Can you create a trace of what messages are actually going back and forth when you open a connection. Since you use containers, e.g., use netshoot alongside the server container:

docker run -it --rm --net container:calimero-container-name nicolaka/netshoot tcpdump

[aschamberger]

I'll send you the tcpdump of a home assisstant restart which hopefully includes the connect problems. I did't want to publicly upload the tcpdump as it might include passwords as the all containers run in host networking mode. Thanks for your support.

[farmio]

Hi 👋! I did try this too and ran into similar problems. Unfortunately I have no experience whatsoever with Java so debugging this is quite hard for me. I have set up calimero docker server with port forwarding (network_mode: "host" didn't work properly on macOS)

<knxServer name="calimero-server" friendlyName="Calimero KNX IP Server">
    <!-- KNXnet/IP search & discovery -->
    <discovery listenNetIf="all" outgoingNetIf="all" activate="true" />

    <!-- Provides the KNXnet/IP-side configuration for access to one KNX subnet -->
    <serviceContainer activate="true" routing="true" networkMonitoring="true" 
        udpPort="3671" listenNetIf="any">
        <knxAddress type="individual">7.1.0</knxAddress>
        <routing/>

and <knxSubnet type="virtual">virtual-subnet</knxSubnet> and tried establish a tunneling connection to it form xknx. The ConnectionRequest was never answered but it seems calimero has received it:

calimero  | [calimero-server control endpoint virtual-subnet] INFO calimero.server.calimero-server - virtual-subnet: setup data endpoint (channel 1) for connection request from 127.0.0.1:50756 (udp)
calimero  | [calimero-server control endpoint virtual-subnet] INFO calimero.server.calimero-server - assign additional individual address 7.1.10 to channel 1

Is there a way to enable some kind of debug logging for the calimero conatiner? Is it even possible to tunnel to a calimero server? Do I have to configure it somehow - I didn't find anything in the readme.

[bmalinowsky]

@farmio Hi, Docker on MacOS runs in a VM, so host networking won't work as you'd expect it to. With port forwarding, I can't remember right now how that reacts if you open a tunneling connection, as the communication ports change after connection setup (unless the server only supports 1 connection).

In the config file, for a container I would set listenNetIf to something more specific, "any" is basically if you know that the chosen netif is the one you want without needing its name.

Debug: well, yeah there is, but one of the first things I did after the 2.5 release was to make that easier, because it was quite cumbersome to change in a container that you throw away. It's not on Docker hub yet.

Tunneling is always supported, configured as stated in the configuration (reuse ctrl endpoint attribute vis-à-vis additionalAddresses).

[aschamberger]

Regarding debugging: I'm setting the loglevel via an env variable for the container and set it via JVM Parameter. @see: https://github.com/aschamberger/hassio-addons/blob/bbee72a87a91980792655d49fa5e134cfee4fa85/calimero-server/run.sh#L39 and https://github.com/aschamberger/hassio-addons/blob/bbee72a87a91980792655d49fa5e134cfee4fa85/calimero-server/run.sh#L89

[farmio]

Ah great, I'll look into that. I've moved to an x86 Linux host now (aarch64 Ubuntu VM on Mac wouldn't start Calimero Docker) and got a ConnectionResponse - so that's great! When I try to send a TunnellingRequest it doesn't ACK - but it responds to the then following DisconnectRequest properly so that's something. Step by step.

What is the difference between emulated and virtual subnets? I'd like to use the calimero container as unittest target for xknx so I don't need an actual subnet but proper ACSs and L_Data.con to test our connection logic.

Is there an example/documentation of the datapoint xml file for an emulated subnet?

[aschamberger]

Here both home-assistant.log and tcpdump (docker run --net container:addon_local_calimero_server nicolaka/netshoot tcpdump port 3671 -w - > /tmp/tcpdump.out) at the same time. This is from a HA restart with routing configured and HA config as below. I also tried to send manually KNX commands via knx.send to 1/1/1. This was not processed on the KNX side as this would have resulted in a 1/4/1 state response.

logger:
  default: warning
  logs:
    homeassistant.components.fritz: error
    fritzconnection: error    ## it seems this one is the problematic
    fritzprofiles: error    
    # For most debugging needs `xnx.log` and one of `xknx.knx` or `xknx.telegram` are a good choice.
    #xknx: debug  # sets the level of all loggers
    xknx.log: debug  # provides general information (connection, etc.)
    xknx.raw_socket: debug  # logs incoming UDP frames in raw hex format
    xknx.knx: debug  # logs incoming and outgoing KNX/IP frames at socket level
    xknx.telegram: debug  # logs telegrams before they are being processed at device level or sent to an interface
    #xknx.state_updater: debug  # provides information about the state updater

knx:
  individual_address: 1.1.105
  routing:

home-assistant.log tcpdump.out.log

[bmalinowsky]

@farmio I currently upload only amd64 builds on Dockerhub, so aarch64 cannot pull the required image. But the Dockerfiles for it are in the calimero-docker repo.

In the latest build, you can now supply -v, -vv, or -vvv for info/debug/trace, respectively. I.e., calimeroproject/knxserver -vvv ....

An emulated subnet will store process communication values for any datapoints, and serve them back to you on request. You basically don't need a datapoints file if the emulation should keep all datapoints. The datapoints file limits them to a specific set. A datapoints file example is here.

A virtual subnet creates a virtual link so that a certain installation can be modeled. For example, calimero-testnetwork uses this to add two knx devices as a small knx network with a single line. Using a virtual link requires some programming.

[farmio]

if you open a tunneling connection, as the communication ports change after connection setup (unless the server only supports 1 connection).

Well this seems to be the curlpit of my tunneling problems. Xknx expects the tunneling control- and data-endpoint to be the same. I'll have to add support for such a 2-socket tunnel (I have not seen a knx device doing this 😅). Is it possible to xml-configure the used ports for the data endpoints so eg. a firewall can be configured accordingly (it took some time until i figured out why these frames don't reach my client device 🙃 -vv helped a lot 🙏)? I tried to enable just 1 connection but it still seems to use a new random port.

        <additionalAddresses>
            <knxAddress type="individual">7.1.10</knxAddress>
        </additionalAddresses>

@aschamberger I think, when this is supported, using tunneling would be the way to go, as multicast in a docker environment may cause problems.

[bmalinowsky]

Every server that supports >1 tunneling connections will use a UDP port range (or randomized), so supporting data endpoint ports is the way to go :)

You can use the control endpoint for a connection if you configure reuseCtrlEP in the serviceContainer element. With that setting you don't need additional addresses, and the server will only allow a single client connection at any time.

[bmalinowsky]

@farmio No, in the server config there is currently no way to assign specific server udp ports besides control endpoints. It's handled by the OS the server is running on.

It certainly can be done, it seems a quite fragile setup. (Simplest case: port already used)

[farmio]

Thank you very much for your help! With reuseCtrlEP="true" xknx can create a tunneling connection, send telegrams to the emulated subnet and read the values back with GroupValueReads without a problem! Great! @aschamberger I'd suggest you try this configuration too with your HA addon container - it doesn't need multiple connections anyway.

Every server that supports >1 tunneling connections will use a UDP port range (or randomized), so supporting data endpoint ports is the way to go :)

Not sure about that - every remotely modern interface supports multiple connections, but I never heard of one not compatible with xknx / HomeAssistant. They all just use 3671 for both data and control. I guess they internally rout solely on the channel id. See ConnectResponses of my Gira and MDT routers.

Anyway it is defined in the KNX specs so we should definitely support it form xknx too.

@bmalinowsky Thank you again for your help! This is a great project, I'm sure I can learn a lot from it.

[aschamberger]

@aschamberger I'd suggest you try this configuration too with your HA addon container - it doesn't need multiple connections anyway.

I need multiple connections as I use it for ETS and HA. So I'm looking forward to xknx supporting the standard required to connect with Calimero as this seems to be the problem when I understand you right.

[marvin-w]

I tested this as well with my Weinzierl KNX IP BAOS 777 which supports multiple connections as well. It seems it's also using only 3671 as the control and data endpoint.

[aschamberger]

I close the issue as the implementation of https://github.com/XKNX/xknx/pull/832 fixed the problem.