Issues getting 4.0.0 working

[jacksontj]

I've been using v3.5.7 and decided to start trying out 4.0.0; I installed the latest version and change my config files to the new format, but with the new version installed packets don't flow. I'm running this all through jool file handle jool.conf -- but I have tried doing it through the CLI directly and i'm also unable to get things working.

My config:

{
    "global": {
        "source-icmpv6-errors-better": true,
        "pool6": "64:ff9b::/96"
    },
    "framework": "netfilter",
    "instance": "default"
}

And with pings running this continues to increment:

# jool -i default stats display --explain
JSTAT_POOL4_MISMATCH: 7
Translations cancelled: IPv4 packet's destination address and transport protocol did not match pool4. (ie. Packet was not meant to be translated.)

From my reading in the docs this should be okay to be empty (it was on the previous version and the docs specifically call it out) so I'm unsure why it thinks the pool4 mismatches. Any help would be greatly appreciated!

[ydahhrk]

Yes, it's perfectly normal for that counter to increment on its own on Netfilter Jool. It's because Netfilter Jool intercepts all traffic, and of course not all of it is meant to be translated ("They drop packets deemed corrupted, translate packets which can be translated (...) and return everything else to the kernel."). The counter should be more of a warning if you choose to use iptables Jool instead, since it probably signals that there is some mismatch between the packet-matching rule and Jool's configuration.

Jool 3.5.7 also suffered from this quirk, but it lacked stats counters, so it was harder to notice it.

I tried your configuration and got the following counters after about 8 seconds of a successful ping from IPv6 to IPv4:

JSTAT_SUCCESS: 8
Successful translations. (Note: 'Successful translation' does not imply that the packet was actually delivered.)

JSTAT_BIB_ENTRIES: 1
Number of BIB entries currently held in the BIB.

JSTAT_SESSIONS: 1
Number of session entries currently held in the BIB.

JSTAT_UNKNOWN_ICMP6_TYPE: 3
Translations cancelled: ICMPv6 header's type value was unknown (and thus, untranslatable).

JSTAT_POOL4_MISMATCH: 24
Translations cancelled: IPv4 packet's destination address and transport protocol did not match pool4. (ie. Packet was not meant to be translated.)

I also enabled logging. This was the output (I trimmed a bunch of samey entries to show off the important stuff):

[  569.651120] NAT64 Jool: NAT64 Jool v4.0.0.0 module inserted.
[  583.696861] NAT64 Jool: ===============================================
[  583.696929] NAT64 Jool: Jool instance 'default': Received a v6 packet.
[  583.696971] NAT64 Jool: Packet addresses: 2001::8->64:ff9b::c000:208
[  583.697010] NAT64 Jool: <Some output trimmed here>
[  583.697389] NAT64 Jool: Tuple: 192.0.2.1#64133 -> 192.0.2.8#64133 (ICMP)
[  583.697508] NAT64 Jool: <Some output trimmed here>
[  583.697549] NAT64 Jool: Packet routed via device 'enp0s8'.
[  583.697586] NAT64 Jool: Sending skb.
[  583.697784] NAT64 Jool: Success.
[  583.697936] NAT64 Jool: ===============================================
[  583.697939] NAT64 Jool: Jool instance 'default': Received a v4 packet.
[  583.697943] NAT64 Jool: Packet addresses: 192.0.2.8->192.0.2.1
[  583.697975] NAT64 Jool: <Some output trimmed here>
[  583.697980] NAT64 Jool: Tuple: 64:ff9b::c000:208#19411 -> 2001::8#19411 (ICMP)
[  583.697992] NAT64 Jool: <Some output trimmed here>
[  583.698002] NAT64 Jool: Packet routed via device 'enp0s3'.
[  583.698004] NAT64 Jool: Sending skb.
[  583.698120] NAT64 Jool: Success.
[  584.698526] NAT64 Jool: ===============================================
[  588.751788] NAT64 Jool: Jool instance 'default': Received a v6 packet.
[  588.751795] NAT64 Jool: Packet addresses: 2001::8->fe80::a00:27ff:fe30:6506
[  588.751799] NAT64 Jool: Step 1: Determining the Incoming Tuple
[  588.751803] NAT64 Jool: Unknown ICMPv6 type: 136.
[  588.751806] NAT64 Jool: Done step 1.
[  588.751809] NAT64 Jool: Returning the packet to the kernel.
[  593.890337] NAT64 Jool: ===============================================
[  594.695844] NAT64 Jool: Jool instance 'default': Received a v4 packet.
[  594.695849] NAT64 Jool: Packet addresses: 127.0.0.1->127.0.0.53
[  594.695853] NAT64 Jool: Step 1: Determining the Incoming Tuple
[  594.695859] NAT64 Jool: Tuple: 127.0.0.1#60368 -> 127.0.0.53#53 (UDP)
[  594.695861] NAT64 Jool: Done step 1.
[  594.695864] NAT64 Jool: Step 2: Filtering and Updating
[  594.695866] NAT64 Jool: Packet does not belong to pool4.
[  594.695870] NAT64 Jool: Returning the packet to the kernel.

As you can see from that last packet, the kernel even sends some loopback noise to Jool. Of course, that does not match pool4, so it increments the JSTAT_POOL4_MISMATCH counter. It's normal.

[ydahhrk]

but with the new version installed packets don't flow

Works for me. Guess there's something else getting in the way.

[jacksontj]

@ydahhrk thanks for the quick reply! The behavior of this counter makes sense -- it was just the only thing I saw that seemed out of place. I have also enabled debug logging but it doesn't seem to be printing at all -- maybe something to do with the fact that it was originally a dkms install?

Do you have any suggestions for troubleshooting why (1) the packets aren't flowing or (2) why the debug logging isn't working?

For now I'm going to build a new VM and do the install manually instead of through dkms and see if it works there.

[jacksontj]

Also if there is a more synchronous communication channel that would make debugging easier (slack/irc/etc. let me know :) )

[ydahhrk]

If you have both a DKMS installation and a Kbuild installation, then the DKMS one takes priority.

That is probably the reason why it's not printing anything. (The instructions for enabling debugging install a Kbuild Jool.)

[ydahhrk]

Also if there is a more synchronous communication channel that would make debugging easier (slack/irc/etc. let me know :) )

Goggle Hangouts? My Gmail is in my profile.

[ydahhrk]

BTW: To remove a DKMS module, I do

$ dkms status
jool, 4.0.0.git.v4.0.0, 4.17.0-041700-generic, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!)
$ sudo dkms remove -m jool -v 4.0.0.git.v4.0.0 --all

[jacksontj]

Thanks for your help over hangouts! To summarize here for posterity, my issue with debug logs was exactly this. Then we realized that I was testing in the default netns on the VM -- which jool doesn't work on. After I spawned a netns its working fine. Thanks for the help and the quick responses!