search

ntop / n2n

Peer-to-peer VPN
GNU General Public License v3.0
6.26k stars 941 forks source link

Communication issue between Windows and Linux edge nodes, when encryption is enabled #617

Closed franzflasch closed 3 years ago

franzflasch commented 3 years ago

This is my setup:

edge-node[linux, x86] <---> super-node[linux, x86] <---> edge-node[windows,x86]

I encountered a problem when connecting edge nodes between linux and windows.

The issue seems to only occur when encryption is enabled, using the '-k' option. If i omit the -k parameter windows and linux edge nodes are communicating just fine.

When encryption is enabled both edges seem to connect fine to the supernode, however when i try pinging from the linux edge to the windows edge i get some strange output on the windows machine:

EtherType 0x6CF1EtherType 0xB77DEtherType 0x6201

And pinging obviously does not work.

This is how I start the nodes:

Supernode: ~# sudo ./supernode -p 7037 -c community -f -vv

Community file:

~# cat ../community
mynetwork

Edgenodes ~# sudo ./edge edge.conf

edge config files: Linux:

-c mynetwork
-k test
-a 192.168.126.10
-l localhost:7037
-A5
-f

Windows:

-c mynetwork
-k test
-a 192.168.126.8
-l 172.17.17.2:7037
-A5
-f

The setup works just fine, if encryption is not used ("Using null cipher"). I also tested with and without openssl and with different Ciphers. It makes no difference.

I used the same version on both machines. (Commit f352c992555cfd4bb2f127dea799c2ef8d796206, from the dev branch)

Logan007 commented 3 years ago

Thank you for reporting!

The EtherType output originates from the network filter which is unable to determine the packet type. The output values do no seem to be even real/valid ones, so maybe the filter gets to see still encrypted or wrongly decrypted packets. The reason needs to be found....

Maybe, the key parameter is interpreted differently on Windows than on Linux? Can't imagine and have not heard of that before, but I will have a look, I can't test on Windows though. I might need your help inserting some printfs into parameter handling code section and packet handling. I am on it.

Also, if you run the edges on full -vvvvv, is there anything special in the edge's log debug output?

Logan007 commented 3 years ago

I have recently applied some changes (4e2548a98c43ab97c899d61acf7a9871a1614ad0) to the command line parameter code and especially the conf-file portion. While working well on Linux, it might not work so well for Windows... let's see.

Pulling in a current getoopt to win32/ directory might help it as the new conf file parsing uses optind which might not have the same effect on the older version. I ruled this out by manually overwriting the lib code with this code.

Logan007 commented 3 years ago

May I ask you to add a printf("!!! OPTION '%c' --- '%s' !!!\n", c, optarg); line to the loadFromCLI() function (it also gets called during .conf file reading) in the src/edge.c file around line 679 as follows

        ...
        if(c == 255) break;
        printf("!!! OPTION '%c' --- '%s' !!!\n", c, optarg);
        setOption(c, optarg, ec, conf);
        ...

and share the output to see if the key is read correctly?

franzflasch commented 3 years ago

Hi!

Of course, I am glad that I can help. I will test this asap. However currently I am not at home, so I can test it in the evening at the earliest.

franzflasch commented 3 years ago

OK, now I tested this with the latest revision:

Linux output:

!!! OPTION 'c' --- 'mynetwork' !!!
!!! OPTION 'k' --- 'test' !!!
!!! OPTION 'a' --- '192.168.126.10' !!!
31/Jan/2021 20:27:43 [edge_utils.c:3031] Adding supernode = localhost:7037
!!! OPTION 'l' --- 'localhost:7037' !!!
!!! OPTION 'A' --- '5' !!!
!!! OPTION 'f' --- '(null)' !!!

Windows output:

!!! OPTION 'c' --- 'mynetwork' !!!
!!! OPTION 'k' --- 'test' !!!
!!! OPTION 'a' --- '192.168.126.8' !!!
31/Jan/2021 20:33:57 [/working_dir/n2n_new/src/edge_utils.c:3031] Adding supernode = 172.17.0.2:7037
!!! OPTION 'l' --- '172.17.0.2:7037' !!!
!!! OPTION 'A' --- '5' !!!
31/Jan/2021 20:33:57 [/working_dir/n2n_new/src/edge.c:635] WARNING: Unknown option -f: Ignored
!!! OPTION 'f' --- '(null)' !!!

Seems to be pretty much the same.

Please let me know if you want me to do any further testing.

Logan007 commented 3 years ago

I took the hard way to setup a Windows VM and finally suceeded following n2n's Windows build instructions (first time! – updated the instructions along the way).

I followed exactly your setup, even the localhost (but my own Linux IP address of course), typed in the edge.conf file on Windows using its notepad.

First of all, I was not able to ping the Windows computer at all. So, I turned off its firewall and restarted the edge and got it working then.

And, there was absolutely no issue in using encrypted traffic!

I then changed the key on the Linux computer from test to testt (or any other different key) and then saw exactly the issue you described.

So, I think that key passing is the problem. Maybe character set or unicode encoding? Maybe whitespaces, line-feed, or carriage-return?

Could you please try to somehow rule those out, maybe by explicitly checking the .conf files' -k lines as well as trying to provide the key through command line -k test which will work in addition to the .conf file if .conf file is first, e.g. sudo ./edge edge.conf -k test?

franzflasch commented 3 years ago

This was my first guess too. However even if I give all parameters in the cmdline without any configfile it does not work either. Supernode, linux:

sudo ./supernode -p 7037 -c ../community -f -vvvv
01/Feb/2021 13:43:18 [sn.c:114] Added allowed community 'mynetwork' [total: 1]
01/Feb/2021 13:43:18 [sn_utils.c:609] Assigned sub-network 10.164.148.0/24 to community 'mynetwork'.
01/Feb/2021 13:43:18 [sn.c:161] Loaded 1 fixed-name communities from ../community
01/Feb/2021 13:43:18 [sn.c:164] Loaded 0 regular expressions for community name matching from ../community
01/Feb/2021 13:43:18 [sn.c:697] traceLevel is 6
01/Feb/2021 13:43:18 [sn.c:704] supernode is listening on UDP 7037 (main)
01/Feb/2021 13:43:18 [sn.c:712] supernode is listening on UDP 5645 (management)
01/Feb/2021 13:43:18 [sn.c:721] Dropping privileges to uid=65534, gid=65534
01/Feb/2021 13:43:18 [sn.c:737] supernode started

edge-node, linux:

sudo ./edge -a 192.168.126.11 -c mynetwork -k test -l 172.17.0.1:7037 -d tap1 -f
01/Feb/2021 12:43:42 [edge_utils.c:3031] Adding supernode = 172.17.0.1:7037
01/Feb/2021 12:43:42 [edge.c:891] WARNING: Switching to AES as key was provided.
01/Feb/2021 12:43:42 [edge.c:902] Starting n2n edge 2.9.0.r854.c2741ac Feb  1 2021 12:36:03
01/Feb/2021 12:43:42 [edge.c:908] Using compression: none.
01/Feb/2021 12:43:42 [edge.c:909] Using AES cipher.
01/Feb/2021 12:43:42 [edge_utils.c:227] Number of supernodes in the list: 1
01/Feb/2021 12:43:42 [edge_utils.c:229] supernode 0 => 172.17.0.1:7037
01/Feb/2021 12:43:42 [edge.c:932] Use manually set IP address.
01/Feb/2021 12:43:42 [edge.c:1043] Created local tap device IP: 192.168.126.11, Mask: 255.255.255.0, MAC: 6E:02:94:72:A1:04
01/Feb/2021 12:43:42 [edge.c:1091] WARNING: n2n has not been compiled with libcap-dev. Some commands may fail.
01/Feb/2021 12:43:42 [edge.c:1096] Dropping privileges to uid=65534, gid=65534
01/Feb/2021 12:43:42 [edge.c:1120] edge started
01/Feb/2021 12:43:42 [edge_utils.c:753] Successfully joined multicast group 224.0.0.68:1968
01/Feb/2021 12:43:42 [edge_utils.c:2176] [OK] Edge Peer <<< ================ >>> Super Node
^CEtherType 0x8549EtherType 0x82A2EtherType 0xE8CFEtherType 0x1633EtherType 0x18D8EtherType 0x7425EtherType 0xC8E5EtherType 0x253DEtherType 0x84BFEtherType 0xFA57EtherType 0x60CCEtherType 0x4CF5EtherType 0xBF0BEtherType 0x0B29EtherType 0x3A02EtherType 0x1D3CEtherType 0xA70CEtherType 0x8A0501/Feb/2021

edge-node, windows:

.\edge.exe -a 192.168.126.8 -c mynetwork -k test -l 172.17.0.1:7037
01/Feb/2021 12:43:51 [/n2n_new/src/edge_utils.c:3031] Adding supernode = 172.17.0.1:7037
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:891] WARNING: Switching to AES as key was provided.
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:902] Starting n2n edge 2.9.0.r854.c2741ac Feb  1 2021 12:36:55
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:908] Using compression: none.
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:909] Using AES cipher.
01/Feb/2021 12:43:51 [/n2n_new/src/edge_utils.c:227] Number of supernodes in the list: 1
01/Feb/2021 12:43:51 [/n2n_new/src/edge_utils.c:229] supernode 0 => 172.17.0.1:7037
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:932] Use manually set IP address.
Open device [name={EC3E764B-54EB-4328-99BA-8198708B83E6}][ip=192.168.126.8][ifName=OpenVPN TAP-Windows6][MTU=1290][mac=00:FF:EC:3E:76:4B]
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:1043] Created local tap device IP: 192.168.126.8, Mask: 255.255.255.0, MAC: 00:FF:EC:3E:76:4B
01/Feb/2021 12:43:51 [/n2n_new/src/edge.c:1120] edge started
01/Feb/2021 12:43:51 [/n2n_new/src/edge_utils.c:753] Successfully joined multicast group 224.0.0.68:1968
01/Feb/2021 12:43:51 [/n2n_new/src/edge_utils.c:2176] [OK] Edge Peer <<< ================ >>> Super Node
EtherType 0x73B4EtherType 0xE455EtherType 0x8CAF

Which Windows are you using? I tested this on Windows 10.

franzflasch commented 3 years ago

The same setup running solely on linux does work without any issues. The problem only occurs when mixing OSes. I didn't test this with Windows only machines, though.

franzflasch commented 3 years ago

FYI: I built the windows version in Linux with mingw using a cross-compilation CmakeToolchain file. But I also built it with VisualStudio2019, with the same issue.

Logan007 commented 3 years ago

Which Windows are you using? I tested this on Windows 10.

Same here for the VM (Windows 10, ver in cmd shows Microsoft Windows [Version 10.0.17763.107]). I used Visual Studio 2019 build tool as described in the Building doc, I am not too familiar with any other tool-chains on Windows.

The problem only occurs when mixing OSes.

Yes, I have mixed setup, too.

This is getting really strange now... It really looks like a key-passing error. Maybe some hard to find bug in the code, I would not deny this possibility. By the way, I will make the "EtherType" message more clear.

What about openSSL? I have compiled without OpenSSL support. But internal cipher implementation is tested against OpenSSL edges and has proven to be 100 percent compatible (SPECK and Twofish are not supported by OpenSSL anyway).

Has anyone else experienced such a bug before or is currently experiencing it? Please, speak up!

franzflasch commented 3 years ago

I even added a debug code to n2n_transop_aes_int() like this:

int n2n_transop_aes_init (const n2n_edge_conf_t *conf, n2n_trans_op_t *ttt) {

    transop_aes_t *priv;
    const u_char *encrypt_key = (const u_char *)conf->encrypt_key;
    size_t encrypt_key_len = strlen(conf->encrypt_key);

    int i = 0;
    printf("KEYLEN %d\n", encrypt_key_len);
    for(i=0;i<encrypt_key_len;i++)
    {
        printf("KEY: %d '%x'\n", i, encrypt_key[i]);
    }

    memset(ttt, 0, sizeof(*ttt));
    ttt->transform_id = N2N_TRANSFORM_ID_AES;

    ttt->tick         = transop_tick_aes;
    ttt->deinit       = transop_deinit_aes;
    ttt->fwd          = transop_encode_aes;
    ttt->rev          = transop_decode_aes;

    priv = (transop_aes_t*)calloc(1, sizeof(transop_aes_t));
    if(!priv) {
        traceEvent(TRACE_ERROR, "n2n_transop_aes_init cannot allocate transop_aes_t memory");
        return -1;
    }
    ttt->priv = priv;

    // setup the cipher and key
    return setup_aes_key(priv, encrypt_key, encrypt_key_len);
}

Just to be sure it is the exact same key on both machines. This is the output:

Linux:

01/Feb/2021 13:16:25 [edge_utils.c:227] Number of supernodes in the list: 1
01/Feb/2021 13:16:25 [edge_utils.c:229] supernode 0 => 172.17.0.1:7037
KEYLEN 4
KEY: 0 '74'
KEY: 1 '65'
KEY: 2 '73'
KEY: 3 '74'
01/Feb/2021 13:16:25 [edge.c:940] Use manually set IP address.

Windows:

01/Feb/2021 13:16:38 [n2n_new/src/edge_utils.c:227] Number of supernodes in the list: 1
01/Feb/2021 13:16:38 [/n2n_new/src/edge_utils.c:229] supernode 0 => 172.17.0.1:7037
KEYLEN 4
KEY: 0 '74'
KEY: 1 '65'
KEY: 2 '73'
KEY: 3 '74'
01/Feb/2021 13:16:38 [/n2n_new/src/edge.c:940] Use manually set IP address.

Now I am running out of ideas...

Logan007 commented 3 years ago

Maybe, and I hardly dare to say it, could you please check if your Linux machine has another edge running in the background? It happened to myself once when I ran without -f first and then forgot about the one in the background...

You probably are aware of how to do it: ps -Af | grep edge

franzflasch commented 3 years ago

Yeah this happened to me with the supernode too. However I already checked that there is no instance running in the background.

Logan007 commented 3 years ago

Now I am running out of ideas...

Honestly, same here.

Logan007 commented 3 years ago

Any VLAN (tags) in use which could cause different ethernet header format?

franzflasch commented 3 years ago

I've now I tried again with a fresh Windows 10 VM and installed all the tools exactly as it is described in the doc. Now it works for me too! :)

Don't know what I did wrong with my first build with Visual Studio...

It's still strange that it does not work when building with mingw, though.

Anyway, thanks for your great support! :+1:

Logan007 commented 3 years ago

Thank you for reporting! We really need this feedback from the community.

I just hope that there isn't a deep bug in n2n. Anyway, let's blame Windows for now! :wink: