Unable to ping the other peer

[saket424]

I followed the wonderful instructions in the demo https://screencast-o-matic.com/watch/cbjXbPlmdl with the latest ipop-vpn_rel17.08(ubuntu)

However, when I try it the individual ipop_tap0 interfaces do come up on the respective hosts but they are unable to discover each other and ping each other

Here are the controller logs and it stays stuck in the searching state and does not find the other peer

Any ideas?? I am using my own xmpp server if that matters

root@mike-desktop:/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/logs# head ctrl.log [20171114 22:32:17.420] INFO:TincanInterface: TincanInterface Loaded [20171114 22:32:17.425] INFO:TincanInterface: Creating Tincan control response link [20171114 22:32:17.426] INFO:TincanInterface: Setting Tincan log level to DEBUG [20171114 22:32:17.432] INFO:TincanInterface: Creating Vnet ipop_tap0 [20171114 22:32:17.433] INFO:TincanInterface: Ignoring interfaces [u'ipop_tap0'] [20171114 22:32:17.434] INFO:LinkManager: LinkManager Loaded [20171114 22:32:17.434] INFO:XmppClient: Key-ring module not installed. [20171114 22:32:17.435] DEBUG:XmppClient: Started XMPP handling [20171114 22:32:17.435] INFO:XmppClient: XmppClient module Loaded [20171114 22:32:17.435] INFO:ArpCache: ArpCache Loaded root@mike-desktop:/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/logs# head -50 ctrl.log [20171114 22:32:17.420] INFO:TincanInterface: TincanInterface Loaded [20171114 22:32:17.425] INFO:TincanInterface: Creating Tincan control response link [20171114 22:32:17.426] INFO:TincanInterface: Setting Tincan log level to DEBUG [20171114 22:32:17.432] INFO:TincanInterface: Creating Vnet ipop_tap0 [20171114 22:32:17.433] INFO:TincanInterface: Ignoring interfaces [u'ipop_tap0'] [20171114 22:32:17.434] INFO:LinkManager: LinkManager Loaded [20171114 22:32:17.434] INFO:XmppClient: Key-ring module not installed. [20171114 22:32:17.435] DEBUG:XmppClient: Started XMPP handling [20171114 22:32:17.435] INFO:XmppClient: XmppClient module Loaded [20171114 22:32:17.435] INFO:ArpCache: ArpCache Loaded [20171114 22:32:17.436] INFO:BaseTopologyManager: BaseTopologyManager Loaded [20171114 22:32:17.436] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'started', 'uid_mac_table': {}, 'ipop_state': {}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {}, 'link_type': {}, 'successor': {}} [20171114 22:32:17.436] INFO:BaseTopologyManager: ipop_tap0 P2P State: STARTED [20171114 22:32:17.437] INFO:BroadcastForwarder: BroadcastForwarder Loaded [20171114 22:32:17.437] INFO:TincanInterface: Received data from Tincan: Operation: CreateCtrlRespLink. Task status::{u'Message': u'Controller endpoint successfully created.', u'Success': True} [20171114 22:32:17.437] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'LinkManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 4, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4} [20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: ConfigureLogging. Task status::{u'Message': u'Tincan logging successfully configured.', u'Success': True} [20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: CreateVnet. Task status::{u'Message': u'', u'Success': True} [20171114 22:32:17.438] INFO:TincanInterface: Received data from Tincan: Operation: SetIgnoredNetInterfaces. Task status::{u'Message': u'', u'Success': True} [20171114 22:32:17.438] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'BaseTopologyManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 5, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4} [20171114 22:32:17.438] DEBUG:TincanInterface: Tincan Request: {'Request': {'Initiator': 'BaseTopologyManager', 'UID': None, 'MAC': '', 'Command': 'QueryNodeInfo', 'ProtocolVersion': 4, 'InterfaceName': u'ipop_tap0'}, 'TransactionId': 6, 'ControlType': 'TincanRequest', 'ProtocolVersion': 4} [20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'} [20171114 22:32:17.439] INFO:LinkManager: LM Local Node Info UID:dd7e18d641c0f18b4f80a1886b589cc2525a931e MAC:66F5F6C9A3EA IP4: 10.11.0.12 [20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'} [20171114 22:32:17.439] DEBUG:TincanInterface: current state of dd7e18d641c0f18b4f80a1886b589cc2525a931e : {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'} [20171114 22:32:27.419] DEBUG:LinkManager: Peer Nodes:: {} [20171114 22:32:27.429] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'started', 'uid_mac_table': {u'dd7e18d641c0f18b4f80a1886b589cc2525a931e': [u'66F5F6C9A3EA']}, 'ipop_state': {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'mac': u'66F5F6C9A3EA', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {u'66F5F6C9A3EA': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e'}, 'link_type': {}, 'successor': {}} [20171114 22:32:27.430] INFO:BaseTopologyManager: IPOP local state: dd7e18d641c0f18b4f80a1886b589cc2525a931e [20171114 22:32:27.430] INFO:BaseTopologyManager: ipop_tap0 P2P State: SEARCHING [20171114 22:32:37.419] DEBUG:LinkManager: Peer Nodes:: {} [20171114 22:32:37.431] DEBUG:BaseTopologyManager: BTM Table::{'discovered_nodes': [], 'p2p_state': 'searching', 'uid_mac_table': {u'dd7e18d641c0f18b4f80a1886b589cc2525a931e': [u'66F5F6C9A3EA']}, 'ipop_state': {'mac': u'66F5F6C9A3EA', '_uid': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e', '_ip4': u'10.11.0.12', '_fpr': u'sha-1 2B:56:C8:ED:59:11:C1:B5:13:3B:3A:40:96:87:CB:FF:97:40:40:6C', 'interface_name': u'ipop_tap0', 'type': 'local_state'}, 'ip_uid_table': {}, 'xmpp_client_code': u'XmppClient', 'mac': u'66F5F6C9A3EA', 'peer_uid_sendmsgcount': {}, 'GeoIP': '', 'mac_uid_table': {u'66F5F6C9A3EA': u'dd7e18d641c0f18b4f80a1886b589cc2525a931e'}, 'link_type': {}, 'successor': {}} [20171114 22:32:37.433] INFO:BaseTopologyManager: ipop_tap0 P2P State: SEARCHING ...

[saket424]

BTW, I am using the sample group-vpn config file as my starting point and filling in the xmpp credentials

The two peers are 10.11.0.11 and 10.11.0.12 and neither can ping the other

Also, my tincan log file is 0 size even when I enabled tincan_logging to 2 or 3 I only see the controller log at this point

[vahid-dan]

Hi @saket424 and thanks for your interest in IPOP VPN!

Make sure your XMPP username is in this format: <your_xmpp_user>@<your_xmpp_server>, not just <your_xmpp_user>. Could you please copy the XMPP config here? Keep in mind to remove the XMPP password after copying. :-)

Cheers, Vahid

[saket424]

Vahid, I tried with both my ejabberd xmpp server and the xmpp.jp xmpp server that you allude to in your tutorial -- Same issue in both instances. I definitely am using the correct set of xmpp credentials in the format your_xmpp_user>@<your_xmpp_server since the ejabberd server states I am logged in and I get valid ifconfig output for ipop_tap0

So I am confused whether you want the json file that the controller uses or the xmpp config file that ejabberd uses when you ask for the XMPP config

I think you mean you want the former. Here is the gvpn json file $/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/config# cat gvpn.json { "CFx": { "Model": "GroupVPN" }, "Logger": { "Enabled": true, "LogLevel": "DEBUG", "LogOption": "File", "LogFilePath": "./logs/", "CtrlLogFileName": "ctrl.log", "TincanLogFileName": "tincan.log", "LogFileSize": 5000000, "BackupLogFileCount": 5 }, "TincanInterface": { "ctrl_recv_port": 5801,
"localhost": "127.0.0.1", "ctrl_send_port": 5800,
"localhost6": "::1", "dependencies": ["Logger"], "Vnets": [ { "Name" : "ipop_tap0", "IP4": "10.11.0.12", "IP4PrefixLen": 16, "MTU4": 1200, "XMPPModuleName": "XmppClient", "TapName": "ipop_tap0", "Description": "Beta 2 Test Network", "IgnoredNetInterfaces": [ "ipop_tap0" ], "L2TunnellingEnabled": true } ], "Stun": [ "stun.l.google.com:19302" ], "Turn": [{ "Address": "turn:turnserver-ip:port", "User": "", "Password": "" }] }, "XmppClient": { "XmppDetails": [ { "Username": "dq12@xmpp.jp", "Password": "***", "AddressHost": "xmpp.jp", "Port": "5222", "TapName": "ipop_tap0", "AuthenticationMethod": "password", "AcceptUntrustedServer": true } ], "TimerInterval": 10 } }

---

$ifconfig ipop_tap0 ipop_tap0 Link encap:Ethernet HWaddr 16:8d:26:31:01:cf
inet addr:10.11.0.12 Bcast:10.11.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

[renatof]

Hello there,

It might also be possible that the two endpoints are behind NAT/firewalls where STUN traversal does not work (we have noticed problems and are debugging issues with TURN before, though I don't think you configured a turn server in your case)

--rf

On Wed, Nov 15, 2017 at 12:00 PM, saket424 notifications@github.com wrote:

Vahid, I tried with both my ejabberd xmpp server and the xmpp.jp xmpp server that you allude to in your tutorial -- Same issue in both instances. I definitely am using the correct set of xmpp credentials in the format

@ since the ejabberd server states I am logged in and I get valid ifconfig output for ipop_tap0 So I am confused whether you want the json file that the controller uses or the xmpp config file that ejabberd uses when you ask for the XMPP config I think you mean you want the former. Here is the gvpn json file $/home/mike/tincan-stuff/ipop-vpn_rel17.08(ubuntu)/config# cat gvpn.json { "CFx": { "Model": "GroupVPN" }, "Logger": { "Enabled": true, "LogLevel": "DEBUG", "LogOption": "File", "LogFilePath": "./logs/", "CtrlLogFileName": "ctrl.log", "TincanLogFileName": "tincan.log", "LogFileSize": 5000000, "BackupLogFileCount": 5 }, "TincanInterface": { "ctrl_recv_port": 5801, "localhost": "127.0.0.1", "ctrl_send_port": 5800, "localhost6": "::1", "dependencies": ["Logger"], "Vnets": [ { "Name" : "ipop_tap0", "IP4": "10.11.0.12", "IP4PrefixLen": 16, "MTU4": 1200, "XMPPModuleName": "XmppClient", "TapName": "ipop_tap0", "Description": "Beta 2 Test Network", "IgnoredNetInterfaces": [ "ipop_tap0" ], "L2TunnellingEnabled": true } ], "Stun": [ "stun.l.google.com:19302" ], "Turn": [{ "Address": "turn:turnserver-ip:port", "User": "***", "Password": "***" }] }, "XmppClient": { "XmppDetails": [ { "Username": "dq12@xmpp.jp", "Password": "***", "AddressHost": "xmpp.jp", "Port": "5222", "TapName": "ipop_tap0", "AuthenticationMethod": "password", "AcceptUntrustedServer": true } ], "TimerInterval": 10 } } ------------------------------ $ifconfig ipop_tap0 ipop_tap0 Link encap:Ethernet HWaddr 16:8d:26:31:01:cf inet addr:10.11.0.12 Bcast:10.11.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:576 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub , or mute the thread .

[saket424]

@renatof, I did use a turnserver in my config but blanked it out before pasting to the issue list just to protect the credentials from being abused.

[renatof]

I see, thanks for clarifying - I thought it might be the case, but wasn't sure.

We've noticed cases where turn did not work as expected after the tincan code was refactored to catch up with WebRTC; there's a chance it might be what is happening in your case. We are looking into this issue.

--rf

On Wed, Nov 15, 2017 at 2:24 PM, saket424 notifications@github.com wrote:

@renatof https://github.com/renatof, I did use a turnserver in my config but blanked it out before pasting to the issue list just to protect the credentials from being abused.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipop-project/ipop-project.github.io/issues/47#issuecomment-344700572, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlShcLXpS6lLP9Pvx4sAI3f0WwnAitsks5s2zp6gaJpZM4QeWw5 .

[saket424]

should'nt stun config be prefixed with a stun: ? "Stun": [ stun:stun.l.google.com:19302 ] similar to how the Turn has a turn: prefix Once you get it working on your end, please share with me the json file minus the specific credentials of course. Also, if there is any additional logging you would like me to enable, do let me know. BTW, I could not have to enable the tincan logging yet. I am guessing that log may have some more clues on what the issue is.

Thanks

[vahid-dan]

@saket424,

STUN server configuration is alright. Have you ever tried without the TURN? If not, please remove the lines related to the TURN server and try again. Don't forget to remove the semi-colon at the end of the STUN configuration line. We'll investigate the problem and get back to you.

Thanks, Vahid

[saket424]

@vahid-dan and @renatof Your initial hunch was correct. I created a bank of 4 xmpp user-ids dq11 through dq14 Till I updated their rosters that each could see each other's presence, they could not receive presence updates and hence they could not get each other's mac address Now with the roster update, I get further and i have gone beyond the searching state. It now enters the CONNECTING state but the connection does not fully succeed. Any ideas as to why not ? Your ipop-16.08.0 version of software succeeds while the newer ipop-17.08 version still does not successfully connect. I'll wait till you finish looking at the issue on your end.

[saket424]

removing the TURN config did not seem to help any. The relay candidates are no longer there but the net result is still it is stuck in the CONNECTING state and never successfully CONNECTED

[vahid-dan]

@saket424,

Thanks for spending time on IPOP. :-)

One more thing to test: Have you ever tried to connect all the nodes with just one XMPP user?

Sometimes it is hard for us to reproduce the errors and that makes the debugging difficult. In those cases, user reports such as what you currently do are very helpful. We really appreciate it.

Bests, Vahid

[rogora]

Hi everyone, I'm experiencing exactly the same problem as saket424. I think it might be a problem with TURN servers. I have tried with different TURN servers without success. I have also tried with xirsys TURN server, as suggested by the configuration video. Also, the peers connect correctly when they are behind the same NAT.

Finally, I confirm that I can connect successfully with version 16.08, in the same scenario where 17.08 fails.

[saket424]

@vahid-dan , Switching both peer config files to use the same xmpp user made no difference -- still stuck in the CONNECTING state

[vahid-dan]

Hi @saket424 and @rogora;

Our team is investigating the problem, and we most probably have an update for you by tomorrow night. BTW, in this case, even if the TURN works, the functionality is not ideal for IPOP VPN. If IPOP needs the TURN server, it means both sides of the connection are behind symmetric NAT or very restricted firewall. So you may want to check what prevents the IPOP nodes to create a direct tunnel. Firewall rules or router settings may be the cause of that.

We'll get back to you as soon as we have an explanation for you.

Bests, Vahid

[saket424]

@vahid-dan The fact that ipopvpn16.08 does work for @rogora and me without TURN is proof that we are not behind a symmetric NAT or restricted firewall. It is some problem specific to the new ipop17.08 build. Glad you are investigating.

Thanks Anand

[vahid-dan]

@saket424,

So you mean even without setting the TURN server configuration in the config file, IPOP 16.08 works for you? If so, you shouldn't need the TURN server to connect the nodes and if the nodes don't connect, the problem may be something else rather than the TURN server. Anyway, we hopefully know in the next 24 hours. :-)

Vahid

[saket424]

@vahid-dan

Precisely my point. Problem is elsewhere in 17.08 since I can connect in 16.08 with no turn server and i cannot connect in 17.08 with or without a turn server. Eagerly awaiting your findings in 24 hours

[kcratie]

@saket424 - We are still working to identify the root cause of the TURN bug with the v17.08 and resolve the issue. Independently of this, I would like to assist in determining what is causing your deployment to fail. To do this will need the controller and Tincan logs from both systems. Please ensure that the IPOP config json file sets the Logger LogLevel to DEBUG on both systems. The is just as was done in the previous config file you submitted. Also, feel free to increase LogFileSize if convenient. I may appear to be asking you repeat steps you have already taken, but from following the thread I can see that progressive changes have been made and I need a clear snapshot of the systems as they are now. Since we know tunnels are not being built via TURN, please omit the TURN parameters from the config. Once the peers are on separate subnets and they both have access to the STUN service, they will attempt NAT traversal. As the files will be verbose, it will be more convenient to put them in an archive as an attachment. If possible, please include your config file after removing any sensitive information. This should provide me with sufficient information pinpoint the cause of the failure. Also, you can try our IPOP test script [https://github.com/ipop-project/Release-Management/tree/master/Test/ipop-scale-test], which creates the IPOP environment within a single Ubuntu VM using containers. I expect this base case to work and can be used a sanity check to validate deploying the build successfully. Thanks for taking the time to work through this with us.

[saket424]

201.zip 204.zip

@kcratie Attached are the logs and associated config files for both ends of the tunnerl. per @vahid-dan 's suggestion, i used the same xmpp id for both config files(no turn involved). I suspect the ipop-scale-test will probably work but i have not tried it. I can confirm to you ipop16.08 worked with no turn specified, hence i expect ipop17.08 to establish the tunnel without any turn needed also but it remains stuck in the connecting state.

If you would like to do a more interactive test, send me a private email and we can do it together

Thanks Anand

[kcratie]

@saket424 I've made a commit to the Tincan and Controller repos to address the problem you have been experiencing. You will have to checkout the branch 'bh1' on both repos and build Tincan using its Makefile until the maintenance release is created. TURN should now work and NAT traversal should be improved. Please let me know the outcome.

[vahid-dan]

Hi @saket424 and @rogora;

While you're building the IPOP on the branch 'bh1', keep in mind that we have recently changed the way to get the WebRTC libraries for building IPOP. The documentation is up to date. Just follow them and you should be fine.

Bests, Vahid

[saket424]

@kcratie and @vahid-dan I tried the 'bh1' branch and I am happy to report that the tunnel did establish without the need for TURN. So NAT traversal with STUN seems to have improved

Tunnel creation takes a little while upto 2 minutes. I think it is probably because xmpp peer discovery is slow?

Also is this a layer2 tunnel if "L2TunnellingEnabled": true and otherwise defaults to layer3 ?

[kcratie]

@saket424 It is good to hear the fix worked for you. We are currently working on improvements to the signalling that's done using XMPP. We expect the new approach will reduce the connection setup time. Currently we only do layer 2 tunneling based on Ethernet frames. However, there is a road map to integrate layer 3 info into aspects of of the tunneling process.

[saket424]

Since only layer2 is supported, What does the json config option "L2TunnellingEnabled":false do? I assume it is just a no-op flag and it does not currently matter if it true or false and results in the same behavior either way?

[saket424]

@vahid-dan and @kcratie

I am unable to replicate the success i had in late November. From what I can gather, I am having xmpp issues and each peer is not getting the other's IP/mac information and hence does not even attempt a peer connection and stuck in the searching state. Can you think of why I am seeing xmpp unreliability and is anyone else also running into this issue of late?

Thanks Anand

[vahid-dan]

Hi @saket424;

We will have a maintenance release for our current version of IPOP and also a new release in a few weeks. Do you think you can wait till then? Maybe you should put your effort in the new release.

Thanks, Vahid