How do I connect "emane0" from one Virtual Machine (vm) to "emane0" on another Virtual Machine (vm)?

WorksWellWithOthers commented 5 years ago

I currently have EMANE setup on a host machine with 6 NEMs. I'm trying to replace two of the NEMs with Virtual Machines running on the same host.

VM-1 and VM-2 can ping each other via the Virtual Network Interface Bridge (ens10).

If I launch the LXC containers with ./demo-start I can SSH into [VM1]Node-1 and ping [VM1]10.100.0.2. Outside of the nodes VM-1 and VM-2 can ping each other. I don't know how to ping [VM2]10.100.0.2 from VM-1 or [VM1]Node-1 or [VM1]Node-2.

Would using "brctl" to bridge ens10 with emanenode0 be the proper way to go about this?

The other thing I tried was to change"otamanagerdevice" and "eventservicedevice" values to "ens10". Then launch "sudo emane platform1.xml -r -d -l 3 -f persist/1/var/log/emane.log" in VM-1 and "sudo emane platform2.xml -r -d -l 3 -f persist/2/var/log/emane.log" in VM-2. Followed by "brctl addbr test", "brctl addif test ens10 emane0". Thinking I can "ping 10.100.0.2" from VM-1 afterwards. However, if I do this I can't ping 10.100.02 or VM-2 from VM-1.

The end goal is to send packets from VM-1 to VM-2 with the EMANE network being in the middle conducting events.

sgalgano commented 5 years ago

Assuming you want VM-1 and VM-2 acting as hosts hanging off of two radios being emulated with emane in two LXCs:

[ VM-1 ] <-- lan1 --> [ Radio LXC 1 (NEM 1) ] <--- OTA --> [ Radio LXC 2 (NEM 2) ] <-- lan2 --> [ VM-2 ]

Add an additional interface in the container configuration of both Radio LXCs to create a lan connection to the VMs.
On the host system, bridge the host system side of the LXC lan vif with the host system side of its respective VM lan interface vif. Do this for both lan1 and lan2.
Add the necessary routes or default route to each VM.

WorksWellWithOthers commented 5 years ago

To check my understanding of what I need to do:

Figure out how to create a lan connection to the VMs using an interface in the container configuration. a. Is this something I have to do outside of EMANE? Configure the LXC directly?
Figure out how to bridge the newly created lans with the existing host lan vif.
Figure out how to route to each VM. a. Is this something I have to do outside of EMANE? Configure OLSR directly?

Is it similar to this pdf? "Both of our EMANE clusters have three virtual interfaces per virtual machine (eth0, eth1, eth2) each connected a separate VLAN which is connected to a tap on the host. The host side of the eth0 taps are added to a bridge with the physical eth0 interface of the host, connecting the eth0 interfaces of the virtual machines to the management network. The virtual eth1 taps are handled in the same way (being bridged with the host event interface or bonded channel)." From Page-3: https://pdfs.semanticscholar.org/65ff/cad6cfeb8c9cd6e161e7d19ce92466960ed1.pdf

It seems I have a lot of learning to do since I do not yet know how to go about steps 1, 2, and 3.

Once this is successful I would like to add physical radios (VM) and have EMANE control the events to simulate various scenarios.

sgalgano commented 5 years ago

I just added an additional example (exp-03) to the letce2-tutorial. This example produces the topology you are interested in investigating but uses LXCs in place of VMs for the application hosts. Building and reviewing the configuration may prove helpful.

[ host-1 ] <-- lan --> [ node-1 ] <-- OTA --> [ node-2 ] <-- lan --> [ host-2 ]
   .2      10.98.1.0/24    .1                     .1     10.98.2.0/24   .2

WorksWellWithOthers commented 5 years ago

I studied the letce2-tutorial and the configuration in exp-03 helped clear a lot of the initial fog in my head. I'm having a difficult time trying to create physical interfaces for each container and setting up the correct routing for them. I do feel like I improved my understanding a bit but I'm not sure if I'm close to solving this or if I still have a long ways to go in terms of learning to figure it out.

From inside [VM-1][Node-1] (10.99.0.1), "ip route add default via 10.99.0.1", does this create LAN1?

Use Case: ssh [VM-1][Node-1] ping 10.100.0.3 ([VM-2][Radio-1])

What I tried to build: 10.100.0.1 = [VM-1][Radio-1] 10.100.0.2 = [VM-1][Radio-2] 10.100.0.3 = [VM-2][Radio-1] 10.100.0.4 = [VM-2][Radio-2]

Looking at this:

[ host-1 ] <-- lan --> [ node-1 ] <-- OTA --> [ node-2 ] <-- lan --> [ host-2 ]
   .2      10.98.1.0/24    .1                     .1     10.98.2.0/24   .2

It looks like my issue may be related to not having my Nodes in the same subnet (not sure how to do this). My [VM-1][Node-1] is "10.99.0.1" and my [VM-2][Node-1] is also "10.99.0.1".

sgalgano commented 5 years ago

The 10.99.0.0/16 is a back channel network for controlling the emulation. You should never have to add routes for that.

In exp-03, each radio node has a lan interface for one or more hosts to hang off. The olsr instance running in the radio node advertises this host network. Looking at the routing table of any radio node shows both the OTA addresses 10.100.0/24 of the other radio nodes and the host networks being advertised by those radio nodes.

[me@host exp-03]$ ssh node-1
[me@node-1 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.98.1.0       0.0.0.0         255.255.255.0   U     0      0        0 lan0
10.98.2.0       10.100.0.2      255.255.255.0   UG    2      0        0 emane0
10.99.0.0       0.0.0.0         255.255.255.0   U     0      0        0 backchan0
10.100.0.0      0.0.0.0         255.255.255.0   U     0      0        0 emane0
10.100.0.2      10.100.0.2      255.255.255.255 UGH   2      0        0 emane0
224.0.0.0       0.0.0.0         240.0.0.0       U     0      0        0 bmf0

If your desire is to replace the LXC hosts (host-1 and host-2) in exp-03 with VMs, you will need to set the address of one of the interfaces inside each VM to be on its respective radio lan. You will also need to add the host side of the VM's interface to the corresponding linux bridge containing its radio node's lan vif.

For example, exp-03 sets up a bridge for each radio node lan: r1_0 and r2_0.

[me@host ~]$ brctl show
bridge name bridge id       STP enabled interfaces
letce0      8000.fe03e17d031d   no      aveth.1.0
                            aveth.2.0
                            aveth.3.0
                            aveth.4.0
r1_0        8000.fe6d372e1fce   no      aveth.1.1
                            aveth.4.1
r2_0        8000.fe9139906998   no      aveth.2.1
                            aveth.3.1

Where, aveth.1.1 is the host side of the radio node node-1's lan0 and aveth.4.1 is the host side of the application-host host-1's lan0; aveth.2.1 is the host side of the radio node node-2's lan0 and aveth.3.1 is the host side of the application-host host-2's lan0.

WorksWellWithOthers commented 5 years ago

Thank you for your patience. It seems I was doing things incorrectly by having emane running inside both VM-1 and VM-2. I will be reattempting everything with emane running outside of VM-1 and VM-2 on the host machine. I'll update here as I make progress.

WorksWellWithOthers commented 5 years ago

I've reattempted everything and organized what I did. Doesn't work yet but I'm learning so that's a plus.

It looks like radio-1 and radio-2 are associated with veth1.2 and veth2.2 (not sure how to confirm this). I created and added veth1.2 to radio1Bridge and veth2.2 to radio2Bridge using brctl on the host machine. I then added radio1Bridge to VM-1 as a Virtual Network Interface and it now shows up as ens11.

When I run "ip route get 10.100.0.1" (radio-1) on VM-1 it responds with "10.100.0.1 dev ens10 src 10.100.0.190". When I run "ip route get" on the host machine it responds with "10.100.0.1 dev virbr0 src 10.100.0.100". I was surprised that the response from the host machine wasn't "10.100.0.1 dev veth1.2 src 10.100.0.100".

I'm trying to ping radio-1(running inside an LXC on the host machine -- emane-tutorial/0) from VM-1. The host can't ping radio-1 without first SSHing into Node-1. This is the correct behavior. I'm just bringing this up to ask if VM-1 will be able to ping radio-1 when the host can't do it without remoting into Node-1.

I'm wondering if I'm on the right track. I'm thinking of tinkering with "ip route add" in hopes of things eventually(magic) working. I'm also having doubts that veth1.2 and veth2.2 are the correct interfaces for me to feed to the VMs.

I think I need to somehow bridge emane0 from within Node-1 and make that available to the host then somehow feed that to the VM.

That's what I thought I did by bridging veth1.2 on the host and making it a NIC on VM-1. I did not setup routes (I was hoping that part was handled automatically -- wishful thinking).

Edit: I think I confused myself trying to simplify things. The end goal is to replace one of the radios with a VM. I think I lost sight of this...

some time passed

I am somehow able to ping a radio running on the host from a VM running on the host. I am able to ping that same radio from the host without SSHing into the node first.

I think this was the behavior I wanted... I just don't remember how I got here.

I think it was a combination of:

Changing the VM IP to be the same as the Radio IP: ifconfig interface_name radio_ip netmask 255.255.255.0
I tried throwing the VM into the same Bridge as the Radio on the host machine. I don't remember exactly what I did. I think I threw what I thought was the radio interface to virbr0. Feeling that I messed things up or that I wasn't doing things correctly I tried to undo everything.
I tried to reset the IP (auto-assign): ifconfig interface_name down ifconfig interface_name up
Started pinging VMs, Hosts, and Radios to make sure things were responsive and unresponsive to confirm expected behavior.
Surprised that I can ping a radio like I wanted I am trying to retrace my steps.

Maybe I added a route and forgot to remove it? route add default gw <host_ip> Probably not because adding it didn't give me the behavior I wanted.

I must have done something to the host. VM-2 which I haven't touched in a long time is now able to ping that same radio.

Thinking about this a bit, this is probably not what I want. If things were setup correctly, I would expect to be connected to the other radios through the one I'm able to ping.

Update: It looks like I was able to ping not the radio but the VM that is now using the same IP as the radio. I thought I removed that duplicate IP I set up.

Next thing to try is to figure out how to setup multiple interfaces on the NEM / add a physical interface to the existing containers. Maybe in doing so I can replace the Radio with a VM eventually...

WorksWellWithOthers commented 5 years ago

Is it possible to add a physical interface to each of the containers through the .xml scripts in the emane-tutorials?

sgalgano commented 5 years ago

You will need to place the physical interface on the host system in a bridge and then add the container vif to the bridge via lxc.conf. This is outside the scope of emane.

WorksWellWithOthers commented 5 years ago

Note to self: There's probably other ways to do it but the one method I found was through the realization that lxc.conf files are pointed to at the call of lxc-create. Then I found a file "persist/1/var/lxc.conf.1" which turned out to be a generated file. Since "demo-ctl-host" is responsible for this generated file you can edit it to add the radio interface to a bridge:

lxc.network.type=veth lxc.network.name=eth2 ... lxc.network.link = name_of_bridge

But this adds all the Radios to that one bridge.

To have individual radios go to a separate bridge you have to add a condition check for $nodeid then provide the "lxc.network.link = name_of_bridge" when that condition is met. It's probably better to make this a function instead since this will duplicate a lot of code.

Example bash syntax: if [[ $nodeid -eq 1]]; then cat < persist/$nodeid/var/run/lxc.conf.$nodeid lxc.ustname=$name ... lxc.network.link = name_of_bridge EOF else cat < persist/$nodeid/var/run/lxc.conf.$nodeid ... EOF fi

I'm not sure if this is much different from just launching "./demo-start" without editing any files then just bridging veth1.2 to a bridge.

"brctl addif name_of_bridge veth1.2"

WorksWellWithOthers commented 5 years ago

I explored creating another interface on the LXC containers. I'm thinking once I can establish a connection between an interface on the LXC container with an interface on my VM I can work on next steps to replacing the container with the VM.

I was able to successfully have the democtl-host create another VETH (VETH1.3). Then I realized I have to actually remote into Node-1 then launch Emane to get a NEM (radio). ETH1 and ETH2 are the LXC interfaces. Which means I didn't create the interfaces in the LXC containers like I thought I did.

Questions:

What exactly is VETH?
How is EMANE0 created? (I'm relating EMANE0 with a Radio)
Do I need to know these details if I am attempting to replace one of the Radios with a VM? (Making sure I'm not going the wrong direction)
Is there a difference between adding VETH1.2 (container vif) to the bridge (myBridge) via lxc.conf versus doing it after the container creation using "brctl addif myBridge VETH1.2"?

Here are some other things I discovered and I am confused about...

10.99.0.100 -- Emanenode0 is a bridge containing all the node (backchannels). 10.99.0.1 -- You have to go into one of these nodes to create a Radio (NEM / emane0) by launching Emane. 10.100.0.1 -- emane0 is the Radio created by launching emane platform.xml

I think 10.99.0.100 can also be used as the Host IP.

What is the difference between the Host-IP and 10.99.0.100?

sgalgano commented 5 years ago

There is a chance things have become more complicated than they need to be. If your goal is to setup an experiment using VMs instead of containers, you don't need any of this. If you are using a single VM instance as an emulated radio, all you need is to start multiple VMs with your virtualizer and configure them with a network they can communicate over. As long as you have connectivity between them, you can configure and run emane on each. If you look at emane-tutorial 0, you will find examples of the configuration and commands you need to run in each VM.

If your goal is to mix VM instances and LXCs in the same experiment using the same server, you have to deal with the items we have been discussing in this thread.

My suggestion would be to stick with either all VMs or all LXCs until you have a better handle on virtualization plumbing.

WorksWellWithOthers commented 5 years ago

I'm now using only VMs to get a better handle on the virtualization plumbing. I'm assuming I have to launch emane without ./demo-start?

VM-1: emane platform1.xml -r -d -l 3 -f persist/1/var/log/emane.log VM-2: emane platform2.xml -r -d -l 3 -f persist/2/var/log/emane.log

sgalgano commented 5 years ago

The emane-tutorial is designed to give you 10 nodes with network stack isolation (using LXCs) to walk through simple emane experiments. Don't confuse the simple demo test flow with emane. emane is just an application (see FAQ).

How you run your experiments and control them is outside the scope of the emulator. We have some open source tools that do test control with varying degrees of formality (etce, letce2), if you are looking for something.

WorksWellWithOthers commented 5 years ago

Thank you for the help. I now have a VM acting as a host hanging off of radios being emulated with emane.

adjacentlink / emane

How do I connect "emane0" from one Virtual Machine (vm) to "emane0" on another Virtual Machine (vm)? #102