search

sonata-nfv / son-emu

Attention! Legacy! This repo will be replaced with https://github.com/containernet/vim-emu
https://github.com/containernet/vim-emu
Apache License 2.0
36 stars 32 forks source link

addLink error at 113th vnf created in same datacenter #265

Closed cosminandreicontu closed 6 years ago

cosminandreicontu commented 6 years ago

Hello, I am using the virtual image SONATA Emulator Demo VM 2017 Download (~4-5GB) from https://github.com/sonata-nfv/son-tutorials/tree/master/upb-containernet-emulator-summerschool-demo, and I had encountered the following issue: After creating one datacenter, I was trying to start multiple vnfs (around 150 in the same datacenter), and at the 113th vnf the emulator gave me the following error:

ERROR:root:API error. Traceback (most recent call last): File "/home/sonata/son-emu/src/emuvim/api/rest/compute.py", line 74, in put compute_name, image=image, command=command, network=nw_list) File "/home/sonata/son-emu/src/emuvim/dcemulator/node.py", line 276, in startCompute self.net.addLink(d, self.switch, params1=nw, cls=Link, intfName1=nw.get('id')) File "/home/sonata/son-emu/src/emuvim/dcemulator/net.py", line 185, in addLink link = Containernet.addLink(self, node1, node2, params) File "/home/sonata/containernet/mininet/net.py", line 402, in addLink link = cls( node1, node2, options ) File "/home/sonata/containernet/mininet/link.py", line 440, in init link=self, mac=addr1, params1 ) File "/home/sonata/containernet/mininet/link.py", line 61, in init self.config( params ) File "/home/sonata/containernet/mininet/link.py", line 194, in config self.setParam( r, 'setIP', ip=ip ) File "/home/sonata/containernet/mininet/link.py", line 177, in setParam result = f( value ) File "/home/sonata/containernet/mininet/link.py", line 77, in setIP return self.ifconfig( ipstr, 'up' ) File "/home/sonata/containernet/mininet/link.py", line 69, in ifconfig return self.cmd( 'ifconfig', self.name, args ) File "/home/sonata/containernet/mininet/link.py", line 65, in cmd return self.node.cmd( args, *kwargs ) File "/home/sonata/containernet/mininet/node.py", line 959, in cmd out = self.dcli.exec_start(exec_dict) File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 35, in wrapper return f(self, args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/docker/api/exec_api.py", line 144, in exec_start return self._read_from_socket(res, stream) File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 357, in _read_from_socket return six.binary_type().join(frames_iter(socket)) File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 73, in frames_iter n = next_frame_size(socket) File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 60, in next_frame_size data = read_exactly(socket, 8) File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 45, in read_exactly next_data = read(socket, n - len(data)) File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 27, in read select.select([socket], [], []) ValueError: filedescriptor out of range in select()

After this error, if I try to create other vnfs in the same datacenter, the following error appears:

ERROR:root:API error. Traceback (most recent call last): File "/home/sonata/son-emu/src/emuvim/api/rest/compute.py", line 74, in put compute_name, image=image, command=command, network=nw_list) File "/home/sonata/son-emu/src/emuvim/dcemulator/node.py", line 276, in startCompute self.net.addLink(d, self.switch, params1=nw, cls=Link, intfName1=nw.get('id')) File "/home/sonata/son-emu/src/emuvim/dcemulator/net.py", line 185, in addLink link = Containernet.addLink(self, node1, node2, params) File "/home/sonata/containernet/mininet/net.py", line 402, in addLink link = cls( node1, node2, options ) File "/home/sonata/containernet/mininet/link.py", line 430, in init node1, node2, deleteIntfs=False ) File "/home/sonata/containernet/mininet/link.py", line 474, in makeIntfPair deleteIntfs=deleteIntfs ) File "/home/sonata/containernet/mininet/util.py", line 202, in makeIntfPair ( intf1, intf2, cmdOutput ) ) Exception: Error creating interface pair (vnf227-eth0,dc1.s1-eth113): RTNETLINK answers: File exists

This problem appeared each time I had done this test (using other datacenter or restarting the virtual machine)

Could you please provide a feedback and/or a solution regarding this issue?

Thank you!

mpeuster commented 6 years ago

Interesting. Looks like docker is not able connect to the 113th container and configure it properly. The issue might be related to the particular version of the emulator and the actual setup inside the demo VM.

I would suggest to use the latest version of the emulator (which is now called vim-emu) and part of the OSM project as mentioned in the README.md of this repository:

You should install it in a fresh Ubuntu 16:04 machine with at least 8 GB memory if you want to run so many emulated VNFs. Pretty sure that it should work with the latest version of the emulator since we used it recently to do some experiments with up to 256 VNFs.

cosminandreicontu commented 6 years ago

Hello Mr. Peuster,

Thank you for your response.

I tried to install vim-emu by following the installation steps:

sudo apt-get install lxd lxd-client

$ wget -O install_osm.sh "https://osm.etsi.org/gitweb/?p=osm/devops.git;a=blob_plain;f=installers/install_osm.sh;hb=HEAD" $ chmod +x install_osm.sh

$ ./install_osm.sh --lxdimages --vimemu

After running the last command, the following error appears

luni 26 februarie 2018, 22:28:47 +0200 container_exec_stderr: exec in RO "curl -sL -w %{http_code}\n http://www.google.com/ -o /dev/null luni 26 februarie 2018, 22:28:47 +0200 wait_container_up: waiting for container RO to start luni 26 februarie 2018, 22:28:47 +0200 wait_container_up: expected '200' in Permission denied, are you in the lxd group?

Could you please tell me the appropriate configuration of lxd and if any other prerequisites are needed?

I also tried the manual installation having the same result.

Thank you!

mpeuster commented 6 years ago

I guess you do not plan to use OSM, right? So you should go with the manual bare metal installation that installs vim-emu without OSM:

see Option 1: Bare-metal installation (https://osm.etsi.org/wikipub/index.php/VIM_emulator#Option_1:_Bare-metal_installation)

Option 1: Bare-metal installation

 $ sudo apt-get install ansible git aptitude

Step 1: Containernet installation

 $ cd
 $ git clone https://github.com/containernet/containernet.git
 $ cd ~/containernet/ansible
 $ sudo ansible-playbook -i "localhost," -c local install.yml

Step 2: vim-emu installation

 $ cd
 $ git clone https://osm.etsi.org/gerrit/osm/vim-emu.git
 $ cd ~/vim-emu/ansible
 $ sudo ansible-playbook -i "localhost," -c local install.yml
 $ cd ~/vim-emu
$ sudo python setup.py install

This is the nightly tested way of installing vim-emu which should work in any case.

Important: This should be performed on a FRESH Ubuntu 16.04 LTS installation.