search

kimchi-project / ginger

WoK plugin for host management
Other
66 stars 41 forks source link

Failing to run any OVS Bridge related command after adding a new interface #236

Closed samhenri closed 8 years ago

samhenri commented 8 years ago

Running ovs-vsctl 2.3.0. -Created a bridge using ovs-vsctl add-br command; -Run "ovs-vsctl show":

ovs-vsctl show:
ff9fa361-f8b4-46a3-bf80-6494dcfa3d06
    Bridge ovsbr
        Port ovsbr
            Interface ovsbr
                type: internal

Then issue the following curl command: curl -k -u root -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges" -X GET

It shoud display the same ovs bridge as above;

-Add a new interface to the recently created bridge: curl -k -u root -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges/ovsbr/add_interface" -X POST -d'{"interface":"dummy0"}'

-Running "ovs-vsctl show" should return the following output:

Bridge ovsbr
        Port "dummy0"
            Interface "dummy0"
        Port ovsbr
            Interface ovsbr
                type: internal

Now if you try to run any other curl command again, it fails throwing a "500 Internal Server Error":

curl -k -u root  -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges"  -X GET
Enter host password for user 'root':
{
  "reason":"The server encountered an unexpected condition which prevented it from fulfilling the request.",
  "code":"500 Internal Server Error"
}

Wok log:

[21/Mar/2016:15:33:29] HTTP
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 217, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 61, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 407, in index
    return self.get(params)
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 385, in get
    resources = self._get_resources(flag_filter)
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 348, in _get_resources
    res.lookup()
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 172, in lookup
    self.info = lookup(*self.model_args)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 393, in lookup
    port_info = get_port_info(br_port)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 251, in get_port_info
    port_info = get_interface_info(iface_uuid)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 234, in get_interface_info
    info = parse_listinterface_output(out)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 156, in parse_listinterface_output
    interface[key] = get_stats_dict(value)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 137, in get_stats_dict
    key, value = key_values.split('=')
ValueError: need more than 1 value to unpack
127.0.0.1 - - [21/Mar/2016:15:33:29] "GET /plugins/ginger/ovsbridges HTTP/1.0" 500 513 "" "curl/7.29.0"
[21/Mar/2016:15:35:24] HTTP
Request Headers:
  AUTHORIZATION: Basic cm9vdDp2bXBhc3M=
  Content-Length: 22
  HOST: localhost
  CONNECTION: close
  Remote-Addr: 127.0.0.1
  X-REAL-IP: 127.0.0.1
  ACCEPT: application/json
  USER-AGENT: curl/7.29.0
  X-FORWARDED-FOR: 127.0.0.1
  Content-Type: application/json
[21/Mar/2016:15:35:24] HTTP
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 217, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 61, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 121, in wrapper
    self.lookup()
  File "/home/kimchi_master/wok/src/wok/control/base.py", line 172, in lookup
    self.info = lookup(*self.model_args)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 393, in lookup
    port_info = get_port_info(br_port)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 251, in get_port_info
    port_info = get_interface_info(iface_uuid)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 234, in get_interface_info
    info = parse_listinterface_output(out)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 156, in parse_listinterface_output
    interface[key] = get_stats_dict(value)
  File "/home/kimchi_master/wok/src/wok/plugins/ginger/model/ovsbridges.py", line 137, in get_stats_dict
    key, value = key_values.split('=')
ValueError: need more than 1 value to unpack
127.0.0.1 - - [21/Mar/2016:15:35:24] "POST /plugins/ginger/ovsbridges/ovsbr/del_interface HTTP/1.0" 500 513 "" "curl/7.29.0"
samhenri commented 8 years ago

I've removed the dummy0 interface and it is running ok again:

 curl -k -u root  -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges"  -X GET
Enter host password for user 'root':
[
  {
    "name":"DMZ",
    "ports":[]
  },
  {
    "name":"LAN",
    "ports":[]
  },
  {
    "name":"WAN",
    "ports":[]
  },
  {
    "name":"ovsbr",
    "ports":[]
  }
]#
danielhb commented 8 years ago

I wasn't able to reproduce it in a Fedora 23 using ovs-vsctl 2.4.0. What OS did you use?

[danielhb@arthas ginger]$ sudo ovs-vsctl list-br
[sudo] password for danielhb: 
[danielhb@arthas ginger]$ 
[danielhb@arthas ginger]$ sudo ovs-vsctl add-br ovsbr 
[sudo] password for danielhb: 
[danielhb@arthas ginger]$ curl -k -u root -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges" -X GET
Enter host password for user 'root':
[
  {
    "name":"ovsbr",
    "ports":[]
  }
][danielhb@arthas ginger]$ curl -k -u root -H "Content-Type: application/json" -H "Accept: appliction/json" "https://localhost:8001/plugins/ginger/ovsbridges/ovsbr/add_interface" -X POST -d'{"interface":"dummy0"}'
Enter host password for user 'root':
{
  "name":"ovsbr",
  "ports":[
    {
      "statistics":{
        "rx_over_err":"0",
        "tx_dropped":"0",
        "rx_packets":"0",
        "rx_frame_err":"0",
        "rx_bytes":"0",
        "tx_errors":"0",
        "rx_crc_err":"0",
        "collisions":"0",
        "rx_errors":"0",
        "tx_bytes":"0",
        "rx_dropped":"0",
        "tx_packets":"0"
      },
      "name":"dummy0",
      "mac_in_use":"5e:92:dd:df:78:b7",
      "link_state":"down",
      "admin_state":"down",
      "type":"interface"
    }
  ]
}[danielhb@arthas ginger]$ sudo ovs-vsctl show 
227ef3fa-fa39-4d13-8a47-34386bdf0785
    Bridge ovsbr
        Port "dummy0"
            Interface "dummy0"
        Port ovsbr
            Interface ovsbr
                type: internal
    ovs_version: "2.4.0"
[danielhb@arthas ginger]$ curl -k -u root  -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges"  -X GET
Enter host password for user 'root':
[
  {
    "name":"ovsbr",
    "ports":[
      {
        "statistics":{
          "rx_over_err":"0",
          "tx_dropped":"0",
          "rx_packets":"0",
          "rx_frame_err":"0",
          "rx_bytes":"0",
          "tx_errors":"0",
          "rx_crc_err":"0",
          "collisions":"0",
          "rx_errors":"0",
          "tx_bytes":"0",
          "rx_dropped":"0",
          "tx_packets":"0"
        },
        "name":"dummy0",
        "mac_in_use":"5e:92:dd:df:78:b7",
        "link_state":"down",
        "admin_state":"down",
        "type":"interface"
      }
    ]
  }
]
danielhb commented 8 years ago

Tested in an Ubuntu 15.10 VM, didn't reproduce the error too. ovs-vsctl version 2.4.0

danielhb@ubuntu1510vm:~$ sudo ovs-vsctl show
780b120f-4aff-4e1e-89fe-601986d0fe76
    Bridge ovsbr
        Port ovsbr
            Interface ovsbr
                type: internal
    ovs_version: "2.4.0"
danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ sudo ovs-vsctl list-br
ovsbr
danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ curl -k -u danielhb -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges" -X GET
Enter host password for user 'danielhb':
[
  {
    "name":"ovsbr",
    "ports":[]
  }
]danielhb@ubuntu1510vm:~$ sudo modprobe dummy
danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ curl -k -u danielhb -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges/ovsbr/add_interface" -X POST -d'{"interface":"dummy0"}'
Enter host password for user 'danielhb':
{
  "name":"ovsbr",
  "ports":[
    {
      "statistics":{
        "rx_over_err":"0",
        "tx_dropped":"0",
        "rx_packets":"0",
        "rx_frame_err":"0",
        "rx_bytes":"0",
        "tx_errors":"0",
        "rx_crc_err":"0",
        "collisions":"0",
        "rx_errors":"0",
        "tx_bytes":"0",
        "rx_dropped":"0",
        "tx_packets":"0"
      },
      "name":"dummy0",
      "mac_in_use":"76:eb:df:ba:01:b1",
      "link_state":"down",
      "admin_state":"down",
      "type":"interface"
    }
  ]
danielhb@ubuntu1510vm:~$ curl -k -u danielhb -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges" -X GET
Enter host password for user 'danielhb':
[
  {
    "name":"ovsbr",
    "ports":[
      {
        "statistics":{
          "rx_over_err":"0",
          "tx_dropped":"0",
          "rx_packets":"0",
          "rx_frame_err":"0",
          "rx_bytes":"0",
          "tx_errors":"0",
          "rx_crc_err":"0",
          "collisions":"0",
          "rx_errors":"0",
          "tx_bytes":"0",
          "rx_dropped":"0",
          "tx_packets":"0"
        },
        "name":"dummy0",
        "mac_in_use":"76:eb:df:ba:01:b1",
        "link_state":"down",
        "admin_state":"down",
        "type":"interface"
      }
    ]
  }
]danielhb@ubuntu1510vm:~$ 
danielhb@ubuntu1510vm:~$ sudo ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.4.0
Compiled Oct 16 2015 09:22:33
DB Schema 7.12.1
danielhb@ubuntu1510vm:~$
samhenri commented 8 years ago

Tried on Ubuntu 15.10 and CentOS 7.2, both of them running ovs-vsctl 2.3.0.

danielhb commented 8 years ago

It can be related to the output of the ovs-vsctl commands in 2.3.0 version

Can you post the output of sudo ovs-vsctl list Interface dummy0in your setup? I'll try to find out a machine with ovs-vsctl 2.3.0 myself to test it too.

samhenri commented 8 years ago

I think I figured. Whenever I try to create a new interface, the cli hangs until I stop it. Checking for openvswitch status returns this:

systemctl status openvswitch -l
● openvswitch.service - LSB: Open vSwitch switch
   Loaded: loaded (/etc/rc.d/init.d/openvswitch)
   Active: active (running) since Mon 2016-03-28 08:08:29 EDT; 1min 8s ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/openvswitch.service
           ├─652 /bin/sh /usr/share/openvswitch/scripts/ovs-ctl start --system-id=random
           ├─653 tee -a /var/log/openvswitch/ovs-ctl.log
           ├─714 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
           ├─715 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
           └─716 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor

Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: [FAILED]
Mar 28 08:08:29 localhost.localdomain ovs-vsctl[16719]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.6.0
Mar 28 08:08:29 localhost.localdomain ovs-vsctl[16719]: ovs|00002|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied)
Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Permission denied)
Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: ovs-vswitchd is already running.
Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: Enabling remote OVSDB managers 2016-03-28T12:08:29Z|00001|unixctl|WARN|failed to connect to /var/run/openvswitch/ovsdb-server.11616.ctl
Mar 28 08:08:29 localhost.localdomain ovs-appctl[16721]: ovs|00001|unixctl|WARN|failed to connect to /var/run/openvswitch/ovsdb-server.11616.ctl
Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: ovs-appctl: cannot connect to "/var/run/openvswitch/ovsdb-server.11616.ctl" (Permission denied)
Mar 28 08:08:29 localhost.localdomain openvswitch[16693]: [FAILED]
Mar 28 08:08:29 localhost.localdomain systemd[1]: Started LSB: Open vSwitch switch.
danielhb commented 8 years ago

Have you ran the add interface command with 'sudo' ?

If yes, then it looks like a configuration issue. Perhaps it's worth trying to update do 2.4.0?

samhenri commented 8 years ago

Tried with Fedora 20, same behavior. Installed default package running yum install openvswitch, manually added bridges by running:

ovs-vsctl add-br LAN
ovs-vsctl add-br WAN
ovs-vsctl add-br DMZ

Listed bridges:

ovs-vsctl show
1e2dbd55-9e96-4e55-aad2-464e5f41466b
    Bridge DMZ
        Port DMZ
            Interface DMZ
                type: internal
    Bridge WAN
        Port WAN
            Interface WAN
                type: internal
    Bridge LAN
        Port LAN
            Interface LAN
                type: internal
    ovs_version: "2.3.1-git4750c96"

Also added by curl command:

curl -k -u root  -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges"  -X POST -d'{"name":"ovsbr"}'
Enter host password for user 'root':
{
  "name":"ovsbr",
  "ports":[]
}#

Listed bridges again:

ovs-vsctl show
1e2dbd55-9e96-4e55-aad2-464e5f41466b
    Bridge DMZ
        Port DMZ
            Interface DMZ
                type: internal
    Bridge WAN
        Port WAN
            Interface WAN
                type: internal
    Bridge ovsbr
        Port ovsbr
            Interface ovsbr
                type: internal
    Bridge LAN
        Port LAN
            Interface LAN
                type: internal
    ovs_version: "2.3.1-git4750c96"

Wok response when listing the OVS bridges:

[
  {
    "name":"DMZ",
    "ports":[]
  },
  {
    "name":"LAN",
    "ports":[]
  },
  {
    "name":"WAN",
    "ports":[]
  },
  {
    "name":"ovsbr",
    "ports":[]
  }
]
danielhb commented 8 years ago

The Fedora 20 behavior you posted is absolutely right. 'ovs-vsctl show' shows the bridge as a port, but the backend are fetching the ports using ovs-vsctl list-ports <bridge>

In the above scenario if you execute ovs-vsctl list-ports ovsbr it will return nothing.

samhenri commented 8 years ago

I don't know exactly what I triggered but after reinstalling openvswitch packages from RPM it worked on all distros. I've followed some guides and here's the output:

Added a new interface to ovsbr by running ovs-vsctl add-port ovsbr virbr3

ovs-vsctl show
1e2dbd55-9e96-4e55-aad2-464e5f41466b
    Bridge DMZ
        Port DMZ
            Interface DMZ
                type: internal
    Bridge WAN
        Port WAN
            Interface WAN
                type: internal
    Bridge ovsbr
        Port "virbr3"
            Interface "virbr3"
        Port ovsbr
            Interface ovsbr
                type: internal
    Bridge LAN
        Port LAN
            Interface LAN
                type: internal
    ovs_version: "2.3.1-git4750c96"

And then Wok response:

[
  {
    "name":"DMZ",
    "ports":[]
  },
  {
    "name":"LAN",
    "ports":[]
  },
  {
    "name":"WAN",
    "ports":[]
  },
  {
    "name":"ovsbr",
    "ports":[
      {
        "statistics":{
          "rx_over_err":"0",
          "tx_dropped":"0",
          "rx_packets":"0",
          "rx_frame_err":"0",
          "rx_bytes":"0",
          "tx_errors":"0",
          "rx_crc_err":"0",
          "collisions":"0",
          "rx_errors":"0",
          "tx_bytes":"0",
          "rx_dropped":"0",
          "tx_packets":"0"
        },
        "name":"virbr3",
        "mac_in_use":"52:54:00:84:7c:52",
        "link_state":"down",
        "admin_state":"up",
        "type":"interface"
      }
    ]
  }
]

By curl:

curl -k -u root  -H "Content-Type: application/json" -H "Accept: application/json" "https://localhost:8001/plugins/ginger/ovsbridges/ovsbr/add_interface"  -X POST -d'{"interface":"virbr2"}'
Enter host password for user 'root':
{
  "name":"ovsbr",
  "ports":[
    {
      "statistics":{
        "rx_over_err":"0",
        "tx_dropped":"0",
        "rx_packets":"0",
        "rx_frame_err":"0",
        "rx_bytes":"0",
        "tx_errors":"0",
        "rx_crc_err":"0",
        "collisions":"0",
        "rx_errors":"0",
        "tx_bytes":"0",
        "rx_dropped":"0",
        "tx_packets":"0"
      },
      "name":"virbr2",
      "mac_in_use":"52:54:00:3d:35:29",
      "link_state":"down",
      "admin_state":"up",
      "type":"interface"
    },
    {
      "statistics":{
        "rx_over_err":"0",
        "tx_dropped":"0",
        "rx_packets":"0",
        "rx_frame_err":"0",
        "rx_bytes":"0",
        "tx_errors":"0",
        "rx_crc_err":"0",
        "collisions":"0",
        "rx_errors":"0",
        "tx_bytes":"0",
        "rx_dropped":"0",
        "tx_packets":"0"
      },
      "name":"virbr3",
      "mac_in_use":"52:54:00:84:7c:52",
      "link_state":"down",
      "admin_state":"up",
      "type":"interface"
    }
  ]
}#