do OpenflowHelloFlood attack for port 6640, found ovsdb-server's memory keeps increasing until killed by system.

[coolMarlon]

OVS version :2.12.0

attack packet:

attacking 30MIN ,ovsdb-server's memory increased 11GB。

ovsdb-server coredump call Stack

i want to know if this attack happens, how ovsdb-servers's can identify and defend from this attack to avoid emory keeps increasing?

Thanks.

[igsilya]

Hi. Thanks for the report!

Though I don't think you're testing what you're trying to test. ovsdb-server doesn't accept OpenFlow. It is a database to store the local configuration for ovs-vswitchd. If you want to actually test that OpenFlow related issue, you need to create a listening controller interface on one of OVS bridges and send messages there. The listening process will be ovs-vswitchd, not ovsdb-server.

ovsdb-server socket speaks JSON RPC, not OpenFlow, so it expects the message to be a JSON object. But yes, there seems to be some memory leak in the json parsing library that should be fixed.

And FWIW, you're using a fairly old version of OVS. At least there is a 2.12.4 stable release with about 230 bug fixes on top of 2.12.0. And, actually, 2.12 reached end of life a while back and no longer supported. I'd recommend upgrading to latest stable 2.17.

As a general rule though, I'd not recommend exposing either of ovsdb-server or ovs-vswitchd interfaces externally. And in any case you should not use plain TCP for communication. You need to use TLS connections, so switch and controller can authenticate each other. See the 6.3.6 Encryption section of the OpenFlow 1.5.1 spec: https://opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.5.1.pdf

[igsilya]

@coolMarlon I'm not able to reproduce the issue. If you can run ovsdb-server under valgrind --log-file=valgrind.%p --leak-check=full --track-origins=yes add provide the results, that should give more information for further analysis. Thanks.

[coolMarlon]

Though I don't think you're testing what you're trying to test. ovsdb-server doesn't accept OpenFlow. It is a database to store the local configuration for ovs-vswitchd. If you want to actually test that OpenFlow related issue, you need to create a listening controller interface on one of OVS bridges and send messages there. The listening process will be ovs-vswitchd, not ovsdb-server.

I know this, It's all a coincidence.

ovsdb-server socket speaks JSON RPC, not OpenFlow, so it expects the message to be a JSON object. But yes, there seems to be some memory leak in the json parsing library that should be fixed.

I think this is not an memory leak problem, because when i stop attack, the memory restore.

---

This is the output of "ovs-appctl -t ovsdb-server coverage/show", maybe have some help.

Maybe we can add a packet format check for ovsdb-server? if not jsonrpc format, ovsdb-server do not handle.

Thanks. @igsilya

[igsilya]

ovsdb-server socket speaks JSON RPC, not OpenFlow, so it expects the message to be a JSON object. But yes, there seems to be some memory leak in the json parsing library that should be fixed.

I think this is not an memory leak problem, because when i stop attack, the memory restore.

hmm.

This is the output of "ovs-appctl -t ovsdb-server coverage/show", maybe have some help.

Unfortunately, these counters are fairly normal for a server that receives a lot of traffic.

Maybe we can add a packet format check for ovsdb-server? if not jsonrpc format, ovsdb-server do not handle.

It actually has the check and it should close the connection with 'protocol error'. Do you see those in the ovsdb-server log file? If you can run the test under 'valgrind --tool=massif', that may shed some light on where this memory is going.

[igsilya]

FWIW, I found what is actually going on here. You're sending an infinite string of zeroes, json parser thinks that it is a number and accumulates the string waiting for some kind of delimiter, but there will be no delimiter, so the string grows forever. Unfortunately, I'm not sure what the solution to this problem. It is equivalent to just sending a very large correct JSON-RPC request, and I don't think we want to limit such requests in size. So, yes, "don't do that" and "protect connection with SSL/TLS" seems to be the answer here for the time being.

[igsilya]

Closing this since there is nothing to fix. The solution is to not expose unprotected ovsdb-server ports. If it is required, the connection should use SSL/TLS and not allow connections from unverified clients this way.