CPqD / ofsoftswitch13

OpenFlow 1.3 switch.
http://cpqd.github.com/ofsoftswitch13
304 stars 192 forks source link

Switch crash/stop after lot of traffic #189

Closed lyndon160 closed 6 years ago

lyndon160 commented 9 years ago

In mininet, using the CPqD switch I run two hosts connected together through one switch. This switch is controlled by a ryu simple switch 13.

After doing an iperf or large wget between two hosts communication fails and the switch has to be restarted for any communication to continue.

Pings and wget of small files work fine.

jm-boley commented 9 years ago

This sounds very similar to the problem I was experiencing, the switch misbehaving under load, though I didn't have the crashing symptoms you're experiencing. I'd try checking /var/log/messages to see if the TCP stack or anything else is logging any errors that coincide. Grepping for ofdatapath might give you some results in that case. In mine, I found that the switch was trying to push out packets that exceeded the MTU, so you might want to see if yours is doing something similar. I'd also try running iperf in UDP mode (-u) to see if the crashing persists. If it doesn't then our cases may have a similar root cause.

If you're at all familiar with the linux system call API, running strace -ff -o <trace file name> mn <options> might not be a bad idea. Since the process seems to be crashing you could try grepping for SIGCHLD, any child process that exits with a non-zero status code (e.g., you'll often see 1) is usually failing out. That should put you on the trail of the child process that's dying and let you zero in on whatever system calls it's trying to make that are failing. You might also try grepping for SIGKILL or SIGTERM just in case it's being killed off by another process, though I'm not aware of any that would off the top of my head.

Anyway, best of luck.

lyndon160 commented 9 years ago

Thanks for the help.

Yes, UDP iperf works fine. tshark shows that the switch receives one packet way over the MTU. After that the switch does not forward any packets.

Our issues do seem similar, it is strange that your version does not crash.

As for strace, I cannot pick out anything useful from the output but It's a bit out of my depth.

aneesalnajjar commented 9 years ago

Hi I have faces the same problem with Ofdatapath 1.3 and Ryu controller. When I send small TCP requests to the server, it successes and there is problem with CPqD switch. However, If I increase the number of TCP requests, the switch crashes and I have to restart my topology again. I have not faced this problem using Ofdatapath 1.0 with Pox.

I think this is a shotage in the switch. It should be solved

fr6nco commented 9 years ago

Turn off TCP offloading on the machine, where the switch is running. Im not sure whether this will be possible in mininet. I was binding real eth interfaces to the ofsoftswitch and turning off offloading helped.