Can controller listen to packet-in from switch using gRPC?

[hittingnote]

Recently, I want the controller to listen to the packet-in from switches, and then dispatch flow table entry to the specific switch sending packet-in. However, the given examples only contain a python code that directly download flow table entry to switch. So how can controllers listen to packet-in?

[kevinbird61]

After my experiments, currently bmv2 support packet-in via p4runtime. Here is my scenario of making a L2 learning switch. I've modified the python scripts from p4lang/tutorials (mostly switch.py and helper.py) to adapt packet-in message.

[hittingnote]

After my experiments, currently bmv2 support packet-in via p4runtime. Here is my scenario of making a L2 learning switch. I've modified the python scripts from p4lang/tutorials (mostly switch.py and helper.py) to adapt packet-in message.

Thanks for your reply! I hope that your scenario will help me a lot.

[hittingnote]

After my experiments, currently bmv2 support packet-in via p4runtime. Here is my scenario of making a L2 learning switch. I've modified the python scripts from p4lang/tutorials (mostly switch.py and helper.py) to adapt packet-in message.

I just encounter a problem. When I start mininet and then run start_p4_controller.sh, it shows that, File "p4runtime_lib/switch.py", line 20, in from p4.v1 import p4runtime_pb2 ImportError: No module named v1

So did you install patches of P4. How to install module p4.v1?

[kevinbird61]

p4.v1 is the latest version of p4runtime, you have to update your p4 dependencies. I've sorted out those dependencies from p4lang org. You can find something useful here.

[hittingnote]

After my experiments, currently bmv2 support packet-in via p4runtime. Here is my scenario of making a L2 learning switch. I've modified the python scripts from p4lang/tutorials (mostly switch.py and helper.py) to adapt packet-in message.

Hello! I meet another problem. I modify basic.p4 from p4-tutorial and controller.py from your learning-switch to implement packet-in and packet-out communication by myself, but it seems that controller read the packet from the wrong byte. The output is as follows: src: 00:00:01:01:08:00, dst: 00:00:01:01:00:00, type: 17664

But actually, dst is 00:00:00:00:01:01, src is 00:00:00:00:01:01 and type is 0x0800, as is captured with wireshark. I think one of the problems may be that I didn't use the header files codex/l2.p4, codex/l3.p4, etc. So what's the problem you think? Quite hope for your reply.

[kevinbird61]

I think the problem is there have some other packets in the simulated network created by mininet (e.g ICMP、MDNS ...). So if your send all the unknown packet to controller, it will see all these packets. You have to filter out these packets by yourself, like this line in your controller.

[antoninbas]

It kind of looks like the values are shifted. Based on that, I'm tempted to say that you are either missing a header (most likely) or you have an extra header. I haven't looked at the P4 program, but do you have a controller header that you are forgetting to make valid?

[hittingnote]

It kind of looks like the values are shifted. Based on that, I'm tempted to say that you are either missing a header (most likely) or you have an extra header. I haven't looked at the P4 program, but do you have a controller header that you are forgetting to make valid?

Thank you very much! I finally found out that I forgot emitting hdr.packet_in in deparser block. But now there's another problem. I tried to write flow entry to switch, but the output is as follows: Traceback (most recent call last): File "p4_controller.py", line 179, in main(args.p4info, args.bmv2_json) File "p4_controller.py", line 130, in main writeIPForward(p4info_helper, sw=s1, ip_dst_addr=pkt_ip_dst, eth_dst_addr="00:00:00:02:02:00", port=2) File "p4_controller.py", line 57, in writeIPForward "port": port File "/home/p4/p4-researching/src/fundamental/p4-project/../../../utils/p4runtime_lib/helper.py", line 228, in buildTableEntry for match_field_name, value in match_fields.iteritems() File "/home/p4/p4-researching/src/fundamental/p4-project/../../../utils/p4runtime_lib/helper.py", line 109, in get_match_field_pb lpm.value = encode(value[0], bitwidth) File "/home/p4/p4-researching/src/fundamental/p4-project/../../../utils/p4runtime_lib/convert.py", line 92, in encode assert(len(encoded_bytes) == byte_len) AssertionError

My code of writing table is as follows: def writeIPForward(p4info_helper, sw, ip_dst_addr, eth_dst_addr, port): table_entry = p4info_helper.buildTableEntry( table_name = "MyIngress.ipv4_lpm", match_fields = { "hdr.ipv4.dstAddr": ip_dst_addr }, action_name = "MyIngress.ipv4_forward", action_params = { "dstAddr": eth_dst_addr, "port": port } ) sw.WriteTableEntry(table_entry) print("Installed MyIngress.ipv4_lpm rule via P4Runtime")

My respective code in P4 is shown below: action ipv4_forward(bit<48> dstAddr, bit<9> port) { standard_metadata.egress_spec = (bit<9>)port; hdr.ethernet.srcAddr = hdr.ethernet.dstAddr; hdr.ethernet.dstAddr = dstAddr; hdr.ipv4.ttl = hdr.ipv4.ttl - 1; }

And the error occurs since the following code is being executed: writeIPForward(p4info_helper, sw=s1, ip_dst_addr=pkt_ip_dst, eth_dst_addr="00:00:00:02:02:00", port=2)

I made a similar modification in demo called learning-switch, but it works well cause there's no error info. Besides, I directly add "def writeIPForward" code to python script in learning-switch, so I think the problem may be at the definition of port in P4 program, but not know what on earth is that problem? I am looking forward that your reply will help.

[antoninbas]

I don't think you are providing a valid LPM match. The value for "hdr.ipv4.dstAddr" should be a tuple (IPAddr, prefixLength). See this example: https://github.com/p4lang/tutorials/blob/master/exercises/p4runtime/mycontroller.py#L45 On a side note, you cannot start modifying some of the tutorial code and expect everything to keep working as it is. If you start writing your own P4 program, you will need to write your own control-plane code, even though you can probably re-use the helpers.py code.

[guiladkatz]

Hello Antonin. I have began developing a little bit with P4 and P4Runtime, and I came across a problem i find it very hard to solve.

I took your example of a learning switch, and modified it such that instead of arp, the switch will forward IPv4 packets to the controller, who will then reply with a "Packet_out" message to the switch, and then (If desired) it can install IPV4 forwarding rules into the switch. I have everything working fine, and my hosts are pinging each other successfully, while at the begininng the switch learns their ip address and forwards packest accordingly.

My problem is, that i came to understand that forwarding packets to the switch's CPU is much slower than forwarding them directly through one of the ethernet ports. Therefore, my question is: Is there a way to forward packets to the controller through one of the ethernet ports? and if so, do i still need to use the "Packet_in" and "Packet_out" headrs?

Thank you in advance for your help!

Guilad.

[jafingerhut]

Guilad, some P4-programmable devices do have Ethernet ports specifically intended for connecting to a local (or perhaps even remote) CPU, and even if a device did not, if you were willing to create a logic board that dedicated one of the regular Ethernet ports to connecting to a local CPU (or a remote one) for that purpose, then you can do that. There is nothing in the P4 language or P4Runtime API that mandates this, nor that prevents it. It is an option.

You can create a virtual network, e.g. via mininet, that uses one of the ports of a switch in this way, if you arrange the switches and links in that way yourself.

If you do this, then as far as the P4Runtime API is concerned, this is just another Ethernet link between a switch and a host. Packet_in and Packet_out headers are neither required, nor forbidden -- you get to decide in your system exactly what headers should appear on packets crossing those links. It is up to you writing appropriate software on the CPU side that can recognize such received packets, and send them to the switch, so you might want to consider starting with an Ethernet header, and perhaps even IPv4 and UDP, if you want to make it easy to write user-space programs that can send and receive such packets on the host. Otherwise you may need to do special configuration in the operating system kernel to ensure that packets are sent and received with full control over all packet headers.

[guiladkatz]

Hi Andy, thank you for your kind response.

Indeed, that is what i am trying to do. My goal is to send packets to the controller directly, without going through the CPU on port 255 first. At first, the problem was that the controller wasnt running on any hosts conneected to the ehernet ports on the switch, so i run the controller inside one of the hosts.

For example: Like in the Learing_switch example from Antonin's GitHub, i ran in mininet 3 hosts connected to one switch. Then, i ran the controller code on ohst h1. Then, i sent ping packets from h2 to h3. My expectation was that the switch would forward those ICMP packets to h1 (in his IPv4_forward table i set the "Default Action" to be send_to_cpu() while specifing the egress port as 1). With wireshark, i examined the s1<->h1 link, and i saw that when i issue the PING from h2 to h3, i get LLC packets going from the switch to h1.

Maybe my implementation is right, but i have a different problem? maybe my hole approach to find the solution is wrong (running the controller on h1)?

Thank you very much for your help!

Guilad.

[jafingerhut]

Running controller software on a host that is connected directly to a switch via an Ethernet port does sound like the correct way to enable sending packets from a switch directly to that host without going through P4Runtime.

ARP packets are not ICMP. ICMP has Ethernet + IPv4 + ICMP headers, in that order, with ethertype=0x0800 as for all other IPv4 packets. ARP have Ethernet + ARP with ethertype=0x0806, so in your P4 code they should have hdr.ipv4.isValid()=false, and it would not make sense to look them up in a table that uses IPv4 header fields in the key. You would need some other table or if statement in your P4 program somewhere to detect and direct the ARP packets to the host running the controller.

Also note that if you send them to the host running the controller software with no modifications, they will arrive to that host, probably running a Linux kernel with all of its networking code running in the kernel, as an ARP packet, which the Linux kernel networking software will treat as an ARP request to that Linux host. If you want the ARP packets to reach your controller software, you either need to have some software monitoring packets on that Ethernet interface in promiscuous mode (e.g. the way tcpdump does, with superuser privileges), or something similar. Alternately, you can have the switch encapsulate the ARP packet inside of some additional headers before sending them to the host running the controller software.

[antoninbas]

My problem is, that i came to understand that forwarding packets to the switch's CPU is much slower than forwarding them directly through one of the ethernet ports. Therefore, my question is: Is there a way to forward packets to the controller through one of the ethernet ports? and if so, do i still need to use the "Packet_in" and "Packet_out" headrs?

This is true, and you can be faster that P4Runtime by sending packets out of an Ethernet port to a high-performance control-plane application. But if your control-plane application is running on a CPU and depending on its complexity, the difference between the 2 solutions may not be that significant. You would need a very optimized or simple control plane to be able to support Gbps of control-plane traffic. I would expect the P4Runtime stream channel to be able to support a few 100,000 CPU packets per second. It is worth noting that ONOS, which is a production-grade SDN controller, uses the P4Runtime PacketIn / PacketOut mechanism.

Additionally, in practice the management network for your switch may be separate, so sending controller packets through an Ethernet port assumes that having inband control-plane traffic is possible / acceptable.

[guiladkatz]

Thank you both very much fr your replies.

If i am understanding correctly, you are saying that in "Real life" switches, they are not supposed to communicate with the P4Runtime controller via ethernet ports? They do that via the switch CPU? Including ONOS?

I do want to use P4Runtime in my control plane controller. Lets assume, that i will use the Packet_in and Packet_out headers. I want to implement my topology in mininet. I am trying to set up a remote controller (which can be done in Mininet), and my remote controller will be the p4_controller from your learning_switch example (with the neded modifications in the python code, obviously).

My question is then, how do i configure my switch in P4 to forward packets to that controller? which port do i specify in my send_to_cpu action (instead of the CPU 255 port)? I cant seem to find the port number where the controller is connected to the switch in Mininet.

Once again, i thank you very much for your patient responses!

Guilad.

[antoninbas]

Thank you both very much fr your replies.

If i am understanding correctly, you are saying that in "Real life" switches, they are not supposed to communicate with the P4Runtime controller via ethernet ports? They do that via the switch CPU? Including ONOS?

Since you mention "P4Runtime controller" explicitly, then I assume you refer to the PacketIn / PacketOut mechanism. In that case, I find it unlikely that communication would happen through a data port without involving the switch CPU, as the gRPC protocol is used. That would require the switch ASIC to support gRPC directly. Typically packets are sent to the switch CPU over PCIe and then to the P4Runtime controller over the management port (which would also be an Ethernet port, but is not a switch port obviously). That's what ONOS "does". BTW, packets could also be sent to the switch CPU over Ethernet, that depends on the switch hardware configuration.

If you are not referring to P4Runtime (PacketIn / PacketOut mechanism), then you can send packets to your controller over a data port without involving the switch CPU at all. You can have a dedicated network for that, or do it in-band. I would imagine that's a more uncommon scenario.

I do want to use P4Runtime in my control plane controller. Lets assume, that i will use the Packet_in and Packet_out headers. I want to implement my topology in mininet. I am trying to set up a remote controller (which can be done in Mininet), and my remote controller will be the p4_controller from your learning_switch example (with the neded modifications in the python code, obviously).

Which example are we talking about? This one: https://github.com/antoninbas/p4runtime-go-client/tree/master/cmd/l2_switch? That's the only one I am aware of that uses P4Runtime (but it doesn't use PacketIn / PacketOut, it uses learning / digests).

My question is then, how do i configure my switch in P4 to forward packets to that controller? which port do i specify in my send_to_cpu action (instead of the CPU 255 port)? I cant seem to find the port number where the controller is connected to the switch in Mininet.

I am not 100% sure which code you are referring to, but simple_switch_grpc has a --cpu-port argument (https://github.com/p4lang/behavioral-model/tree/master/targets/simple_switch_grpc#running-simple_switch_grpc). You need to use it if you want PacketIn / PacketOut to work, and you can set it to any value you want to use. The value you pick is the value you need to use in your P4 program.

Once again, i thank you very much for your patient responses!

Guilad.

[guiladkatz]

Ok. I didnt know that if i use gRPC, like P4Runtime does, all packets have to go through the switch's CPU.

The example i am talking about is this one: https://github.com/kevinbird61/p4-researching/tree/master/src/fundamental/learning-switch

I modified the "basic_tutorial_switch.p4" to handle packets based on IPv4 headers, and implemented a IPv4_forwarding table without handling ARP request (For now, i installed ARP rules at the mininet hosts manually, just for testing). When there is a table-miss in the IPv4 table, the packet is forwarded to the controller which i took from the same link above, it is called "p4_controller.py". I modified the code according to my needs.

yes i am aware that i can specify the CPU port with --cpu-port in the simple_switch_grpc. What i was referring to, is that if i build a Mininet topology, with a remote controller, how do i know where it is connected to the switch? at which port number?

In general, i will tell you that what i am trying to do. I am trying to build a topology in mininet, which has several hosts, switches, and a controller. The switches pipeline will be written in P4, a IPv4 table will be configured, and the hosts will communicate with each other. Upon a miss, the switch will forward to the controller the missed packet and the controller will reply to the switch with the forwarding instruction, and insert the necessary rule to the switch.

Now i managed to achieve all of the above! i had a perfectly working environment. Everything was properly configured with P4 and P4Runtime, all hosts were communicating, the switch was succesfully updating his flow table, and i had a happy life. All thanks to you and the modifications you made to "helper.py" and "switch.py".

But then, i was told that sending the packets to the CPU is much slower than sending them to the controller over ethernet ports, so i need to change my implementation (still using P4Runtime and gRPC). This is were i am stuck.

I'll just clarify that i described my work to give you some background to my questions, with no intention for you to do the job for me :)

[antoninbas]

yes i am aware that i can specify the CPU port with --cpu-port in the simple_switch_grpc. What i was referring to, is that if i build a Mininet topology, with a remote controller, how do i know where it is connected to the switch? at which port number?

if you want to bypass P4Runtime (bypass the switch CPU), you have to take care of the plumbing yourself. You can create a veth pair and attach one end to bmv2 (you can select any port number you want to use). Then you can create a container to run your controller and move the other end of the veth to that container's network namespace. What you do next is up-to-you, e.g. your controller could use raw sockets to capture all packets on that veth interface (one for each switch). You can also create a bridge if you want and have a star topology (one link for each switch + one for the controller). That works well if everything is on the same machine. If the controller is running on a different machine from Mininet, things are more complicated. You can also use Linux forwarding. But the bottom line is there is no magic, you have to build that "network" yourself.

But then, i was told that sending the packets to the CPU is much slower than sending them to the controller over ethernet ports, so i need to change my implementation (still using P4Runtime and gRPC). This is were i am stuck.

that's not realistic as I explained before, and not possible with bmv2

BTW, I am assuming that by port, you always mean switch data port number, and not the TCP port number used for P4Runtime (communications between P4Runtime server process running on the switch CPU and P4Runtime client running in the controller). Because I'm starting to be confused...

[guiladkatz]

Yes, indeed I mean the switch data port when I say "port", I'm sorry for the confusion.

Also i am aware that if i want to bypass P4Runtime i need to take care of the necessary configurations. But I was explicitly told that I should use P4Runtime.

Very well. So basically what I am trying to implement is not realistic.

I thank you very much for the time you spent on me, you helped me a lot.

Best regards, Guilad Katz.

[Safaamahrach]

Ok. I didnt know that if i use gRPC, like P4Runtime does, all packets have to go through the switch's CPU.

The example i am talking about is this one: https://github.com/kevinbird61/p4-researching/tree/master/src/fundamental/learning-switch

I modified the "basic_tutorial_switch.p4" to handle packets based on IPv4 headers, and implemented a IPv4_forwarding table without handling ARP request (For now, i installed ARP rules at the mininet hosts manually, just for testing). When there is a table-miss in the IPv4 table, the packet is forwarded to the controller which i took from the same link above, it is called "p4_controller.py". I modified the code according to my needs.

yes i am aware that i can specify the CPU port with --cpu-port in the simple_switch_grpc. What i was referring to, is that if i build a Mininet topology, with a remote controller, how do i know where it is connected to the switch? at which port number?

In general, i will tell you that what i am trying to do. I am trying to build a topology in mininet, which has several hosts, switches, and a controller. The switches pipeline will be written in P4, a IPv4 table will be configured, and the hosts will communicate with each other. Upon a miss, the switch will forward to the controller the missed packet and the controller will reply to the switch with the forwarding instruction, and insert the necessary rule to the switch.

Now i managed to achieve all of the above! i had a perfectly working environment. Everything was properly configured with P4 and P4Runtime, all hosts were communicating, the switch was succesfully updating his flow table, and i had a happy life. All thanks to you and the modifications you made to "helper.py" and "switch.py".

But then, i was told that sending the packets to the CPU is much slower than sending them to the controller over ethernet ports, so i need to change my implementation (still using P4Runtime and gRPC). This is were i am stuck.

I'll just clarify that i described my work to give you some background to my questions, with no intention for you to do the job for me :)

HI I have already create python program for P4Runtime , with packet-in and packet-out, but it doesn't work. I read your comments @guiladkatz and I understand that you have realize that. It is possible to post the right program many thanks