Packets not enqueued at all?

[ermin-sakic]

Hello,

I've set up a queue as follows: EDIT: dpctl tcp:127.0.0.1:6634 queue-mod 2 5 300

Thus the queue ID is 5. Following that, I've set up a simple flow which matches the 10.0.0.2 destined packets and applies the set-queue action as follows:

dpctl tcp:127.0.0.1:6634 flow-mod cmd=add, table=0, idle=0, hard=0, prio=2048 eth_type=0x800,ip_dst=10.0.0.2 apply:queue=5

However, packets are not being enqueued (stats-queue shows 0 packets) nor sent out on the output port.

Setting up the following flow results in only the tx_drops value being incremented on stats-port calls, still no signs of any packets being enqueued on stats-queue output: dpctl tcp:127.0.0.1:6634 flow-mod cmd=add, table=0, idle=0, hard=0, prio=2048 eth_type=0x800,ip_dst=10.0.0.2 apply:queue=5,output=2

Setting only apply:output=2, without any enqueuing, sends out packets on port 2 as expected.

Am I missing something here?

With regards, Ermin

[ZoltanLajosKis]

I guess what you miss is that queues are local to ports, and not something global. I.e., with the queue-mod command you created a queue 5 on port 1.

With your first flow you don't specify an output port, so packets will be dropped. With your second flow you ask the switch to send packets to queue 5 on port 2. But port 2 does not have a queue 5. With the third example you send out packets on port 2 without queuing, and it works.

I would suggest you to experiment with the port on which you created the queue.

[ermin-sakic]

Ah sorry about the typo, of course that port 2 was meant at queue creation time: dpctl tcp:127.0.0.1:6634 queue-mod 2 5 300

Each queue is associated with a port, so much is clear, and sending a packet on that output port, when marked with set-queue before, is expected to be enqueued at output:"port" execution time.

And exactly this is not working for me - packets are not enqueued at all.

[ermin-sakic]

Added log to make clear what I mean with missing packets at the queue 5. **** QUEUE CREATION ubuntu@ubuntu:~$ dpctl tcp:127.0.0.1:6634 queue-mod 2 5 300 SENDING: expmodqueue{port="2", queue={q="5", props=[minrate{rate="300"}]}} OK. **** FLOW CREATION ubuntu@ubuntu:~$ dpctl tcp:127.0.0.1:6634 flow-mod cmd=add,table=0,idle=0,hard=0,prio=2048 eth_type=0x800,ip_dst=10.0.0.2 apply:queue=5,output=2 SENDING: flow_mod{table="0", cmd="add", cookie="0x0", mask="0x0", idle="0", hard="0", prio="2048", buf="none", port="any", group="any", flags="0x0", match=oxm{eth_type="0x800", ipv4_dst="10.0.0.2"}, insts=[apply{acts=[queue{q="5"}, out{port="2"}]}]} OK. **** STATS AFTER MATCHED TRAFFIC ubuntu@ubuntu:~$ dpctl tcp:127.0.0.1:6634 stats-flow SENDING: stat_req{type="flow", flags="0x0", table="all", oport="any", ogrp="any", cookie=0x0", mask=0x0", match=oxm{all match}} RECEIVED: stat_repl{type="flow", flags="0x0", stats=[{table="0", match="oxm{eth_type="0x800", ipv4_dst="10.0.0.2"}", dur_s="48", dur_ns="231000", prio="2048", idle_to="0", hard_to="0", cookie="0x0", pkt_cnt="21", byte_cnt="2058", insts=[apply{acts=[queue{q="5"}, out{port="2"}]}]}]} **** QUEUE STATS AFTER MATCHED TRAFFIC ubuntu@ubuntu:~$ dpctl tcp:127.0.0.1:6634 stats-queue SENDING: stat_req{type="queue", flags="0x0", port="any", q="all"} RECEIVED: stat_repl{type="queue", flags="0x0", stats=[{port="2", q="5", tx_bytes="0", tx_pkt="0", tx_err="0"}]} **** PORT STATS AFTER MATCHED TRAFFIC ubuntu@ubuntu:~$ dpctl tcp:127.0.0.1:6634 stats-port SENDING: stat_req{type="port", flags="0x0", port="any"} RECEIVED: stat_repl{type="port", flags="0x0", stats=[{port="1", rx_pkt="403", tx_pkt="0", rx_bytes="50078", tx_bytes="0", rx_drops="0", tx_drops="0", rx_errs="0", tx_errs="0", rx_frm="0", rx_over="0", rx_crc="0", coll="0"}, {port="2", rx_pkt="169", tx_pkt="0", rx_bytes="37928", tx_bytes="0", rx_drops="0", tx_drops="308", rx_errs="0", tx_errs="0", rx_frm="0", rx_over="0", rx_crc="0", coll="0"}, {port="local", rx_pkt="0", tx_pkt="0", rx_bytes="0", tx_bytes="0", rx_drops="0", tx_drops="0", rx_errs="0", tx_errs="0", rx_frm="0", rx_over="0", rx_crc="0", coll="0"}]}

---

Notice the tx_drops value for port 2.

[ZoltanLajosKis]

Sorry, I had no time to investigate this, and probably won't have before next week...

The queue handling relies on tc and htb; look at COMMAND_ADD_DEV_QDISC in lib/netdev.c

What you could do is query the qdiscs with tc, and check if what's there is consistent with what you want. If it is, this might not be a switch issue, but something with tc and htb implementation on your system...

[ermin-sakic]

The output of "tc class show dev IF" shows the appropriate traffic classes being created on queue-mod command. However, the packets do not seem to be enqueued (marked?) and on "tc -s class show dev IF" call, no packet counters are incremented for the created queues.

If you do get the expected results in your environment and tc/htb implementation is at fault on my side, it would be great if you could update me with your configuration so I can set it up as well for further experimenting.

Thanks!

[ermin-sakic]

Hi Zoltan, any updates on this matter?

With regards, Ermin

[balotr]

Hi @ederlf

Any enhancements about this issue? If not can you help me about this issue?

Having same problem, when I add a queue from dpctl, I can see it when I query but there is no flow out of that queue. I can see flow has packet received and sent but they are not going out of queues:

flow stats reply: stat_repl{type="flow", flags="0x0", stats=[{table="0", match="oxm{eth_type="0x800", ipv4_dst="10.0.0.1", ip_proto="17", udp_dst="5001"}", dur_s="1080", dur_ns="259000000", prio="65533", idle_to="0", hard_to="0", cookie="0x1", pkt_cnt="19661", byte_cnt="29727432", insts=[meter{meter="1"}, apply{acts=[queue{q="1"}]}, goto{table="1"}]},

queue stats reply:

stat_repl{type="queue", flags="0x0", stats=[{port="1", q="0", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="1", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="2", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="3", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="4", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="5", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="6", tx_bytes="0", tx_pkt="0", tx_err="0"}, {port="1", q="7", tx_bytes="0", tx_pkt="0", tx_err="0"}]}

when I query with tc class show dev s1-eth1 command I can see queues are really created:

class htb 1: parent 1:ffff prio 0 rate 100000Kbit ceil 10000Mbit burst 1600b cburst 0b class htb 1:ffff root rate 10000Mbit ceil 10000Mbit burst 0b cburst 0b class htb 1:1 parent 1:ffff prio 0 rate 5000Mbit ceil 10000Mbit burst 625b cburst 0b class htb 1:fffe parent 1:ffff prio 0 rate 10000Kbit ceil 10000Mbit burst 1600b cburst 0b class htb 1:2 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b class htb 1:3 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b class htb 1:4 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b class htb 1:5 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b class htb 1:6 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b class htb 1:7 parent 1:ffff prio 0 rate 1000Mbit ceil 10000Mbit burst 1375b cburst 0b

however with tc filters show dev s1-eth1 does not show anything.

Can you please explain the match mechanism of flows entering queues in OfSoftSwitch13 switch? How it is implemented? Where can I see if flows are matched with right egress queues?

Thanks in advance for your help

[ederlf]

@balotr The switch does not have queue matching as it is not an OpenFlow feature. Queues are external to the switch and handled by tc.

I see you are using metering with queuing. How is that meter configured? You also need to have an outpt action to the port your queue was created.

[balotr]

Hi @ederlf sorry for late response

I use ryu as controller. I configured that meter with RYU QoS REST-API. In standard OpenFlow Switch Specification Version 1.3.1 (Wire Protocol 0x04) September 6, 2012 set queue action is written like this (page 58):

The Set-Queue action sets the queue id that will be used to map a flow entry to an already-configured queue on a port, regardless of the ToS and VLAN PCP bits. The packet should not change as a result of a Set-Queue action. If the switch needs to set the ToS/PCP bits for internal handling, the original values should be restored before sending the packet out. A switch may support only queues that are tied to specific PCP/ToS bits. In that case, we cannot map an arbitrary flow entry to a specific queue, therefore the Set-Queue action is not supported. The user can still use these queues and map flow entries to them by setting the relevant fields (ToS, VLAN PCP). A Set Queue action uses the following structure and fields: / OFPAT_SET_QUEUE action struct: send packets to given queue on port. / struct ofp_action_set_queue { uint16_t type; / OFPAT_SET_QUEUE. / uint16_t len; / Len is 8. / uint32_t queue_id; / Queue id for the packets. / }; OFP_ASSERT(sizeof(struct ofp_action_set_queue) == 8);

Ryu handles routing automatically. Do I still need to add another flow entry?

Here is my meter script:

curl -X POST -d '{"meter_id":"1", "flags":"STATS", "bands":[{"type":"EXPERIMENTER","rate": "500000","prec_level": "1"}]}' http://localhost:8080/qos/meter/0000000000000001

Here is the answer from ryu:

[{"switch_id": "0000000000000001", "command_result": [{"result": "success", "details": "Meter added. : meter_id=1"}]}]

[balotr]

Hi @ederlf

I added the following flow to the switch with following code:

curl -X POST -d '{"dpid": 1, "cookie": 1, "cookie_mask": 1, "table_id": 1, "priority": 11111, "match":{"eth_type": "0x800", "ipv4_dst": "10.0.0.1", "ip_proto":"17", "udp_dst": "5001"}, "actions":[{"type":"OUTPUT", "port": 1}]}' http://localhost:8080/stats/flowentry/add

and it worked here is my output:

mininet@mininet-vm:~$ dpctl tcp:localhost:6634 stats-queue

SENDING (xid=0xF0FF00F0):
stat_req{type="queue", flags="0x0", port="any", q="all"}

RECEIVED (xid=0xF0FF00F0):
stat_repl{type="queue", flags="0x0", stats=[{port="1", q="0", tx_bytes="0", tx_pkt="0", tx_err="0"},
{port="1", q="1", tx_bytes="1935360", tx_pkt="1280", tx_err="0"}]}

but why is the case? Is that means my packets were not routed?

[ederlf]

@balotr Metering and queuing are two different things. Metering is per flow based, while queues are per port. What are you trying to achieve?

To guarantee queuing is used, you need to make sure that your application have an action with the created queue and an output action to the port the queue is attached.

[balotr]

@ederlf Thanks for explanation