use p4runtime-shell insert a flow rule, but can't displayed in onos-cli.

[La-Joya]

hi, I use p4runtime-shell to insert a flow table to leaf1 node, but this flow table cannot be displayed in onos-cli. I cannot use my own flow table. On the basis of ngsdn, I want to realize the mri experiment in the p4 exercise. So, I inserted a flow table in the egresspipeimpl of switch leaf1. I want to implement mri information collection between h1a and h1b. P4Runtime sh >>> tables EgressPipeImpl.swtrace IngressPipeImpl.acl_table IngressPipeImpl.l2_exact_table IngressPipeImpl.l2_ternary_table IngressPipeImpl.my_station_table IngressPipeImpl.ndp_reply_table IngressPipeImpl.routing_v6_table IngressPipeImpl.srv6_my_sid IngressPipeImpl.srv6_transit The flow table EgressPipeImpl.swtrace has been inserted in P4Runtime-shell, which is used to implement the mri function. But I can't find the EgressPipeImpl.swtrace flow table using "onos@root > flows -s any device:leaf1" in onos-cli. So the mri function can not be realized,

I've been stuck here for a long time and I don't know what to do. Hope to get your answers and help！！ Thanks！

[La-Joya]

A few more things to add: when using p4runtime, the election-id made some changes. Because when I use --election-id 1,0: sdn@tutorial-vm:~/ngsdn-tutorial$ util/p4rt-sh --grpc-addr localhost:50001 --config p4src/build/p4info.txt,p4src/build/bmv2.json --election-id 0,1 *** Connecting to P4Runtime server at localhost:50001 ... You are not master, you only have read access to the server CRITICAL:root:Error when setting config CRITICAL:root:P4Runtime RPC error (PERMISSION_DENIED): SetForwardingPipelineConfig from non-master is not permitted for node 1. When the election-id is changed to 1, 0 and 1, 1, I can enter P4runtime-sh to operate.

[ederollora]

Hi,

This is only a guess so anyone more familiar with the dev environment please check if I am right.

Could this be because you use the p4runtime-shell instead of the ONOS controller to insert the rules? The ONOS controller is probably not aware about the rules inserted since "the actual controller" is the p4runtime-shell and not ONOS, I think those are two independent controllers for the switch.

If you create your own class in ONOS that install rules for the switch you will see them when you execute the flows command.

Ideally you could make a program in ONOS that accepts the rules via the ONOS CLI (not the p4runtime-shell) and then inserts them using your own custom Java class that converts CLI syntax to actual P4Runtime rules for the switch. Therefore the rules inserted with the p4runtime-shell and the ONOS CLI or ONOS Java functions belong to different controllers.

I hope that is the correct answer.

Cheers,

[ederollora]

And to be more specific, you can list the rules by running:

P4Runtime sh >>> for te in table_entry["P4INFO-TABLE-NAME"].read():
            ...:     print(te)
            ...:

This is explained in EXERCISE 1 :)

Good luck,

[ederollora]

A few more things to add: when using p4runtime, the election-id made some changes. Because when I use --election-id 1,0: sdn@tutorial-vm:~/ngsdn-tutorial$ util/p4rt-sh --grpc-addr localhost:50001 --config p4src/build/p4info.txt,p4src/build/bmv2.json --election-id 0,1 *** Connecting to P4Runtime server at localhost:50001 ... You are not master, you only have read access to the server CRITICAL:root:Error when setting config CRITICAL:root:P4Runtime RPC error (PERMISSION_DENIED): SetForwardingPipelineConfig from non-master is not permitted for node 1. When the election-id is changed to 1, 0 and 1, 1, I can enter P4runtime-sh to operate.

This is probably because by defining a different election-id you define a different controller. ONOS seems to be master (so it seems to be the one defining the pipeline), so your capabilities might be reduced on what you can do (as explained in the output message once you connect). That is why you cannot SetForwardingPipelineConfig since that is something that ONOS is only capable of doing. This is explained in the P4Runtime spec but, ideally, you can specify different controllers to do different things like installing rules in different tables (not SetForwardingPipelineConfig as that can only be donde by one as far as I remember). Also I am not sure how far is the dev community into implementing the multiple controller support in Stratum so while this is possible according to the standard it might not be supported yet.

If you want to go deeper into the topic , please check: https://p4.org/p4-spec/p4runtime/main/P4Runtime-Spec.html#sec-client-arbitration-and-controller-replication

Cheers,

[La-Joya]

Hi,

Thanks for your reply.

1. ONOS web UI

Back to the flow table. Through exercise 3, I found the Pipeconf UI in the onos web interface. In that panel click on the Pipeconf icon(the last one), to open the Pipeconf view for that device. I found that the inserted egresspipeimpl.swtrace appears in the first column. But the second column entries used is zero value. I'm guessing because of this zero value.

Pipeline Model

Table | Entries used -- | -- EgressPipeImpl.swtrace | 0 IngressPipeImpl.acl_table | 5

2. What i want to achieve

Now what I want to do is to implement the INT function in this development environment. Before that, I need to verify the feasibility of INT. So I followed the Multi-Hop Route Inspection (MRI) in P4-tutorals and added some code to this development environment:

    action add_swtrace(switchID_t swid) { 
        hdr.mri.count = hdr.mri.count + 1;
        hdr.swtraces.push_front(1);
        hdr.swtraces[0].setValid();
        hdr.swtraces[0].swid = swid;
        hdr.swtraces[0].qdepth = (qdepth_t)standard_metadata.deq_qdepth;
    }
    table swtrace {
        key = {
            hdr.ipv6.next_hdr: exact;
        }
        actions = { 
            add_swtrace; 
            NoAction; 
        }
        default_action = NoAction();      
    }
......
......
        if (hdr.ipv6.next_hdr == 150) {
            swtrace.apply();
        }

3. Insert flow table

In ngsdn development environment, I didn't find a way to statically load flow table similar to P4-tutorals. So I can only use two ways such as p4runtime-shell and onos.

• Regarding the problem of p4runtime-shell, I have already described it earlier.

• I haven't read closely about the ONOS app. So it is difficult for me to create a java class in a short time. I will learn about this later. Regarding onos, I really need to learn more about this.

In P4-tutorals, the way to insert flow table is to load static json(or txt) file. As follows:

{
  "target": "bmv2",
  "p4info": "build/mri.p4info",
  "bmv2_json": "build/mri.json",
  "table_entries": [
    {
      "table": "MyEgress.swtrace",
      "default_action": true,
      "action_name": "MyEgress.add_swtrace",
      "action_params": {
         "swid": 1
       }
    }
  ]
}
​

I was wondering if there is a similar way to insert into a flow table. I saw a method for the ONOS REST API last night and I'm going to try it out today.

[ederollora]

Hi, here is some reply to some of your comments:

1. ONOS web UI

(...) But the second column entries used is zero value. I'm guessing because of this zero value. Table Entries used EgressPipeImpl.swtrace 0 IngressPipeImpl.acl_table 5

As you mentioned in your first or second comment, ONOS is the master controller. ONOS pushed the P4 program to the switch, so it is aware of the existence of the swtrace table. The problem is that you are trying to insert rules with a second controller, that according to your second comment, only has read access. So your second controller (the p4runtime-shell) should not be able to push rules if the "read access" is correctly established.

Now what I want to do is to implement the INT function in this development environment. Before that, I need to verify the feasibility of INT. So I followed the Multi-Hop Route Inspection (MRI) in P4-tutorals and added some code (...)

I beautified your code here:

action add_swtrace(switchID_t swid) {
    hdr.mri.count = hdr.mri.count + 1;        
    hdr.swtraces.push_front(1);        
    hdr.swtraces[0].setValid();        
    hdr.swtraces[0].swid = swid;        
    hdr.swtraces[0].qdepth = (qdepth_t)standard_metadata.deq_qdepth;    
}

table swtrace {
    key = {
        hdr.ipv6.next_hdr: exact;
    }        
    actions = {             
        add_swtrace;             
        NoAction;         
    }        
    default_action = NoAction();          
}

if (hdr.ipv6.next_hdr == 150) {            
    swtrace.apply();        
}

The issue here is that you tried the MRI exercise in P4 tutorials (that uses a Python-based controller) to run along with ONOS. Besides, you use the rules in a JSON file to install rules (Python code reads them and translates them to P4Runtime rules) but also use the ONOS controller. I could be wrong, but this approach will not work (for now). You need a controller to install rules, a CLI tool OR the Python controller that reads a JSON file to translate JSON to rules (or use its own Python methods instead of files) OR the ONOS controller and its own Java functions. Whatever the controller you choose to install rules, it has to be the master one, as it pushes the P4 pipeline. You cannot have all or several of those controllers simultaneously, installing rules. If this can be configured today, I am not aware of how to do it. My comment is based on the fact that you cannot use ONOS to push the pipeline to BMv2 (and act as master and handle LLDP, etc.) and use the Python controller or the p4runtime-shell to install rules.

If you check the NG-SDN tutorial exercise 3 (like you said in your comment 1), they use the p4runtime-shell but just as a "read" controller. It only shows the rules installed because the master controller installed them. In your case, you try to use ONOS as a master (that installs some rules for handling LLDP and some packet_in rules) but then also use a second controller to install additional rules. This is possible according to the P4runtime spec, but it is probably not implemented right now.

In ngsdn development environment, I didn't find a way to statically load flow table similar to P4-tutorals. So I can only use two ways such as p4runtime-shell and onos.

You also have the ONOS CLI. You need to program an app that takes the ONS CLI input text as a command and translates this to rules. It is similar to the one used by the Python-based controller in P4 tutorials that takes a file and translates it to rules.

Regarding the problem of p4runtime-shell, I have already described it earlier.

I haven't read closely about the ONOS app. So it is difficult for me to create a java class in a short time. I will learn about this later. Regarding onos, I really need to learn more about this.

In P4-tutorals, the way to insert flow table is to load static json (or txt) file. As follows:

I beautified your JSON here:

{
  "target": "bmv2",
  "p4info": "build/mri.p4info",
  "bmv2_json": "build/mri.json",
  "table_entries": [
    {
      "table": "MyEgress.swtrace",
      "default_action": true,
      "action_name": "MyEgress.add_swtrace",
      "action_params": {
        "swid": 1
      }
    }
  ]
}

As mentioned, I don't think you can use the p4runtime-shell and ONOS simultaneously to do the two things you are planning about.

I was wondering if there is a similar way to insert into a flow table. I saw a method for the ONOS REST API last night and I'm going to try it out today.

I am not sure if you can use the REST API right away. I believe you still need a class (similar to when creating an ONOS CLI) that takes the REST request payload and translates it to rules. The controller already does this in the P4 tutorials repository, but I am unsure if ONOS has something like this.

I would recommend using the same controller as in the P4 tutorials for your case; it will be less of a headache, especially if you do not have time or do not want to program ONOS apps right now.

Let me see if I can find someone to review my comment because I'd like to be sure that what I said is correct.

Cheers,

[ederollora]

Hi @ccascone and @charlesmcchan,

Could any of you review that what I commented is correct? Sorry for the long comment. I think reviewing the last comment is enough.

Thanks,

[La-Joya]

Hi, thanks for your reply.

Through your so many answers, I know that perhaps the best answer is to solve the problem of pushing the flow table through ONOS in NG-SDN.

[ederollora]

Yes, implementing everything in ONOS will probably solve your issues. It has steep learning curve so do not discard going with the Python controller in P4 tutorials if you are on a rush.

Best of luck,

[ederollora]

You are welcome to contact back if you need some help. You can close the issue if that's fine.

Cheers,

[La-Joya]

Thank you for your patience, I will try my best to learn about ONOS. Cheers,

[ccascone]

Hi @ccascone and @charlesmcchan,

Could any of you review that what I commented is correct? Sorry for the long comment. I think reviewing the last comment is enough.

Thanks,

Hi @ederollora, what you are saying is mostly correct.

The P4Runtime spec supports writes from multiple master controllers using the concept of "roles". Multiple controllers can write as long as they have different roles: https://p4.org/p4-spec/p4runtime/main/P4Runtime-Spec.html#sec-client-arbitration-and-controller-replication

Unfortunately, this is not implemented neither in ONOS nor Stratum. As a consequence, we cannot use roles in the NG-SDN tutorial.

When using p4runtime-shell one could temporarily become the master (by using the highest election ID) and insert new rules. Unfortunately (or fortunately, depending on your goal), ONOS acts as the authoritative source of truth for forwarding state. As such, ONOS does not assume the presence of other controllers and implements a reconciliation mechanisms to periodically check and remove extraneous flow rules. This is in part described in Exercise 5: https://github.com/opennetworkinglab/ngsdn-tutorial/blob/62705c3cb64fa7c731b5255f5fc793f293ce0665/EXERCISE-5.md#appendix-a-understanding-onos-error-logs

To summarize, whatever you add via p4runtime-shell not only it won't show up in the ONOS CLI, but also will eventually disappear from switches because of such reconciliation.

@La-Joya I agree with Eder's advice to implement everything in ONOS or a simpler Python-based controller depending on your needs.

[ederollora]

Thank you so much for your time @ccascone . It is much appreciated.