Open kevindweb opened 4 years ago
This is a good idea. This would have to be coupled with something that automates packet sending to the NF on startup..maybe an instance of speed_tester with a standard pcap file? I think it might be difficult to do it with pktgen on each test since that'll require a 2 node setup for the unit tests. I also wonder how abstract the NF unit tests should be. Are we just testing for segfaults, or specific NF outputs? It might require the NF unit test structures to be loosely defined -> inspected by us when we do PR reviews.
I was thinking once we get @catherinemeadows pr merged we will create a json with a NF chain of most of our main NFs. We will need to figure out what will be the expected output of this NF chain.
This is a good idea. This would have to be coupled with something that automates packet sending to the NF on startup..maybe an instance of speed_tester with a standard pcap file? I think it might be difficult to do it with pktgen on each test since that'll require a 2 node setup for the unit tests. I also wonder how abstract the NF unit tests should be. Are we just testing for segfaults, or specific NF outputs? It might require the NF unit test structures to be loosely defined -> inspected by us when we do PR reviews.
I think this is true it might be a bit harder to test with CI as maintainers will have to look out for NF outputs as well as other things such as seg-faults. I think we could make a platform such that it is easy for developers and maintainers to run a broad range of test locally.
Here are a few good resources we should use to get started on this. This guy has a nice article talking about CMOCK. It might work for our case, but the main point is that we need to view NFs as infinite processes that need to be "bounded" in some way for the testing of their functionality. The article points to a series of C related testing ideas.
@sdnfv/gw-undergrads I updated the issue guide with the title "06/29/2020 Meeting Notes". Take a look to see if you want to participate in the Unit Testing framework. These notes are from a meeting with Tim this afternoon.
Feature Request for Unit Tests
Is your feature request related to a problem? Please describe. We currently don't have a way to test arbitrary NFs that are developed onto the platform. This means that if new bugs come into ONVM, CI doesn't know unless they affect Speed Tester or Basic Monitor.
Describe the solution you'd like Unit testing would be a great way to fix this. Every new NF would need a testing structure that the developer creates before making a PR. Unit testing tools for C exist already that we should take advantage of if possible.
Describe alternatives you've considered An alternative to the prior Unit Testing strategy, we could make a test version of the onvm manager. The thought here is how do you test an NF? NF's are not library code that provide explicit output, based on easy inputs like strings or integers. As a result, simulation of packets similar to speed tester would be required to test the Firewall NF for example.
Resources This is a thread to get a head start on this for the summer. I will post here more material that I find online. The DPDK Test Suite (DTS), with documentation here, is probably a good place to start. As @Vivek-anand-jain mentioned, we should have a
make test
functionality for initiating all tests at once. I know unit testing was a huge advancement for Gabe's Composite project, so this would help advance the security and reliability of the onvm platform significantly.06/29/2020 Meeting Notes
After a meeting with Tim, here's the roadmap for the ONVM Testing Suite
3 Main Tests
1) Generic "Valid NF" Tests to check that the NF can handle packet flow through the API
Specifically for number 3, how do we verify the test passed? This will be up to the NF developer. There are 3 ways to determine this. They will need to register this choice upon starting the test framework.
NFlib Accommodations
A testing flag (or macro) will be supplied for the manager to simplify the analysis of the unit tests.
libpcap
library to an outfileAnalysis
A python program to analyze the result of the unit test and confirm everything worked correctly. This will also be in charge of the setup and teardown of the NFs. The unit tested NF will need to specify their "choice" in what data will determine their validity. For the PCAP file and/or trace log, the output can be MD5 hashed against what the expected result. If they're different, the NF failed in some way
First Steps
functional_test
NF)make test firewall
functional_test
and calling the firewall's test function.onvm_nflib
code to check for test flag - check the unit tests choice of pcap, trace log, or stdout, and process their results accordingly - probably in a macro to avoid performance issues outside of test