How does `feature.textproto` align with tests?

[robshakir]

cc: @liulk @dplore @bstoll @xw-g

today, we have a number of missing feature.textproto files through the repository - but it's not clear how these are intended to align with the tests that are specified. the contributing doc says that the structure is essentially:

• feature/foo/subfeature
  • feature.textproto
  • (ate|otg)_tests/
  • tests

this makes sense given that feature.textproto would then describe the paths and RPCs that are meant to be covered by the feature profile that is defined there.

however, if we look at the actual layout of the repo:

▶ find featureprofiles/feature/gribi -type d -mindepth 1
./ate_tests
./ate_tests/base_leader_election_test
./ate_tests/leader_failover_test
./ate_tests/get_rpc_test
./ate_tests/ordering_ack_test
./ate_tests/weighted_balancing_test
./ate_tests/route_removal_via_flush_test
./ate_tests/route_removal_non_default_vrf_test
./ate_tests/ack_in_presence_other_routes
./ate_tests/base_hierarchical_route_installation_test
./tests
./tests/gribigo_compliance_test
./otg_tests
./otg_tests/base_leader_election_test
./otg_tests/leader_failover_test
./otg_tests/get_rpc_test
./otg_tests/ordering_ack_test
./otg_tests/route_removal_via_flush_test
./otg_tests/route_removal_non_default_vrf_test
./otg_tests/ack_in_presence_other_routes
./otg_tests/base_hierarchical_route_installation_test

here we now have tests that are subsets of a feature underneath the directories that contain tests. e.g., Flush coverage is its own feature profile. there's also no feature.textproto.

so, two questions:

1. it seems like we should not check in tests that do not have a feature.textproto. what's the logic to not do this?
2. is the gribi example here something that we need to iterate on? (it looks like interface has a clearer breakdown for example, gnmi has the same structure as gribi.

thanks!

[robshakir]

system/gnmi also has a different pattern here:

▶ find system/gnmi -name feature.textproto
system/gnmi/feature.textproto
system/gnmi/cliorigin/feature.textproto
system/gnmi/get/feature.textproto
system/gnmi/metadata/feature.textproto

here, should system/gnmi/feature.textproto really be under something like system/gnmi/base/feature.textproto?

[xw-g]

I don't have all the answers for now, but just to record some quick thoughts for discussion (or come back with more thoughts). I think the problems/questions are about:

1. How to represent API services in Feature Profile.
2. What are the typical use cases and how are we going to consume the representation of the API services.

[liulk]

I have two main proposals:

• Separate feature definition and test code. http://shortn/_PgvVwE5YEE (google private link)
• Remove 1-1 correspondence between feature and test. OC coverage will be defined per test, and we'll supply a tool to find the tests covering a specific feature. http://shortn/_6tnpkwp0a7 (google private link)

I was hoping to get aligned internally first before soliciting public feedback. Is that alright?