Call module pre_validate hook based on attributes present in topology, not module list

[ipspace]

Unfortunately we got into a complicated mess where modules use data structures belonging to other modules (evpn => vxlan => vlan, evpn => vrf).

Even worse, EVPN module does not require VRF module, so there's no way to detect that something is missing.

The "only" challenge identified so far is the lack of data structure normalization. For example, the VRF module cleans up the vrfs dictionary in pre_transform hook, so the EVPN module can safely use those structures, but if the topology does not use the VRF module, those structures could be anything, and the EVPN module dies a horrible death (that's why we have to check the None values in EVPN module).

To solve this conundrum, we should:

• Add a new pre_validate hook that would clean up/validate shared structures (but not necessarily make any other adjustments. This hook SHOULD NOT refer to any external data (because it may not be clean yet). Note that we cannot use the pre_default hook because that one has to add extra attributes.
• Call pre_validate hooks based on attributes present on in the topology/nodes, not on modules listed in the topology.

The change would go into 1.4.

Any thoughts on this one @jbemmel?

[ipspace]

Test case:

#
# Test EVPN with vrfs but without vrf module
#
# Should report an error but not crash
#

module: [ bgp, evpn ]
bgp.as: 65000

vrfs:
  red:

nodes:
  r1:
    device: eos

links:
- r1:
  vrf: red

[jbemmel]

I'm thinking we have these hooks per entity class, currently:

• module
• node
• link ( * provider )

with hooks:

• new pre_validate (which only applies to modules)
• pre_transform
• transform
• post_transform

However, there are also these additional entities / constructs:

• vrfs
• vlans
• interfaces (created as the intersection of links, vlans and nodes, potentially within a vrf)
• groups

[ unfinished thought, draft saved ]

[ipspace]

FYI, the hooks are not based on entity classes but on the steps of the data transformation, which are roughly:

• Setting things up
• Before the transformation
• Transforming nodes
• Transforming links
• After the transformation

See https://netsim-tools.readthedocs.io/en/dev/dev/transform.html for details.

[ipspace]

As a collateral "damage" of thorough attribute checks we can no longer get into a position where a module would use 'vrfs' global or node attribute without VRF module being enabled -- global and node attributes are checked before pre_transform module calls.

Just make sure you don't touch other modules' data in init or pre_default callbacks ;)