kmisak commented 1 month ago

Feature Description

Previous versions of Hyperglass natively supports VRF, new one - not. It can be done with custom directives, but all pretty formatting of output is lost.

Is your feature request related to a problem? Please describe.

Lot of networks use VRFs, removing support for the reduces Hyperglass big flexibility.

Environment & Use Case

Any contemporary router supports VRFs

Additional Context

thatmattlove commented 1 month ago

Try adding a custom directive and setting the following attribute:

your-directive:
    name: BGP Route (VRF x)
    table_output: __hyperglass_juniper_bgp_route_table__

And on the device:

devices:
    - name: your device
      platform: juniper
      directives:
          - your-directive
      structured_output: true

Supported values for table output are:

`table_output`	Platform	Type
`__hyperglass_juniper_bgp_route_table__`	Juniper	BGP Route
`__hyperglass_juniper_bgp_aspath_table__`	Juniper	BGP AS Path
`__hyperglass_juniper_bgp_community_table__`	Juniper	BGP Community
`__hyperglass_arista_eos_bgp_route_table__`	Arista	BGP Route
`__hyperglass_arista_eos_bgp_aspath_table__`	Arista	BGP AS Path
`__hyperglass_arista_eos_bgp_community_table__`	Arista	BGP Community

If this works, I will come up with a way to make this more user friendly and add docs.

kmisak commented 1 month ago

Thank you for suggestion, but that doesn't work. Table output uses | display xml format as input, I tried to use that too - no luck.

Here is device sample:

routers:

name: mx-moscow address: 10.x.x.x group: AS49800 credential: username: xxxx password: xxxxx port: 22 platform: juniper driver: netmiko attrs: source4: x.x.x.x source6: x::x directives:
- builtins: false
- bgp_route_vrf
- ping_vrf
- traceroute_vrf structured_output: true

And directive:

bgp_route_vrf: name: BGP Route rules:

condition: 0.0.0.0/0 action: permit command: "show route protocol bgp table AS49800.inet.0 {target} detail"
condition: ::/0 action: permit command: "show route protocol bgp table AS49800.inet6.0 {target} detail" field: description: "IP Address, Prefix, or Hostname" table_output: hyperglass_juniper_bgp_route_table

cameront-1 commented 4 weeks ago

Confirming this issue exists on Arista as well. Same instructions. Custom directives result in loss of pretty-formatting, instructing table/structured output has no effect

    platform: arista_eos
    directives:
       - test3
    structured_output: true

test3:
  name: BGPtest
  table_output: __hyperglass_arista_eos_bgp_route_table__
  rules:
    - condition: '0.0.0.0/0'
      ge: 4
      le: 32
      command: 'show ip bgp {target} vrf INTERNET | json'

If I modify base files and force the base config to query against another VRF, we get the following errors being thrown:

hyperglass-1  | [CRITICAL] 20240607 13:31:38 |63 | parse_arista → 11 validation errors for AristaBGPTable
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.0.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591610, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.0.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215590574, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591602, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591603, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215596861, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591605, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215593314, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type {'plugin': 'BGPRoutePluginArista'}
hyperglass-1  | [CRITICAL] 20240607 13:31:38 |48 | default_handler → Error {'method': 'POST', 'path': '/api/query', 'detail': 'expected str, got list'}
hyperglass-1  | ERROR - 2024-06-07 13:31:38,675 - litestar - config - Uncaught exception (connection_type=http, path=/api/query):
hyperglass-1  | Traceback (most recent call last):
hyperglass-1  |   File "/opt/hyperglass/hyperglass/plugins/_builtin/bgp_route_arista.py", line 42, in parse_arista
hyperglass-1  |     validated = AristaBGPTable(**routes)
hyperglass-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/pydantic/main.py", line 176, in __init__
hyperglass-1  |     self.__pydantic_validator__.validate_python(data, self_instance=self)
hyperglass-1  | pydantic_core._pydantic_core.ValidationError: 11 validation errors for AristaBGPTable
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.0.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591610, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.0.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215590574, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591602, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.1.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591603, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215596861, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.2.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.0
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215591605, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.1
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=595497215593314, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  | bgpRouteEntries.`1.1.1.0/24`.bgpRoutePaths.3.routeDetail.extCommunityListRaw.2
hyperglass-1  |   Input should be a valid string [type=string_type, input_value=876972192302256, input_type=int]
hyperglass-1  |     For further information visit https://errors.pydantic.dev/2.7/v/string_type
hyperglass-1  |
hyperglass-1  | During handling of the above exception, another exception occurred:
hyperglass-1  |
hyperglass-1  | Traceback (most recent call last):
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/litestar/middleware/_internal/exceptions/middleware.py", line 158, in __call__
hyperglass-1  |     await self.app(scope, receive, capture_response_started)
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/litestar/routes/http.py", line 80, in handle
hyperglass-1  |     response = await self._get_response_for_request(
hyperglass-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/litestar/routes/http.py", line 132, in _get_response_for_request
hyperglass-1  |     return await self._call_handler_function(
hyperglass-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/litestar/routes/http.py", line 152, in _call_handler_function
hyperglass-1  |     response_data, cleanup_group = await self._get_response_data(
hyperglass-1  |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/usr/local/lib/python3.12/site-packages/litestar/routes/http.py", line 200, in _get_response_data
hyperglass-1  |     else await route_handler.fn(**parsed_kwargs)
hyperglass-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/api/routes.py", line 110, in query
hyperglass-1  |     output = await execute(data)
hyperglass-1  |              ^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/execution/main.py", line 69, in execute
hyperglass-1  |     output = await driver.response(response)
hyperglass-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/execution/drivers/_common.py", line 43, in response
hyperglass-1  |     response = self.plugin_manager.execute(output=output, query=self.query_data)
hyperglass-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/plugins/_manager.py", line 187, in execute
hyperglass-1  |     result = plugin.process(output=result, query=query)
hyperglass-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/plugins/_builtin/bgp_route_arista.py", line 91, in process
hyperglass-1  |     return parse_arista(output)
hyperglass-1  |            ^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/plugins/_builtin/bgp_route_arista.py", line 64, in parse_arista
hyperglass-1  |     raise ParsingError(err.errors()) from err
hyperglass-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/exceptions/_common.py", line 176, in __init__
hyperglass-1  |     self._message = self._safe_format(message, **kwargs)
hyperglass-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/exceptions/_common.py", line 62, in _safe_format
hyperglass-1  |     keys = get_fmt_keys(template)
hyperglass-1  |            ^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/opt/hyperglass/hyperglass/util/tools.py", line 117, in get_fmt_keys
hyperglass-1  |     for block in (b for b in string.Formatter.parse("", template) if isinstance(template, str)):
hyperglass-1  |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  |   File "/usr/local/lib/python3.12/string.py", line 288, in parse
hyperglass-1  |     return _string.formatter_parser(format_string)
hyperglass-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyperglass-1  | TypeError: expected str, got list

Pre 2.0 vrf usage wasn't impacted. I would consider this a major regression.

szferguson commented 3 weeks ago

Following up on this, we tried adding the __hyperglass_arista_eos_bgp_route_table__ attribute and referencing this directive on each device.

directives.yaml

test3:
  name: BGPtest
  table_output: __hyperglass_arista_eos_bgp_route_table__
  rules:
    - condition: '0.0.0.0/0'
      ge: 4
      le: 32
      command: 'show ip bgp {target} vrf INTERNET | json'

devices.yaml

  - name: AMSCR6
    ...
    directives:
       - test3
    structured_output: true

However, the only way for the table to appear was adding test3 in the directives tuple inside bgp_route_arista.py:

directives: t.Sequence[str] = (
    "__hyperglass_arista_eos_bgp_route_table__",
    "__hyperglass_arista_eos_bgp_aspath_table__",
    "__hyperglass_arista_eos_bgp_community_table__",
    "test3"
)

thatmattlove / hyperglass

Bring back VRF native support #252

Feature Description

Is your feature request related to a problem? Please describe.

Environment & Use Case

Additional Context