Open WalkerD243 opened 3 weeks ago
I have walked through the code and find out two possiblitiy location that might cause this bug.
at hyperglass/models/directive.py:317
validate_plugins function
316 @field_validator("plugins")
317 def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]:
318 """Validate and register configured plugins."""
319 plugin_dir = Settings.app_path / "plugins"
320
321 if plugin_dir.exists():
322 # Path objects whose file names match configured file names, should work
323 # whether or not file extension is specified.
324 matching_plugins = (
325 f
326 for f in plugin_dir.iterdir()
327 if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
328 )
329 return [str(f) for f in matching_plugins]
330 return []
plugin_dir
, I think that currently it only reads from the file /etc/hyperglass/plugins
. Instead of any location provided by the derivatives.Therefore, the current solution is to
/etc/hyperglass/plugins
Thank you very much for your time. I changed directiy.py as to your solution to: 324 matching_plugins = [ 325 f 326 for f in plugin_dir.iterdir() 327 if f.name.split(".")[0] in (p.split(".")[0] for p in plugins) 328 ]
And put the plugins into the plugins directory but have the same results. I tried to put a print statement into the function in directive.py (without the arrows): def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]: """Validate and register configured plugins.""" plugin_dir = Settings.app_path / "plugins" --->print("PLUGIN_DIR: " + plugin_dir)<---------- if plugin_dir.exists():
# whether or not file extension is specified.
matching_plugins = [
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
]
return [str(f) for f in matching_plugins]
return []
And the live output from the container doesnt show PLUGIN_DIR... should that be there or will a print in that function not work?
It seems that with docker, the code is actually run at /opt/hyperglass
, /etc/hyperglass
are only for setting files. My suggestion is to also mount the directory to /opt/hyperglass
e.i.
volumes:
- ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/etc/hyperglass
- ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/opt/hyperglass
if you have done correclty, it should print something out.
Thanks for the suggestion ! I changed the volumes as above and got the error:
hyperglass-1 | /usr/local/bin/python3: Error while finding module specification for 'hyperglass.console' (ModuleNotFoundError: No module named 'hyperglass')
hyperglass-1 exited with code 1
when starting the container. I changed
- ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/opt/hyperglass
to |
- ${HYPERGLASS_APP_PATH-/opt/hyperglass}:/opt/hyperglass
and it works as expected, printing the output directly into the container output on the cli.
With that I could troubleshoot a bit and found a potential solution: In /opt/hyperglass/hyperglass/models/directive.py I added some prints to show the contents of plugins and plugin_dir.iterdir() beginning in line 316:
@field_validator("plugins")
def validate_plugins(cls: "Directive", plugins: t.List[str]) -> t.List[str]:
"""Validate and register configured plugins."""
plugin_dir = Settings.app_path / "plugins"
if plugin_dir.exists():
# Path objects whose file names match configured file names, should work
# whether or not file extension is specified.
----> for p in plugins:
print(f"ALLOWED PLUGIN: {p.split(".")[0]}")
for f in plugin_dir.iterdir():
print(f"FOUND PLUGIN: {f.name.split(".")[0]}")
matching_plugins = (
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
)
return [str(f) for f in matching_plugins]
return []
and got the following result from the container:
hyperglass-1 | ALLOWED PLUGIN: /etc/hyperglass/plugins/transform_plugin
hyperglass-1 | FOUND PLUGIN: transform_plugin
hyperglass-1 | ALLOWED PLUGIN: /etc/hyperglass/plugins/transform_plugin
hyperglass-1 | FOUND PLUGIN: transform_plugin
To fix that problem I have following two solutions:
1: Swap lines 324-328 from
matching_plugins = (
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.split(".")[0] for p in plugins)
)
to
matching_plugins = []
for f in plugin_dir.iterdir():
for p in plugin_dir.iterdir():
if f.name.split(".")[0] == p.name.split(".")[0]:
matching_plugins.append(f)
2: Transform the strings in plugins.iterdir() to Path Objects and insert a .name. in the comparison of f and p:
plugins = [Path(p) for p in plugins]
matching_plugins = (
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.name.split(".")[0] for p in plugins)
)
But with approach 2. you have to also add
from pathlib import Path
at the beginning.
With these changes applied I am now getting:
hyperglass-1 | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'BGPRoutePluginArista'}
hyperglass-1 | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'BGPRoutePluginJuniper'}
hyperglass-1 | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'MikrotikGarbageOutput'}
hyperglass-1 | [DEBUG] 20241105 11:57:24 |116 | register → Registered built-in plugin {'type': 'output', 'name': 'RemoveCommand'}
hyperglass-1 | [INFO] 20241105 11:57:24 |118 | register → Registered plugin {'type': 'input', 'name': 'TransformCIDR'}
What do you think about it, maybe theres a better solution.
Well, since I am not the author/maintainer of this project. I think that we should leave the answer to him; however he is disappeared from the Internet currently. Still, love to see that you have solve the problem. As for my opinion of the solutions
matching_plugins = (
f
for f in plugin_dir.iterdir()
if f.name.split(".")[0] in (p.name.split(".")[0] for Path(p) in plugins)
)
I think solution 2 with a slightly change might be a better solution, since it have the least change to the original code. Also I think that you might have some typo in solution 1?
matching_plugins = []
for f in plugin_dir.iterdir():
for p in plugins:
if f.name.split(".")[0] == p.split(".")[0]:
matching_plugins.append(f)
Not tested yet, but I think this is your original intention?
Thanks for your help ! Definitely the cleaner approach but the Path(p) throws errors on my instance. I kept
plugins = [Path(p) for p in plugins] matching_plugins = ( f for f in plugin_dir.iterdir() if f.name.split(".")[0] in (p.name.split(".")[0] for p in plugins) )
for further testing and discovered that my transform_input.py plugin is registered and executed. The line
target_network = ip_network(target)
throws errors as well but this is okay for me as I'm going to create own rules in the plugin.
I created a Output plugin juniper-bgp-route.py:
from hyperglass.plugins import OutputPlugin
print("OUTPUT PLUGIN CALLED")
class TestOutput(OutputPlugin):
def process(self, output, query):
print("FUNCTION CALLED")
return f"Plugin Test Successfull: {query}"
which gets registered in the container start but not executed after the query ( yes i bound that in another command in the directives.yaml). For further troubleshooting I did in /opt/hyperglass/hyperglass/plugins/_manager.py:
class OutputPluginManager(PluginManager[OutputPlugin], type="output"):
"""Manage Output Processing Plugins."""
def execute(self: "OutputPluginManager", *, output: OutputType, query: "Query") -> OutputType:
"""Execute all output parsing plugins.
The result of each plugin is passed to the next plugin.
"""
print(f"execute in OutputPluginManager Class in _manager.py called with query id: {query.directive.id} and platform: {query.device.platform}")
result = output
for plugin in self.plugins():
print(f"self plugin: {plugin}")
print(f"plugin directives: {plugin.directives}")
print(f"plugin platform: {plugin.platforms}")
directives = (
plugin
for plugin in self.plugins()
if query.directive.id in plugin.directives and query.device.platform in plugin.platforms
)
print(f"DIRECTIVES: {directives}")
common = (plugin for plugin in self.plugins() if plugin.common is True)
for plugin in (*directives, *common):
log.bind(plugin=plugin.name, value=result).debug("Output Plugin Starting Value")
result = plugin.process(output=result, query=query)
log.bind(plugin=plugin.name, value=result).debug("Output Plugin Ending Value")
if result is False:
return result
# Pass the result of each plugin to the next plugin.
return result
and did the query as before and now got the output:
hyperglass-1 | [DEBUG] 20241108 07:26:43 |51 | collect → Connecting to device {'device': 'Oldenburg Test', 'address': 'None:None', 'proxy': None}
hyperglass-1 | execute in OutputPluginManager Class in _manager.py called with query id: juniper-received-from-peer and platform: juniper
hyperglass-1 | self plugin: TestOutput
hyperglass-1 | plugin directives: ('juniper-received-from-peer',)
hyperglass-1 | plugin platform: ()
hyperglass-1 | self plugin: BGPRoutePluginArista
hyperglass-1 | plugin directives: ('hyperglass_arista_eos_bgp_route_table', 'hyperglass_arista_eos_bgp_aspath_table', 'hyperglass_arista_eos_bgp_community_table__')
hyperglass-1 | plugin platform: ('arista_eos',)
hyperglass-1 | self plugin: BGPRoutePluginJuniper
hyperglass-1 | plugin directives: ('hyperglass_juniper_bgp_route_table', 'hyperglass_juniper_bgp_aspath_table', 'hyperglass_juniper_bgp_community_table')
hyperglass-1 | plugin platform: ('juniper',)
hyperglass-1 | self plugin: MikrotikGarbageOutput
hyperglass-1 | plugin directives: ('hyperglass_mikrotik_bgp_aspath', 'hyperglass_mikrotik_bgp_community', 'hyperglass_mikrotik_bgp_route', 'hyperglass_mikrotik_ping', 'hyperglass_mikrotik_traceroute__')
hyperglass-1 | plugin platform: ('mikrotik_routeros', 'mikrotik_switchos')
hyperglass-1 | self plugin: RemoveCommand
hyperglass-1 | plugin directives: ()
hyperglass-1 | plugin platform: ()
hyperglass-1 | DIRECTIVES: <generator object OutputPluginManager.execute.
so it looks like there's no platform set for the plugin and therefore the plugin fails the check in directives = ( plugin for plugin in self.plugins() if query.directive.id in plugin.directives and query.device.platform in plugin.platforms )
based on the builtin plugins i added the marked line to my plugin juniper-bgp-route.py:
import re, json
from hyperglass.plugins import OutputPlugin
------> from typing import List, Sequence <-----------
print("OUTPUT PLUGIN CALLED")
class TestOutput(OutputPlugin):
-------> platforms: Sequence[str] = ("juniper",) <----------------
def process(self, output, query):
print("FUNCTION CALLED")
return f"Plugin Test Successfull: {query}"
Now the output plugin works as expected and the query output is Plugin Test Successfull...
You mentioned that the creator is currently not responding on this project therefore I'm thinking about forking my changes, so that others with the same problems have a solution until the creator fixes these problems in this original repository .
With that in mind I would also remove the platform check in the /opt/hyperglass/hyperglass/plugins/_manager.py from
directives = (
plugin
for plugin in self.plugins()
if query.directive.id in plugin.directives and query.device.platform in plugin.platforms
)
to
directives = (
plugin
for plugin in self.plugins()
if query.directive.id in plugin.directives
)
I am also thinking of forking one and fix other issues. Perhaps create a hyperglass 2 or something. Do U mind inviting U to create a PR after the new hyperglass is created?
Sure would be glad, after all your help.
Deployment Type
Docker
Version
v2.0.4
Steps to Reproduce
Create directives.yaml: juniper-bgp-route: name: BGP Route Custom plugins:
field: description: 'Your IPv4 or IPv6 BGP Peer Address' validation: '[0-9a-f.\:\/]+'
Create plugin based on https://hyperglass.dev/plugins from ipaddress import ip_network from hyperglass.plugins import InputPlugin
class TransformCIDR(InputPlugin): def transform(self, query): (target := query.query_target) target_network = ip_network(target) if target_network.version == 4: return f"{target_network.network_address!s} {target_network.netmask!s}" return target
put the transform_plugin.py in /etc/hyperglass/plugins and /etc/hyperglass/custom_plugins folder
check that the container can see the plugins: docker container exec hyperglass-hyperglass-1 cat /etc/hyperglass/plugins/transform_plugin.py docker container exec hyperglass-hyperglass-1 cat /etc/hyperglass/custom_plugins/transform_plugin.py
enable debug mode like in https://hyperglass.dev/installation/environment-variables
enter the hyperglass site and enter "192.0.2.0/24" in the query as in https://hyperglass.dev/plugins example
observe the live output from the container and look at the line: hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0/24 detail | display json']}
look at the result in the webinterface, as it should transform 192.0.2.0/24 to 192.0.2.0 255.255.255.0 and the query is executed on a juniper device it should return a syntax error at the 255.255.255.0
Expected Behavior
I would expect that the constructed_query line form step 7. above would be: hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0 255.255.255.0 detail | display json']}
and the result in the webinterface should return a syntax error at the 255.255.255.0 as it is executed on a juniper device in my case
Observed Behavior
I get the line: hyperglass-1 | [DEBUG] 20241030 14:26:43 |129 | queries → Constructed query {'type': 'juniper-bgp-route', 'target': ['192.0.2.0/24'], 'constructed_query': ['show route protocol bgp table inet.0 192.0.2.0 255.255.255.0 detail | display json']}
and a normal result in the webinterface as if the query wasn't changed by the plugin
For me it seems like the plugin is not applied at all, even if change the last line in the plugin config to "return 'test'" the query is still successfull. I tried different folders in /etc/hyperglass and giving 777 full rights to the folder and files inside.
The same thing happens with the output plugins, please have a look. this is ssuch a great platform!
Configuration
Devices
Logs