nornir-automation / nornir

Pluggable multi-threaded framework with inventory management to help operate collections of devices
https://nornir.readthedocs.io/
Apache License 2.0
1.39k stars 235 forks source link

Discussion: Complex connections depending on other hosts #247

Closed dmfigol closed 6 years ago

dmfigol commented 6 years ago

Here is the problem that bothers me and I don't know how it fits into Nornir and if we can do something to support this.

Example 1: ESXI. You have connection params to vCenter (ip, username, password) and list of VMs with their nics, vlans, etc. If you are using python and pyvmomi (plugin to work with esxi), you would create a connection to ESXI, then using this object, you would perform actions on your vms. And the end, if you want to guarantee for task completion, there is wait_for_tasks routine to be used on the connection How would you describe the inventory? Ideologically, I would have vcenter as one host and then every VM as a separate host too, so that I can benefit from threading, but I need a single connection object!

Example 2: requests.Session. Sometimes you need to use requests.Session to set cookies for subsequent calls, set common headers, etc. Again, shared connection, though we don't need an actual connection plugin, so potentially we can pass session object to an existing task plugin

Example 3: netmiko with intermediate SSH proxy (from slack). Yesterday, there was the following conversation: a netmiko user needed to use an intermediate server as SSH proxy, while the original script is run on Window and ssh_config file would not work. Kirk proposed a solution with manual connection to the server, then doing another ssh connection from there and doing redispatch. How would inventory look like in this case? You have a jumpbox as a host and network devices as other hosts, for every network device, you would need to connect to the jumpbox, do some magic and then have that connection available for the host.

Thoughts?

dbarrosop commented 6 years ago

Example 1

So I think this is a bit of a special case where instead of a connection plugin we should use something like a singleton. This has the disadvantage you can only manage one single ESX instance but it'd be simple to implement and use:

class ESX(object):
     connection = None

    @classmethod
    def connect(cls, hostname, username, ....)
         self.connection = connect_to_esx(hostname, username....)

     @classmethod
     def create_vm(cls, task, ...):
            self.connection.create_vm(task.host.name)

     @classmethod
     def destroy_vm(cls, task, ...):
            self.connection.destroy_vm(task.host.name)

From a user perspective this would be used as:

ESX.connect(...)
nr.run(task=ESX.create_vm)

def my_grouped_task(task):
    task.run(task=ESX.create_vm, ...)

nr.run(task=my_grouped_task)

an alternative would be to have that to be a proper class in which case things would work similar:

my_esx = ESX().connect(...)
nr.run(task=my_esx.create_vm)

# the exception is when using grouped_tasks

def my_grouped_task(task, my_esx):
    task.run(task=my_esx.create_vm, ...)

my_esx = ESX().connect(...)
nr.run(task=my_grouped_task, my_esx=my_esx)

Does this make sense?

Example 2

You can already do that, ish. So basically the http_method will accept any parameter that requests accept, meaning you can do something like:

nr.run(task=http_method, url="blah", headers=my_headers)
nr.run(task=http_method, url="bleh", headers=my_headers)
nr.run(task=http_method, url="blih", headers=my_headers)

Example 3

I think that's a non-existent problem. Either run the code in the jumpserver or fix your local ssh setup. According to paramiko's documentation ssh_config should work on windows:

http://docs.paramiko.org/en/2.4/api/config.html

ktbyers commented 6 years ago

The example3, for standard SSH proxy and Linux server, we can have that work with Nornir. The solution will probably be to specify an environment variable or Nornir config option which points to an SSH config file (and says to use it in some way). IIRC we already have this for the Paramiko connection plugin, but I need to integrate it into the Netmiko connection plugin (and ensure we have some behavior consistency there between the Netmiko plugin and the Paramiko plugin).

For end-users that create really complicated scenarios (multiple SSH jump servers, Windows to SSH jump server), I think the onus is on them to come up with a solution.

There potentially are solutions to these setups, but they start getting to be really complicated (and they are complicated because of the setup the user or the user's security team has chosen).

dbarrosop commented 6 years ago

+100000

The Config PR has a mechanism to specify ssh options (well, we already have it but it's cleaner there).

dmfigol commented 6 years ago

@dbarrosop ex1 thanks, interesting idea, haven't thought about it ex2 actually you can't, because requests.requests does not support session parameter: https://github.com/requests/requests/blob/master/requests/api.py#L16 Is it worth a separate feature request? ex3 I will rely on Kirk's opinion here, since I have minimal netmiko knowledge.

dbarrosop commented 6 years ago

re ex2, we could build on the same pattern as "ex1"

dmfigol commented 6 years ago

Ok makes sense. As there is no action item has been resurfaced, I will just proceed with issue closure.

renato-alves-barros commented 4 years ago

Hello @dmfigol,

I was interested in knowing if nornir 3 introduced some enhancements to allow having a jump box (without ssh_config). I have developed a connection plugin to handle this. However, if I try to connect to 2x final hosts that share the same jump box it fails as apparently even with the serial runner on config.yaml, nornir tries to connect to both hosts in parallel.

My problem with the ssh_config file is that I support lots of 3rd party customers/companies and they all offer me a jump box. However, since I do not control those jump servers, I cannot configure the ssh server.

dbarrosop commented 4 years ago

If your problem is the amount of concurrent connections you can open/close connections on demand. Combined with num_workers to control number of threads you should be able to fix your problem. There are docs explaining how to manage connections manually

renato-alves-barros commented 4 years ago

Thanks David,

I just found out. Should be much more simpler then that. I simply cannot connect to more then one host with nornir 3. Even with a very simple configuration.

I opened a thread on the discourse group to see if someone can help:

https://nornir.discourse.group/t/nornir-3-unable-to-connect-to-more-then-one-device/575

Thanks for the effort with nornir!

On Wed, Oct 14, 2020 at 12:06 PM David Barroso notifications@github.com wrote:

If your problem is the amount of concurrent connections you can open/close sessions on demand. Combined with num_workers to control number of threads you should be able to fix your problem. There are docs explaining how to manage connections manually

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nornir-automation/nornir/issues/247#issuecomment-708330914, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHQSJFK334QSHRWSFBEGTYLSKWA2ZANCNFSM4FUWDTNQ .