Multiple tasks on one host at the same time in threads

[dmfigol]

This issue was created to discuss the possibility of running multiple tasks at the same time on one host in threads. There are two things that are needed to make this work:

• Implement an actual grouping mechanism. For example nr.run could accept a tuple of args instead of a single task
• Implement automatic connection handling for concurrent tasks. For example, if both of the tasks use napalm connection, the second task should open a new napalm-2 connection and use that.

[dbarrosop]

* For example `nr.run` could accept a tuple of args instead of a single task

This is a bit more complex than this. You have to be able to pass different arguments to different tasks. Otherwise, you'd be running the same exact thing N times in parallel. So basically it requires a new entire method.

napalm connection, the second task should open a new napalm-2 connection and use that

Why is that necessary?

Overall I feel skeptical about this and my vote would be to keep it on hold until we have more use cases to see (a) if it's really needed and (b) what is it that people actually wants to do and why. In the meantime, a simple workaround for this is as simple as just "hacking" your inventory a little:

# groups.yaml
my_router00:
    #  this group is our actual device
   ...

# users.yaml
# We create one device per thread inheriting from my_router00
my_router00_thread00:
   groups: [my_router00]
my_router00_thread01:
   groups: [my_router00]
my_router00_thread02:
   groups: [my_router00]
my_router00_thread03:
   groups: [my_router00]

[ktbyers]

Yes, I agree with David here i.e. let's punt this issue 6 months or 12 months down the road and see how important of a use case it is. I am definitely concerned about the added complexity it will entail and whether that complexity is worth it.

[dmfigol]

I agree with you all :) We need use-cases from users to understand if we want to add this. We probably also need some issue tag for such issues, something like "use-cases-required"?

[dbarrosop]

yeah, that sounds like a great idea.

[brandomando]

The usecase that I would like to use something like this for would be operations that take a significant amount of time to run, but do not risk pegging the remote device's CPU. For example, I use nornir to run a post-change data collection/validation runbook, and in that runbook, I use seed devices throughout the network and run traceroutes to specific hosts in the data centers to determine if the path to the DC's has changed. The traceroutes take a significant amount of time to complete, and we are running them to about 16 different hosts from each seed device. We can use parallelism between each seed device, but since some of the traces can take a minute or so to complete, the runbook takes 10-13 minutes to complete all of its tasks. If we could run 8 or so parallel threads, that would reduce the traces from taking 9 minutes down to 2 minutes. The runbook is intended to give an engineer feedback on their change and whether they should roll back, so the faster we can return results to the engineer, the faster we can isolate outages or issues after the change.

For me, I could use David's inventory hack, but if the inventory source was something other than a simple inventory file, thisb could be problematic, unless I use the new Inventory methods to dynamically generate the new groups, but the feature would be useful for this purpose.

[dbarrosop]

FWIW I am thinking in looking into this as part of the asyncio interface.

[dbarrosop]

Ok, I think I have an example of how this may work for you:

https://github.com/nornir-automation/nornir/pull/390/files#diff-94f03aeb1ea29ce42ec8454749681fcaR132

print_result won't work out of the box but it should be doable to create one that does.

Don't bother too much with the changes in the API, this is not going to happen anytime soon (might not even happen at all), but I am playing with several ideas that may lead to a cleaner core.

[uthman-s]

Hi guys,

First post here, loving Nornir so far!

I have run into a case, where I would like to configure all 48 interfaces in a C9300, where some of them have to be enabled and others shut. I am currently iterating through each interface from Netbox and using netmiko_send_config to send the config. This takes around 2-3 minutes.

I know that I can send the config for all 48 interfaces as a single string seperated by '\n', however, this won't allow to track the changes and task status for each interface. If the task fails, then it's impossible to track which part of the configuration failed in the string.

Is there a better way?

[dmfigol]

@uthman-s this is off-topic for this issue. I would recommend creating a separate thread on our forum: https://nornir.discourse.group/ Short answer: on cat9k there are NETCONF and RESTCONF available and I would build NETCONF XML for all 48 interfaces and push it via netconf. Nornir also supports netconf now.

[ktbyers]

Another option is to use napalm_configure and probably a merge operation (given what you are doing). This will transfer the configuration via Secure Copy and commit it as a single operation.

You will still potentially have a bit of an issue of tracking where it failed however (though NAPALM should auto rollback on failure so at least it should return the device to its original state).