Consolidate inventory attributes

[dbarrosop]

1. network_api_port/ssh_port -> port
2. os/network_operating_system -> operating_system

[dmfigol]

@dbarrosop While I definitely agree that the consolidation seems like a good idea, I am also thinking how I would specify connections details for my current project, where we use SSH/NETCONF/Telnet to console port for each device - not sure If I would like to duplicate devices in the inventory :(

[dbarrosop]

I was thinking for those cases to use $connection_options. For instance:

nornir_port: 22  # this is the default port
napalm_options:
   port: 443    # override port for napalm

And that should apply for everything (user/password/etc), not just the port.

[ktbyers]

I actually started writing a proposal on this last night. I will add some comments in a couple of hours

Kirk

On Fri, Jul 27, 2018 at 2:29 AM David Barroso notifications@github.com wrote:

I was thinking for those cases to use $connection_options. For instance:

nornir_port: 22 # this is the default port napalm_options: port: 443 # override port for napalm

And that should apply for everything (user/password/etc), not just the port.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nornir-automation/nornir/issues/202#issuecomment-408365572, or mute the thread https://github.com/notifications/unsubscribe-auth/AA26hJ0-GJDdQtaVDOX4qbbEABzB1JdLks5uKt2IgaJpZM4VjIVs .

-- Kirk Byers ktbyers@twb-tech.com ktbyers@twb-tech.com Simplify through Automation

[ktbyers]

/ This can be ignored as I think it wrong...now /

I agree that we should consolidate port to a single attribute and operating_system to a single attribute. I think we need some other properties though.

vendor (optional) operating_system or platform (or whatever we want to term this i.e. what used to be nos) transport port

operating_system would map to the following:

eos junos linux nxos others

transport would map to one of the following:

eapi netconf nxapi ssh telnet serial

I will provide some more details on why below.

[ktbyers]

This is more information for discussion on this issue...

I went through some combinations of drivers (napalm/netmiko/paramiko) and platforms:

# Arista
EAPI
     domain socket (napalm)
     http > port (napalm)
     https > port (napalm)
CLI
     SSH > port (netmiko, paramiko)
     telnet > port (netmiko)
     serial (netmiko)

# Juniper
NETCONF 
      netconf over SSH > port (napalm)
CLI
      SSH > port (netmiko, paramiko)
      telnet > port (netmiko)
      serial  (netmiko)

# NX-OS
NX-API
       http > port (napalm)
       https > port (napalm)
 CLI
       SSH > port (netmiko, napalm, paramiko)
       telnet > port (netmiko)
       serial (netmiko)

# IOS
CLI
      SSH > port (netmiko, napalm)
      telnet > port (netmiko)
      serial (netmiko)

The netmiko standard for handling this issue is a field called device_type. device_type is an overloaded field with the form of vendor_nos_transport. So for example, cisco_ios_ssh or hp_comware_telnet. Optionally, you can drop the transport and have SSH be assumed (so cisco_ios is identical to cisco_ios_ssh.

Netmiko also has some 40 different platforms so using only operating_system for this will be challenging/confusing (i.e. having an optional vendor field would be helpful).

[ktbyers]

One main problem I see with what I stated above is...if you specify the transport in inventory then that is already implying something about what connection type you intend to use.

In other words having an operating_system=eos and a transport=ssh and then calling the napalm driver doesn't make sense. A similar thing can be said about port (i.e. setting port=22 and then trying to go between the napalm connection plugin and the netmiko connection plugin).

[dmfigol]

what do you think about this?

r1:
  nornir_host: 192.168.122.41
  nornir_username: admin
  nornir_password: admin
  type: network_device  # network_device or linux
  netmiko_options:
    cisco_ios:
      port: 2222
      username: special-user  # override default
    terminal_server:
      ip: 1.2.3.4
      port: 2001
  netconf_options:  # assume default port 930
sw1:
  nornir_host: 192.168.122.51
  nornir_username: admin
  nornir_password: admin
  type: network_device
  napalm_options:
    eos:
      # assume default users or overwrite

It this case, it would be up to a plugin developer to document what options are supported. Since plugins are mostly wrappers around existing libraries, one would document, that plugin_options key would be napalm driver name or netmiko connection handler name, etc

[ktbyers]

I need to look at some more examples...doing that now.

[ktbyers]

@dmfigol What is the purpose of type? Where would you use it?

I assume the inner dictionaries in netmiko_options relates to #207?

I would rather do what I mentioned in #207 and have two different netmiko-connection types netmiko and netmiko_serial.

[dmfigol]

@ktbyers

What is the purpose of type? Where would you use it?

I was thinking about the same thing after I posted my suggestion. type seems to be redundant.

I assume the inner dictionaries in netmiko_options relates to #207?

Yes

I would rather do what I mentioned in #207 and have two different netmiko-connection types netmiko and netmiko_serial.

I like this too

[ktbyers]

I think all connection characteristics should not be high-level inventory attributes (i.e. not Host or Group attributes) and should instead:

1. Be automatically handled by the connection plugin or if that doesn't work,
2. Be specified in $connection_options

This would mean the port (ssh_port, network_api_port) attribute would be completely removed.

It doesn't really make sense to have it when you are toggling between different connection plugins using different transports.

Netmiko would default to SSH and if you needed to override this you could do so via $connection_options.

[dmfigol]

@ktbyers Could you clarify what you mean by "be automatically handled"? Specifying one operating system/platform/vendor and then plugin should figure out how to convert it to the driver name?

[ktbyers]

I think we need to talk about nos, platform, operating_system separately i.e. there is an issue of Napalm dev_os versus Netmiko device_type and how to handle that from an inventory perspective.

So this was only referring to the port and transport options (i.e. things that are clearly connection specific).

I do think vendor and nos are characteristics of the Host so it would be nice to have these be in inventory and map in some way (i.e. Host/Group properties). And have a logical way to map from them to NAPALM dev_os and Netmiko device_type (as opposed to duplicating this information in $connection_options).

[ktbyers]

napalm platforms are: eos, ios, iosxr, junos, nxos*, and community platforms

*nxos we have both nxos and nxos_ssh so we have a combination of platform and transport.

netmiko platforms are of this form: vendor_nos_transport (for example, cisco_ios_ssh, cisco_ios_telnet, cisco_ios_serial).

There are about 40 to 45 of them including some potentially with conflicting nos names (I would need to check that).

It would be nice at the inventory level to be able to support both napalm style platforms and netmiko style platforms.

Because of this, I see a few options:

1. Add a vendor field and leave the nos (renamed whatever we want to rename it). Netmiko would then assume vendor + nos + SSH for the default device_type. You could override this device_type using netmiko_options.

2. Change nos/os to be platform. If you have a napalm style platform, then Netmiko will feed it through a dictionary and return a netmiko style format. If you have netmiko style platform, then napalm would extract the nos from this and try to use that (possibly feeding it through a dictionary to make names consistent with napalm requirements).

3. Just keep nos (renamed operating_system) and assume it is napalm format. In practice, this really becomes the same as case2 (except with a bit more confusing naming and without a mapping dictionary in the napalm driver).

[dmfigol]

May be it is just me, but I am not sure I like any of these options. While, in general, every device has an operating system/platform/whatever, underlying libraries have different naming conventions and supported combinations. Maintaining any kind of mapping dictionary is an additional overhead. Also these mappings will become outdated very fast. There are so many differences just between two: netmiko and napalm, but nornir's capabilities don't end with these two libraries. Something else that comes to mind is ncclient, which I also need for my project. Here is the list of what they support:

Juniper: device_params={'name':'junos'} Cisco CSR: device_params={'name':'csr'} Cisco Nexus: device_params={'name':'nexus'} Cisco IOS XR: device_params={'name':'iosxr'} Cisco IOS XE: device_params={'name':'iosxe'} Huawei: device_params={'name':'huawei'} Alcatel Lucent: device_params={'name':'alu'} H3C: device_params={'name':'h3c'} HP Comware: device_params={'name':'hpcomware'} Server or anything not in above: device_params={'name':'default'}

There is also Cisco NSO, which has operating-system-to-NED-package mapping.

How to support this? Mapping dictionaries? What if a new library version adds support of a new vendor? While I understand it can be less convenient for users, I would like to have an explicit setting in Host/Group per ConnectionPlugin which will be transparently passed to the backend library.

[ktbyers]

@dmfigol Yes, that is all fair enough.

Having a vendor attribute and a nos attribute (or operating_system) is reasonable IMO. As these are both generic characteristics representing the device or group of devices. They are optional as to whether they are set and are optional as far as what string they contain.

In other words, I prefer option1 above.

Any mapping of these arguments would:

1. Be done in the connection plugin (i.e. the connection plugin is saying if your inventory is formatted this way, then using the connection plugin will be easier).
2. Would require a fallback option where $connection_options could be passed in (and $connection_options would take precedence).

What if the new library version adds support for a new vendor?

If the new library has a system to their naming scheme, then you could probably incorporate that automatically in your mapping. If the new library doesn't have any systematic approach to naming, then you would need to use $connection_options.

While I understand it can be less convenient for users, I would like to have 
an explicit setting in Host/Group per ConnectionPlugin which will be transparently 
passed to the backend library.

I am not sure what you are asking for here, can you clarify/expand on this a bit?

[dmfigol]

Having a vendor attribute and a nos attribute (or operating_system) is reasonable IMO

I think it could be valuable but I also think it should be optional. Also, as you said it should be generic, and not library specific. Documenting that could be a nightmare though, since it affects connection plugins but stored in a generic place.

Any mapping of these arguments would:

If both options are implemented, I think it is reasonable.

I am not sure what you are asking for here, can you clarify/expand on this a bit?

plugin_options basically

[dbarrosop]

I don't think we should do any of that magic to be honest. Mostly because it's extremely hard to explain/document/reason about. Specially as we start adding other connection plugins. I think we should do something as simple as just mapping simple attributes to their equivalent in each respective plugin and if users needs to do something special use directly the connection options. This would cover in an easy way the vast majority of use cases as people using more than one connection plugin is going to be rare while giving full power to people with more complex workflows.

plugin_options basically

Can you elaborate on this one? Shouldn't plugin_options just be regular parameters to the task plugin?

[dmfigol]

@dbarrosop would you like to have one attribute (operating_system) or two (operating_system and vendor)? Could you give a couple examples what mappings you think we should do?

Can you elaborate on this one? Shouldn't plugin_options just be regular parameters to the task plugin?

I was thinking about having plugin_options in the Inventory, like Kirk proposed in a #207:

  netmiko_ssh_options:
    port: 2222

  netmiko_serial_options:
    device_type: terminal_server
    host: 1.2.3.4
    port: 2001

Of course leaving the support of the command-line arguments

[dbarrosop]

I would do just operating_system as a generic attribute that gets mapped to "driver" in napalm, "device_type" in netmiko, "whatever" in whatever other plugin, etc... Otherwise nornir starts becoming an abstraction layer of abstraction layers adding lots of unnecessary complexity for a problem it's not ours to solve. If a user doesn't want this mapping because it needs to use both napalm and netmiko at the same time the user can ignore the operating_system parameter and user napalm_options and netmiko_options to assign the correct parameters. I am also ok scrapping operating_system altogether and having people specify the right parameter directly under $connection_plugin (maybe even scrapping all the nornir_* parameters and just having everything under $connection_plugin).

If this becomes a problem and we want to address it in the future we can do so but I think right now we should make this as simple and flexible as possible as it's easier to add in the future than to change it if we don't get the right implementation. In summary, if we keep nornit_host, nornir_username, nornir_password,nornir_operating_system, etc. these parameters should be default parameters that the connection plugins use in case they are not specified under$connection_plugin`.

I was thinking about having plugin_options in the Inventory, like Kirk proposed in a #207:

I don't think we are interpreting this that the same way but let's discuss it in that ticket.

[ktbyers]

Yes, I definitely agree with this statement:

these parameters should be default parameters that the connection plugins use in case they are not specified under$connection_plugin

My preference would be:

1. Eliminate port and move it to only $connection_options.
2. Add a nornir_vendor field

But I am also fine with what you proposed...though if we do that I would have a pretty strong preference not to name it operating_system as then it would frequently be a misnomer. In that case, I think something like platform or device_type would be much better term.

[dbarrosop]

Before I start this work. What do you think about this?

dev1.group_1:
    site: bma
    role: spine
    groups:
        - group_1
    connection_options:
        defaults:
            address: my_device.acme
            username: my_user
            password: my_password
            port: 22
            device_type: linux
        napalm:
            device_type: eos
            port: 443
            optional_args:
                blah: bleh
        netmiko:
            device_type: arista_eos
            port: 443
        remote_console:
            plugin: netmiko
            address: my_terminal_server.acme
            username: blah
            password: whatevs
            port: 7787
            device_type: serial_console

Basically all the nornir_* is gone and replaced by the nested object connection_options. Then, basically, we'd pass to the connection type the result of updating the "default" options with the connection specific ones. I added the remote_console one to illustrate how it could work once we implement named connections.

[dmfigol]

I like this idea! Especially:

        remote_console:
            plugin: netmiko
            address: my_terminal_server.acme

The only thing I don't really like is that defaults are under connection_options. If for some reason you need to access them directly, you need to go several levels deep :(

[ktbyers]

I generally like the idea except I would vote username, password, host, and device_type should be primary arguments. And no defaults for connection arguments i.e. the defaults are the defaults provided by the connection type or you could specify defaults at the Nornir group level.

For example:

dev1.group_1:
    host: foo.whatever.com
    username: my_user
    password: my_password
    device_type: linux
    site: bma
    role: spine
    groups:
        - group_1
    connection_options:
        napalm:
            device_type: eos
            port: 443
            optional_args:
                blah: bleh
        netmiko:
            device_type: arista_eos
            port: 443
        remote_console:
            plugin: netmiko
            address: my_terminal_server.acme
            username: blah
            password: whatevs
            port: 7787
            device_type: serial_console

host and device_type are more an aspect of the device than an aspect of the connection. username and password are a bit more in the middle, but most of the time you use the same credentials for different connection types.

[dbarrosop]

Ok, let's do that then.

[ktbyers]

Sounds good.

[dbarrosop]

Two more questions:

1. Should we keep the nornir_ prefix? I vote for killing it.
2. I think instead of host I'd like to use address. I think address is a bit less ambiguous than host.

I am going to be adding DeprecationWarnings to avoid breaking stuff. I think that unless maintaining something is very difficult we should try to keep it for at least a version.

Thoughts?

[ktbyers]

I also vote for killing the nornir_ prefix.

I pretty strongly dislike address. IMO host or hostname is more easy to understand and more of the standardized term to use in this context (i.e. this is the term one generally expects here).

Yes, I agree on the DeprecationWarnings.

[dmfigol]

I agree on killing nornir_ and I really like host (common in majority of libraries too)

[dbarrosop]

Ok, will keep host and kill nornir_.

Regarding the DeprecationWarnings, unfortunately it wasn't that easy. Mostly because we are converging ssh_port/network_api_port and os/network_operating_system :(

[dbarrosop]

Ok, just opened #224

[dmfigol]

@dbarrosop @ktbyers Not to pollute PR (since it does not contain what I am about to ask), but can we have a list of options for device_type (or platformor whatever name we select; I vote for platform though)? Ideally there should be a doc page or section with the list of all options which are going to be supported by built-in connection plugins My thoughts about the naming: I would like it to be as explicit as possible. For my perspective, netmiko naming convention (cisco_ios) is better than napalm (ios). But I would like it be even more explicit than netmiko, for example:

• cisco_ios
• cisco_iosxe
• cisco_iosxr
• arista_eos
• juniper_junos
• etc

The reasoning behind the separation between, let's say, cisco_ios and cisco_iosxe, is that just because netmiko will work the same for IOS and IOS XE, it does mean it is true for other plugins, for example NETCONF is supported only for IOS XE.

[dbarrosop]

Regarding device_type vs platform, I think device_type is a bit more generic but to avoid bikeshedding here and as you prefer platform and I prefer device_type I will let @ktbyers break the tie :)

Regarding the naming issue you were mentioning, my original idea was not to do that sort of abstraction or at least not yet. Note that the current device_type supported just correspond to the underlying library. I think if we want to abstract the connections so users don't need to know about them we need a better story than the current one. My suggestion would be to keep gathering feedback and experience before taking this leap.

[dmfigol]

I thought this is what you wanted to have - built-in plugins to implement common mapping between device_type|platform and whatever they need to be able to work. My original thoughts were not to have this mapping anywhere at all and force that be explicitly passed using connection_options. That said I would really love for us to take some decision here (and it is ok if we decide something now and later based on the user's feedback make changes), because it is stopping me from adding the plugins I am working on (netconf and vmware) to nornir. It is not a big change on the code side, but I don't want to update plugin docs every time.

[ktbyers]

@dmfigol FYI, Netmiko already has that:

 'cisco_asa',
 'cisco_ios',
 'cisco_nxos',
 'cisco_s300',
 'cisco_tp',
 'cisco_wlc',
 'cisco_xe',
 'cisco_xr',

It is just in practice...there haven't been any SSH differences between IOS and IOS-XE.

Given that you are creating additional plugins, I think we should eliminate the idea of trying to do mappings between plugins.

So I agree with @dbarrosop here that we should just provide the field (device_type/platform), but there is no general mapping between drivers. I think you (@dmfigol) brought up previously a lot of issues we were going to run into trying to do this and why it didn't make sense.

That said I would really love for us to take some decision here

FYI, I think the issues contained in this issue here have been decided. At least that is my sense of it.

[ktbyers]

I vote platform.

[dbarrosop]

@dmfigol what is it you feel it hasn't been decided yet? The attributes are pretty much settled and are already reflected in the PR (I just have to rename device_type to platform) and all the abstractions and "unifications" will come later when we gather a bit more experience.

[dbarrosop]

For reference, I updated PR is device_type is now platform.

[dmfigol]

@dbarrosop I am sorry, I guess I just can't understand. 1) How/Where is platform used? 2) What are "accepted values" for it now? 3) Should new plugins use it? If yes, how?

[ktbyers]

In Netmiko, I am going to feed platform directly into device_type. In NAPALM, I expect it to map directly to the get_network_driver driver selection.

In both cases, you will be able to override it with $netmiko_options or $napalm_options. In other words, if I have netmiko format for platform and I am using NAPALM with the same inventory, I can override it with $napalm_options.

Nornir has no notion of accepted values for platform. Any connection-plugin may or may not be able to use what you feed it. The plugins will have to determine what they accept.

I think new plugins should use it. In the case of NETCONF, I think it would feed directly into the NETCONF device_params["name"] that you referenced above. And then there should be a way to override it using $netconf_options.

Does that help?

[dmfigol]

Yes, thank you. This helps and I don't think I have any other questions on this. I agree with this approach, it seems reasonable!

[dbarrosop]

Fixed in #224