search

jedelman8 / cpal

The Common Programmable Abstraction Layer
22 stars 10 forks source link

What's the Plan? #30

Open sjiveson opened 10 years ago

sjiveson commented 10 years ago

All,

I'm keen to do more with the F5 module but I'm not sure it's worthwhile until we all have an idea of what's next and what the future feature roadmap might be. I'm sure we all have a lot of ideas but obviously some agreement and discussion is needed.

I'd like to add the F5 REST API as well as the SOAP one I've used so far but in more general, project-wide terms I've been thinking about;

-SSH support (using Paramiko/pexpect) -Prompting for username/password (per device or per group) -Ability to group devices (including from different vendors) -Standardising (where possible) on output formats across all the supported vendor modules (MAC address formatting is a prime example) -Ability to output to CSV file (relates to previous two) -Full configuration pull (if possible)

and so on, you get the idea.

Thoughts? I'm sure we're all busy but I'd rather not go off on a tangent.

yandy-r commented 10 years ago

I agree with most of your points, been thinking and implementing a few on my own. Just have not ported the code and submitted for the same reasons you state.

sjiveson commented 10 years ago

Hey Yandy, good stuff, thanks for responding :-)

:small_blue_diamond: SSH: Cool, I'll look into this in more detail :small_blue_diamond: Password prompts: did you mention this in an earlier comment - I'll trawl through :small_blue_diamond: Grouping: This would be putting devices into groups, you could then get facts about every device in the group rather than one at a time :small_blue_diamond: Standard Output: Cool, hopefully we can arrange a suitable forum :small_blue_diamond: CSV etc. That would be great, a db would also make a web front end far easier :small_blue_diamond: Config pull: Wasn't thinking that but that's a great idea. If we can pull the whole config there's that and a whole host of cool things we could do I'm sure

Not sure if Jason is still on vacation or what but would be good to cover this in real time somehow.

Forgive me playing with the Emoji icons.

jedelman8 commented 10 years ago

Guys, I am back from vacation and still love the enthusiasm. Getting your ideas on paper is awesome. I still have plenty myself.

For anyone else that has things they are thinking of, please use this thread to get them on "paper." I'll compile them and we can shoot for a real time hangout/webex next week. That should give me enough time to empty the queue too (after I finish catching up on work :) ).

Sound good?

Thanks, Jason

On Thu, Apr 24, 2014 at 7:22 AM, Steven Iveson notifications@github.comwrote:

Hey Yandy, good stuff, thanks for responding :-)

[image: :small_blue_diamond:] SSH: Cool, I'll look into this in more detail [image: :small_blue_diamond:] Password prompts: did you mention this in an earlier comment - I'll trawl through [image: :small_blue_diamond:] Grouping: This would be putting devices into groups, you could then get facts about every device in the group rather than one at a time [image: :small_blue_diamond:] Standard Output: Cool, hopefully we can arrange a suitable forum [image: :small_blue_diamond:] CSV etc. That would be great, a db would also make a web front end far easier [image: :small_blue_diamond:] Config pull: Wasn't thinking that but that's a great idea. If we can pull the whole config there's that and a whole host of cool things we could do I'm sure

Not sure if Jason is still on vacation or what but would be good to cover this in real time somehow.

Forgive me playing with the Emoji icons.

— Reply to this email directly or view it on GitHubhttps://github.com/jedelman8/cpal/issues/30#issuecomment-41269044 .

yandy-r commented 10 years ago

Sounds good.

Here's a couple more things, I added in a few things to allow for calling in options and vrf names to a specific command.

like:

$ ./main/netCalls.py -u admin -p admin -n dev1 -m arista -d veos-01 -f getRoutes --vrf mgmt -o 
# --------------------------------------------------------------------------------
# ARISTA DATASTRUCTURE REPRESENTATION
# --------------------------------------------------------------------------------

{ u'192.168.31.0/24': { 'admin_distance': [ '0'],
                        'metric': [ '0'],
                        'next_hop': [ u'connected'],
                        'next_hop_int': [ u'Management1'],
                        'protocol': u'C'}}

# --------------------------------------------------------------------------------

connected the run function was modified by calling specific device methods, so if manufacturer is arista, then it calls runEAPI, which in turns uses the getattr method to call the appropriate library method, with options or none. Because each manufacturer is different. There will be some generic ones as well.

I have not ported this to cpal, was just waiting on a few things to catch up, cause it could've gotten out of hand even with branches. but should be easy enough to port over as well.

$ ./main/netCalls.py -u admin -p admin -n dev1 -m arista -d veos-01 -f getOSPFNeighbors --vrf default
# --------------------------------------------------------------------------------
# ARISTA DATASTRUCTURE REPRESENTATION
# --------------------------------------------------------------------------------

{ u'1.0.0.1': { 'address': u'1.1.3.1',
                'dead_time': u'00:00:35',
                'interface': u'Ethernet3',
                'priority': u'1',
                'state': u'FULL',
                'vrf': u'default'},
  u'1.0.0.2': { 'address': u'1.2.3.2',
                'dead_time': u'00:00:38',
                'interface': u'Ethernet4',
                'priority': u'1',
                'state': u'FULL',
                'vrf': u'default'},
  u'1.0.0.4': { 'address': u'1.3.4.4',
                'dead_time': u'00:00:39',
                'interface': u'Vlan4094',
                'priority': u'0',
                'state': u'FULL',
                'vrf': u'default'}}

# --------------------------------------------------------------------------------

VRF default because I have no others running ospf, but should get the point of passing in that name..

Or

$ ./main/netCalls.py -u admin -p admin -n dev1 -m arista -d veos-01 -f getRoutes -o ospf
# --------------------------------------------------------------------------------
# ARISTA DATASTRUCTURE REPRESENTATION
# --------------------------------------------------------------------------------

{ u'1.0.0.1/32': { 'admin_distance': [ u'110'],
                   'metric': [ u'11'],
                   'next_hop': [ u'1.1.3.1'],
                   'next_hop_int': [ u'Ethernet3'],
                   'protocol': u'O'},
  u'1.0.0.2/32': { 'admin_distance': [ u'110'],
                   'metric': [ u'11'],
                   'next_hop': [ u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet4'],
                   'protocol': u'O'},
  u'1.0.0.5/32': { 'admin_distance': [ u'110'],
                   'metric': [ u'51'],
                   'next_hop': [ u'1.1.3.1',
                                 u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet3',
                                     u'Ethernet4'],
                   'protocol': u'O'},
  u'1.1.2.0/30': { 'admin_distance': [ u'110'],
                   'metric': [ u'50'],
                   'next_hop': [ u'1.1.3.1',
                                 u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet3',
                                     u'Ethernet4'],
                   'protocol': u'O'},
  u'1.1.2.4/30': { 'admin_distance': [ u'110'],
                   'metric': [ u'50'],
                   'next_hop': [ u'1.1.3.1',
                                 u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet3',
                                     u'Ethernet4'],
                   'protocol': u'O'},
  u'1.1.4.0/24': { 'admin_distance': [ u'110'],
                   'metric': [ u'20'],
                   'next_hop': [ u'1.3.4.4'],
                   'next_hop_int': [ u'Vlan4094'],
                   'protocol': u'O'},
  u'1.1.5.0/30': { 'admin_distance': [ u'110'],
                   'metric': [ u'50'],
                   'next_hop': [ u'1.1.3.1'],
                   'next_hop_int': [ u'Ethernet3'],
                   'protocol': u'O'},
  u'1.2.4.0/24': { 'admin_distance': [ u'110'],
                   'metric': [ u'50'],
                   'next_hop': [ u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet4'],
                   'protocol': u'O'},
  u'1.2.5.0/30': { 'admin_distance': [ u'110'],
                   'metric': [ u'50'],
                   'next_hop': [ u'1.2.3.2'],
                   'next_hop_int': [ u'Ethernet4'],
                   'protocol': u'O'}}

# --------------------------------------------------------------------------------

only getting those OSPF routes..

Or, running multiple commands at once

$ ./main/netCalls.py -u admin -p admin -n dev1 -m arista -d veos-01 -f 'getDetails getOSPFNeighbors'
# --------------------------------------------------------------------------------
# ARISTA DATASTRUCTURE REPRESENTATION
# --------------------------------------------------------------------------------

{ 0: { 'cpu_usage': u'7.8%',
       'free_memory': 16268,
       'host': 'veos-01',
       'hostname': u'veos-01',
       'name': 'dev1',
       'platform': u'vEOS',
       'serial_number': 'not_found',
       'system_mac': u'00:0c:29:63:d2:ba',
       'total_memory': 1000420,
       'uptime': u'6 days',
       'version': u'4.13.5F'},
  1: { u'1.0.0.1': { 'address': u'1.1.3.1',
                     'dead_time': u'00:00:35',
                     'interface': u'Ethernet3',
                     'priority': u'1',
                     'state': u'FULL',
                     'vrf': u'default'},
       u'1.0.0.2': { 'address': u'1.2.3.2',
                     'dead_time': u'00:00:39',
                     'interface': u'Ethernet4',
                     'priority': u'1',
                     'state': u'FULL',
                     'vrf': u'default'},
       u'1.0.0.4': { 'address': u'1.3.4.4',
                     'dead_time': u'00:00:38',
                     'interface': u'Vlan4094',
                     'priority': u'0',
                     'state': u'FULL',
                     'vrf': u'default'}}}

# --------------------------------------------------------------------------------

just a few things.

disregard the script name, it's not cpal code yet, as mentioned above.

sjiveson commented 10 years ago

Forgive the very long text. I thought of a few more things while trying to look at things from the perspective of different types of staff;

General;

Random thoughts: I do wonder if the below is really relevant to the coming 'new world order'? Is this only relevant to the past and smaller, older networks? Won't vendor management/virtualisation tools do all this? Am I using new tools to solve old world problems that will soon disappear anyway? I guess this is a wider discussion on what CPAL is really for. Would it help if we 'hooked' into vswitches and the like or is that just backward?

Operations Tasks;

Configuration Change Tasks;

Auditing

jedelman8 commented 10 years ago

All good stuff, guys. Just updated the Wiki to ensure we are all on the same page. Please take a read through.

What time of day works best for everyone to have a call?

Thanks, Jason

On Mon, Apr 28, 2014 at 5:37 AM, Steven Iveson notifications@github.comwrote:

Forgive the very long text. I thought of a few more things while trying to look at things from the perspective of different types of staff;

General;

  • It's a port or an interface - which is it, we must stick with one.
  • What the format of a MAC address? I vote for AB-CD-AB-CD-AC-CD as it's what the IEEE uses. Not my preferred format but the closest to a standard
  • Decision needed also on standard format of headings and text data, Title Case, always lowercase etc.
  • On a related note, how are we going to handle differences between vendor output where there are completely different fields available
  • Any thoughts on how to handle device capabilities, how to check a device runs BGP before allowing display of BGP only routes for instance
  • Time format standard - I vote 24hr
  • Date format standard - I vote DDMMYY, only the US use MMDD

Random thoughts: I do wonder if the below is really relevant to the coming 'new world order'? Is this only relevant to the past and smaller, older networks? Won't vendor management/virtualisation tools do all this? Am I using new tools to solve old world problems that will soon disappear anyway? I guess this is a wider discussion on what CPAL is really for. Would it help if we 'hooked' into vswitches and the like or is that just backward?

Operations Tasks;

  • Track a MAC, IP, interface, interface description based on input of any one of these items; the description might be too hard I guess
  • Pull in basic information based on one command, like a multilevel show tech;
  • -Basic health checks of CPU, memory usage, memory free, number of up/down interfaces, high sev log entries and uptime for instance
  • -Volatility check; basic algorithm based on number of route, arp, mac, hsrp, vrrp and stp changes over a number of time periods
  • -More...
  • RP checks of some kind (internal matching external...)
  • Bandwidth usage checks; per interface, just in, just out, both, chassis overall etc. with differing options for output (Mb, Gb, % etc.) - only ones that are up, only ones that have seen traffic in last day, week, hour, etc.
  • Interface counters, again differing output; errors, etc. - options to list only up interfaces, only down, only ones with errors, only ones that have seen traffic
  • Perhaps repeating myself but routes, hsrp, vrrp, stp changes (individually) in last day, week etc.
  • STP mode and status, global, per interface etc.
  • HSRP/VRRP status, recent changes etc.
  • Trunk status
  • VLANs status (plus check prior that VLANs are appropriate based on device type)
  • SVI status (plus check prior...)
  • Counters - up interfaces, down interfaces, vlans, l3 interfaces (up/down), hsrp groups, routes (static, dynamic, ospf, bgp, etc.) - again some way of checking it's appropriate/supported first

Configuration Change Tasks;

  • Config backups/save to file
  • Config diff tools (surely there's a module we can use?)
  • logger type functionality like send log xxx on Cisco (could automatically add time, username and other variables)
  • health checks to file and screen
  • other checks just send to file, don't display
  • ping sweep based on address and subnet of a l3 interface, one by one to overcome directed broadcast restrictions
  • list macs (also perhaps interfaces) related to hosts that respond to the ping sweep (some serious text/field matching skills required right)
  • testing tools (or perhaps more rightly, comparison tools) that save date for pre and post change comparison/diff of routes etc. etc. (output like 'hostname'-routes, hostname-routes-static and so on...
  • probably need a file structure that can easily be mapped to a db
  • find free ports (notconnect, no traffic counters etc.)

Auditing

  • just dump all the above data to csv files
  • again, full config
  • log retrieval (perhaps options for last day, week etc.)
  • basic security checks (telnet enabled etc.)
  • Basic vulnerability checks (against known bad versions of code etc.)

— Reply to this email directly or view it on GitHubhttps://github.com/jedelman8/cpal/issues/30#issuecomment-41539482 .

sjiveson commented 10 years ago

Now that's consolidation in action! :-)

I'm in the UK, currently UTC +1 so I'm free from UTC 17:00 but I'd prefer after 19:00 if possible. Alternatively I can do around midday depending on the day of the week, so UTC 11:00 - 14:00.

Cheers

Steve

darrelclute commented 10 years ago

As I mentioned in #28, we need to attribute a license, my personal preference would be a BSD or Apache, if GPL v2 but those are just suggestions. This will have importance for some contributors in addition to inclusion or reference from other projects.

Some additional thoughts, which may not be as important but more aesthetics.

With regard to a couple of @sjiveson suggestions:

I'm open to participating in a real time discussion as well. Not sure on availability though, so please don't take me into account in scheduling it.

jedelman8 commented 10 years ago

Agree with most of this! Check the wiki I started this morning on the github page.

On Apr 28, 2014, at 10:45 AM, Darrel Clute notifications@github.com wrote:

As I mentioned in #28, we need to attribute a license, my personal preference would be a BSD or Apache, if GPL v2 but those are just suggestions. This will have importance for some contributors in addition to inclusion or reference from other projects.

Some additional thoughts, which may not be as important but more aesthetics.

CPAL is very vague and does not properly denote the focus on network device manipulation and access. NDAL, network device abstraction layer, or pynetdev would be more appropriate. Again, this is likely just an aesthetic issue and I'm personally not tied to any of my suggestions.

@jedelman8 stated the original purpose to be a library to be leveraged by others to provide a consistent access method to various vendors equipment. If this is still the case, which I'd actually prefer, then I'd suggest the following:

All code in this repository be just library with anything that is directly executable as examples. If we want to build leveraging utilities, database storage, a web front end, or anything like this it should be a separate project built on top of this one. Provide scaffolding for non-library functions in this project where it makes sense, but specific details should be provided by a referencing project, not enforced upon it based on decisions we make here. This will make it easier to include or gain acceptance in other projects, such as Salt or Ansible. With regard to a couple of @sjiveson suggestions:

Beyond differences between vendors, even certain classes of devices from the same vendor, such as load balancer and firewall, will have very unique data types or terminologies compared to others. We should do our best to effectively build a device type hierarchy so that common things are not duplicated but not include device type specific terminology where it does not make sense. This also means that this will need to abstract out differences between vendors, but that seemed to be an eventual goal anyway.

Time and date format, agree 24 hour should be default. Options to change output, based on keyword arguments for example should be provided though. Would actually suggest a format that provides the most detail, such as what you see in most syslog output, therefore a 4 digit year, 24 hour time and down to second resolution, minimum. This has the advantage of being able to translate it to the output that is desired without any lost data.

I'm open to participating in a real time discussion as well. Not sure on availability though, so please don't take me into account in scheduling it.

— Reply to this email directly or view it on GitHub.

sjiveson commented 10 years ago

Some interesting and thought provoking comments @darrelclute, thanks, I couldn't seem to get out of the weeds!

yandy-r commented 10 years ago

very well put @darrelclute

As far as a call or hangout, my schedule's relatively open, would just need a day or so to plan out. I work from home allot, so flexibility's good.

Thanks

yandy-r commented 10 years ago

FYI, this may be helpful on the wiki to keep track of things Task list for markdown