Documentation is Inaccurate

[james-powis]

Per documentation located here for Output Macros, the return should be a list of list of dictionaries, instead the result is a list of list of list of dictionaries.

Example:

Python 3.7.4 (default, Sep  7 2019, 18:27:02) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> 
>>> from ttp import ttp
>>> 
>>> template = '''
... <input load="text">
... interface Vlan778
...  ip address 2002::fd37::91/124
... !
... interface Loopback991
...  ip address 192.168.0.1/32
... !
... </input>
... 
... <macro>
... def check_svi(data):
...     # data is a list of lists:
...     # [[{'interface': 'Vlan778', 'ip': '2002::fd37::91', 'mask': '124'},
...     #   {'interface': 'Loopback991', 'ip': '192.168.0.1', 'mask': '32'}]]
...     for item in data[0]:
...         if "Vlan" in item["interface"]:
...             item["is_svi"] = True
...         else:
...             item["is_svi"] = False
... </macro>
... 
... <group>
... interface {{ interface }}
...  ip address {{ ip }}/{{ mask }}
... </group>
... 
... <output macro="check_svi"/>
... '''
>>> 
>>> parser = ttp(template=template)
>>> parser.parse()
>>> parser.result()
[[[{'ip': '2002::fd37::91', 'mask': '124', 'interface': 'Vlan778', 'is_svi': True}, {'ip': '192.168.0.1', 'mask': '32', 'interface': 'Loopback991', 'is_svi': False}]]]

[james-powis]

Please note that in almost every other documented example, the documented return is actually stuffed inside of an extra list.

[dmulyalin]

Thank you for raising that issue, fixed in latest commit.

Documentation in general need some rework in terms of making examples precisely match to actual scripts output, probably will be looking at it sometime in the future.

[james-powis]

@dmulyalin This project has promise in ways its competitors in the same category have failed to match.

I would be absolutely happy to assist in helping address anything on your todo list, are you interested in having another active contributor?

[dmulyalin]

Thank you for your feedback. It is really encouraging to see that TTP is useful to people.

What you think would be good to have in TTP in terms of functionality, anything its lacking that would be crucial or great to have?

Are there any specific problems you used TTP for you think can be solved somehow better?

What you did not like about TTP and think could be improved?

TTP has very pluggable architecture, in fact to include additional function all that required is to put .py file in certain directory within TTP module. If you think there are some match/group/output/input functions could benefit to community - feel free to share your ideas or code. Maybe some macro you used very often, we can convert it to built-in function that way.

In simplest case, all revolves around - here is the problem and here is (awesome) way to solve it. If you have such a problems/solutions - feel free to share )).

[james-powis]

So the use-cases we currently using TTP for are automated testing of $network_devices from many different vendors. Typically we manually certify new next gen platforms long before they are generally available to the general public, and with the price tags of some of these devices, it is HIGHLY unlikely that tools such as napalm will support them for quite some time (if ever). As such most automations we write are written to hopefully function for future code revs and platforms because of the way we screen scrape, and TTP helps us by performing the following:

• Nonprinting characters are treated the same
• After taking front of line literally, whitespace is reduced (with the exception of table processing)
• Grouping helps us parse even better without lots of fidgety buggy code.

It goes without saying that Cisco is just about the worst offender when it comes to strange nonprinting stuff:

• Funky nonprinting characters at start of line for specific inconsistent firmwares and smu's randomly across different platforms
• Tables with inconsistent (frequently changing) column widths
• Table overflow and/or wrapping

With all that being said my thoughts on "nice to haves, etc", (some of this I have not tried to solve for but believe it might be a blind spot):

Solve for table {{ headers }} which may wrap to a new line (ie. on Junos show ntp associations a ipv6 address can extend beyond the first column width which causes the second column to be moved down a line, but the indention is correct. I solved for this by not using {{ headers }}

                <group name="associations">
                {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }} {{ method }} {{ stratum | to_int }} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }}
                </group>
                <group name="associations">
                {{ active | re("[ *]") | macro("check_active") }}{{ ip | IP | IPV6 }}
                                  {{ method }} {{ stratum | to_int}} {{ peer_type }} {{ when }} {{ poll }} {{ reach }} {{ delay }} {{ offset }} {{ jitter }}
                </group>

                <macro>
                def check_active(data):
                    if data == '*':
                        return data, { 'type': 'our_master' }
                    else:
                        return data, { 'type': 'candidate' }
                    return data
                </macro>

Solve for variable width headers and column overflow / wrapping. I would imagine this would work something like this (pseudo code)

input = '''
Foo    Bar  Baz
a      b    c

Foo      Bar    Baz
a        b      c

Foo      Baz
a        c
'''
template = '''
<group>
{{ foo | labeled_as('Foo') }} {{ bar | labeled_as('Bar') | optional}} {{ baz | labeled_as('Baz') }} {{ _header_ }}
</group> 
'''

Logic:

1. Build regex for all headers vars (optional grouping for optional vars)
2. Determine order and width of column using 'labeled_as'
3. for each line if full match parse and store

Implement convenience parsers (prewritten parsers others can use or reference) Provide mechism to clean input prior to parsing, (something like {{ _parsestart }} so that everything in input until the match is disregarded. This is an issue when using table grouping when there is extra data from a show command prior to the table. I have solved for this with show blah | b Chassis ID

[james-powis]

Oh one other thing... we prefer to have multiline strings in our code properly style indented. For that reason we use textwrap.dedent to remove the FOL indent before sending to TTP. It might be a nice to have to implement a "Try literally, if not matched try dedented" into ttp so that it is not so dependent on "No multiline indent".