google / textfsm

Python module for parsing semi-structured text into python tables.
Apache License 2.0
1.11k stars 171 forks source link

Feature request to support iterative parsing by reading from file objects #85

Open Ragsboss opened 4 years ago

Ragsboss commented 4 years ago

The use case here is to parse an output from a command that produces 1000s of lines of output. Instead of storing all that output into a string and passing it to the parser, we'd like to send a file object and let the parser call a provided callback after completing each record. This way, the whole output need not be in memory at once. For large outputs this makes a considerable difference in memory footprint. You can see similar support in tools like lxml - https://lxml.de/api/lxml.etree.iterparse-class.html.

Typically these commands are executed remotely using some mechanism such as paramiko which offers a way to read the output from a file object (wrapping a file descriptor). So this file object can directly be passed to textfsm to perform this iterative parsing in a streaming fashion.