Refactoring the render_* methods

[GoogleCodeExporter]

In working on Volshell, I've hit some issues regarding the current usage of the 
render_* methods.

Plugins that currently present their data in a tabulated format, do so by 
taking advantage of the table_header and table_row methods. These methods lend 
themselves to being hooked and so allow tools (e.g. Volshell) to extract the 
end user data before it is printed. In effect, table_header and table_row 
implement a lightweight pretty printer.

I'm proposing that we move away from using outfd (i.e. socket based output) to 
using a general (functional?) pretty printing mechanism with callback support. 

IPython has an excellent looking pretty printer (i.e. pretty [1]) that caters 
for callback functions. In reading [2], we can see plenty of potential in using 
pretty printers for displaying/rendering data structure content.

As such a move allows the data to be hooked, we should then be able to restore 
the functionality offered by methods such as render_csv, render_dot, etc. by 
placing this type of functionality in additional template-like methods.

So, summing up, I'm proposing:
  1. introducing a render method (this would be mostly a renaming of render_text to render!)
  2. replacing writes to outfd with pretty printer interactions (c.f. table_header and table_row usage) - see note [*]
  3. using pretty printer callbacks (c.f. hooking) to generate Python data to be fed into template_csv, template_sql, template_dot, etc. methods - see note [**].

If these ideas are acceptable, let me know and I'll work on them further.

NOTE:
 [*] in the first instance, I'd suggest using IPython's lib.pretty to explore the potential here.
 [**] for the most part, these template methods would be defined generically for general Python and Volatility data structures (thus allowing end users to shim Volatility into their product). Plugins that required specific output formats (e.g. iehistory, timeline, etc.) should still be able to do this by overriding the framework supplied defaults.

REFERENCES:
 [1] IPython Module: lib.pretty (accessed 26/Mar/2013):
       http://ipython.org/ipython-doc/dev/api/generated/IPython.lib.pretty.html
 [2] GDB/Python Pretty printing parts 1 and 2 (accessed 26/Mar/2013):
       http://tromey.com/blog/?p=524

Original issue reported on code.google.com by carl.pulley on 26 Mar 2013 at 3:11

• Merged into: #311

[GoogleCodeExporter]

Hiya Carl!

Yes, the outfd format has been a point of contention since it was first decided 
upon.  I don't think that any changes will be made to the current system since 
the table_* methods successfully support most of the plugins and basically we 
can get by for now with those.

There have been several ideas thrown around for Volatility 3 and we need to get 
the core of that stabilized before we get toward the outer interfaces (the ones 
plugin developers will need to work with most).  Currently the idea is to have 
a single structure, effectively a tree with columns, which can then easily be 
turned into a table by simply not having any children.  The plugin's main body 
would produce the tree and then generic printers (which could be overridden per 
plugin) would then render the standard tree format.  Dealing with particularly 
unusual output will still need thinking about, perhaps some way of indicating 
the data isn't normal and therefore falling back to rendering to console?  
Anyway, ideas around that area would be much appreciated!

So I'd suggest not putting much effort into coding this until the core team 
manages to present a solution they can all agree on, and then during the 
testing phase for that suggest improvements or make additions, etc.  Hopefully 
it won't be too far away, but we can't put a date on it yet I'm afraid...

Original comment by mike.auty@gmail.com on 26 Mar 2013 at 9:05

[GoogleCodeExporter]

Original comment by mike.auty@gmail.com on 26 Mar 2013 at 9:09

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Looking over a few of the plugins that use render_dot, it's clear that the 
output generated by these methods is often based on data structures that differ 
to the underlying render_text ones (e.g. see the malware.svcscan plugin). 

So, if the existing render_* methods were to be based on a pretty printer 
(rather than using outfd writes), we'd probably want to spit them into a 
render_X and template_X methods? For example, with malware.svcscan, render_dot 
would have to be split into a render_dot (based on a pretty printer) and 
template_dot (to produce the actual dot output).

Original comment by carl.pulley on 26 Mar 2013 at 9:16

[GoogleCodeExporter]

For information purposes, I've attached some prototype plugin code that 
outlines the idea that I originally had in mind.

I've focused on using IPython's lib.pretty to (partially) reimplement the 
render_text and render_dot methods for malware.threads and malware.svcscan. To 
see how the code might be hooked for scraping Python data, I've also 
implemented a render_python method.

Hopefully, the code will be of some use?

Original comment by carl.pulley on 28 Mar 2013 at 10:28

Attachments:

• ipython.py

[GoogleCodeExporter]

I couldn't help feeling that the ipython.py code felt unsatisfactory, so here's 
another coding experiment.

This time I've embraced a Python templating library (Mako) and then 
experimented with various html/xml -> XXX format translations.

If nothing else, it shows the other end of the scale here.

Original comment by carl.pulley on 5 Apr 2013 at 9:30

Attachments:

• mako_soup.py

[GoogleCodeExporter]

Original comment by michael.hale@gmail.com on 7 Mar 2014 at 9:22

• Changed state: Duplicate