Customizable tabular reporting system

[mfripp]

It would be nice to have a standardized system for creating tabular output files, where users can call a helper function to register any expressions that they want reported, along with the indexes and (optional) filename to use for them. Then a standard reporting module would create a single output file for each specified filename and/or indexing set, and would automatically create a column for each expression that's been assigned to that file (there would probably also be some standard expressions registered automatically). Tables like this are very handy for reviewing results in Excel or Pandas.

Tables like this are already created by switch_mod.hawaii.util.write_table(), but it would be nice to generalize this approach as described above. This would reduce the amount of nesting that is often needed in calls to write_table() and also make it so that all the code related to one output file doesn't have to be in one place.

[bmaluenda]

This is kind of a late reply, but only just recently have I been venturing into exporting functions.

I want to check if I got your comment right, Matthias: you would like an output which had an index column and several columns with different expressions, all in the same file?

Would you say this example represents that?

'timepoints'    'total_load_MW'    'total_gas_consumed_MBTU'    'total_gas_consumed_CCGT_projs_MBTU' 1    10.5    15    9.2 2    14.3    18    11.6 ...

I wrote a custom tabular reporting system that also prints pdfs with the values plotted against the indices. Though, the tables I'm using consist on an initial column with the index (usually timepoints or periods) and several columns with either dispatch or investment decisions (such as dispatch per generation technology or available transmission capacity, in example).

These functions could potentially be used for what you were proposing, with some tweaks. As soon as I find some time, I'll push them.

[josiahjohnston]

@bmaluenda Did you ever push those changes to either a main branch or experimental branch?

The last time I looked into this in an old version of Pyomo, it wasn't always possible to auto-detect whether components shared the same index set. The current version of Pyomo has better support, you can get the index name like so:

> print(m.gen_availability.index_set().name)
'GENERATION_PROJECTS'

For components that use ad-hoc constructions like TxCapacityNameplate & TxCapacityNameplateAvailable that are both based on the cross product of transmission lines and periods, ...index_set().name yields names like TxCapacityNameplate_index.

mod.TxCapacityNameplate = Expression(
        mod.TRANSMISSION_LINES, mod.PERIODS,
        rule=...

While generalized export code like this seems nice in principle, I'm increasingly thinking it is a poor substitute for the slower process of manually writing good export code that tailors output for human needs. I pushed a few examples of that into the 2.0.6 branch. I'm not opposed to having code like this, but I wouldn't personally prioritize it over hand-crafted export code.

[bmaluenda]

From what I remember, the code I wrote in my local branch was quite simple. It basically took user-specified indexes and expressions and printed out a .tab file with the values. It the index-expression pairing was wrong, the error would simply be ignored and the output file, not printed.

Maybe a nice solution could be to create such a simple function and provide clear documentation so users can use it to export whatever they want.

A side question that touches this point: Do any of you know if Pyomo offers the capability of dumping a single solution file that can later be read by Pyomo itself to load a model with variable values? Maybe an XML or another database format file. Plexos offers that feature and is super helpful for when you want to interactively explore the solution in-console (I usually load solutions in IPython and then interactively query and explore with Pandas), as well as recovering outputs from past models (for e.g., imagine you run a project today and realize -a year from now- that you need certain variable's values that you didn't print out for a new analysis).