Adding component references

[andrewlee94]

In IDAES, we frequently have situations where we would like to reference a Pyomo component (most frequently a Param or Set) from one Block in another Block within our models. The most common case for this is where we have a Param that is used in multiple sub-Blocks within a model, but would like have a single, centralised Param object (so that the parameter only needs to be changed in a single location, rather than in every instance).

At the moment I do this using weakrefs to create the necessary links, which requires me to dereference the weakref whenever I use it. However, this results in some objects in my constraints having () after them and some without, which will make it difficult for new users to understand my code.

It was suggested that a component reference feature could be built into Pyomo to handle this transparently, without the need for the ().

[ghackebeil]

A slightly ugly way to accomplish this from a user standpoint is to bypass the overloaded setattr block method by assigning directly to dict. E.g.,

b1.myparam = Param(...)

b2.dict['myparam'] = b1.myparam

Now one can use b2.myparam like it is any other attributes on b2, it just won't be categorized on that block (and will keep its original parent block). This functionality could be migrated to a new method or added to any existing one using a keyword.

An alternate approach would be to create some kind of alias wrapper object, but that seems like overkill.

[jsiirola]

@ghackebeil: One correction: the Python docs recommend using super or object.__setattr__() and not directly assigning to the __dict__ dictionary.

There is also a question as to whether models that use hard references to refer to other Components will be properly collected (without forcing a run of the garbage collector).

Finally, @andrewlee94: what is the intent of having Components appear in more than one block? For example, when using component_objects/component_data_objects, do you want the "reference" components to appear in the resulting generated list?

[blnicho]

Just to chime in on some of Andrew's use cases, consider a flowsheet(ConcreteModel) containing multiple unit models (Blocks) where each block uses the same parameter (e.g. used to calculate a property of a component). The point is that we'd like to declare the parameter once at the flowsheet level and use references to that parameter within the unit models themselves. Similarly, if you're incorporating time or spatial dynamics you want to declare a single ContinuousSet at the flowsheet level that is then used in both unit models to ensure consistency in the discretizations of the units.

I think we want each unit model to have it's own pointer to these shared components so that we can write the unit models to be completely independent of the flowsheet they are being used in. i.e. we aren't making assumptions about how many units appear in a flowsheet or the hierarchy (flowsheets containing flowsheets containing units ...).

To answer your question, my initial intuition is if you look at the component objects for the unit blocks I would expect the "reference" components to appear in the list.

[andrewlee94]

Bethany pretty much hit the nail on the head, and explained things much better than I probably would have.

I have just been debating with some of our postdocs if and how the "reference" components should appear in the list of components in the unit blocks, and our feeling was that it would be best if the "reference" components appeared, but be clearly indicated as such. This way, a user (with limited knowledge of the inner workings) can see that the "reference" component is being used, but cannot directly change the parameter itself (avoiding the possibility of the user mistaking a "reference" component for a component of the local block). We definitely felt that having a "reference" component simply appear as if it was a component of the local block was a bad idea.

[michaelbynum]

If Block "A" uses parameter "P", which is not defined on Block "A", then why not have an argument for "P" in the function that builds Block "A"?

[qtothec]

@michaelbynum I think that it is an effort at deduplication. If there is a global set of chemical components that is used within multiple process units (Blocks), then it can feel silly having a replicate within each block.

@ghackebeil @jsiirola Is object.__setattr__() still the recommended approach to doing this, and are there any potentially troublesome side effects (e.g. when cloning)?

[jsiirola]

@qtothec: yes, I think that object.__setattr__() is the best route at the moment. I had been hoping that weakref.proxy would be a quick-and-dirty solution that would allow Blocks to hold "weak" references to components stored elsewhere, but (un)fortunately:

import weakref
m = ConcreteModel()
m.x = Var()
a = weakref.proxy(m.x)
type(a) # == <type 'weakcallableproxy'>
isinstance(a, Var) # == True!

Adding a trap for proxy objects is a straightforward thing, although it will potentially impact construction time so some design/testing is warranted. My quick prototypes highlighted some other weirdness around garbage collection for scalar objects (which I suppose I should have known was occurring).

As to your other question re cloning, some quick tests indicate that cloning blocks with object.__setattr__ "hacked" proxy references to other parts of the model works as I would expect it (the referent is cloned based on the Block Scope of the actual component and not the referent).

[qtothec]

@jsiirola Can I use this idiom for parent_unit references as well?

[jsiirola]

I don't see an obvious reason not to.

[whart222]

One potential issue with using setattr logic is that these components would be picked up repeated while walking the model hierarchy. I think it might be useful to have an aliasing mechanism supported on blocks that stores aliased components differently, but transparently enables getattr for those components.

This requires a clear sense of component ownership, but I think we already have that...

[jsiirola]

@whart222: that is the point of using object.__setattr__(block, 'name', component): it bypasses Block's normal processing for components. The "proxy reference" will appear in the Block's __dict__, but not in the list of owned components. As such, it won't be picked up by the PseudoMap or our normal iterators / walkers (like component_data_obejects).

[andrewlee94]

I have implemented the object.setattr(block, 'name', component) approach, and it works well for what I've done so far.

The only catch, which is a fairly obvious one, is that if the Block which owns the object in question is deactivated, any Block with a reference to the object breaks. However, that is more an indication of poorly thought out models than a problem with the approach (i.e. you shouldn't be using a reference in that situation).

[jsiirola]

@andrewlee94: FWIW, variables on deactivated blocks is a bigger problem with the Pyomo writers, and one that I hope to fix soon (it is also pretty severely impacting GDP).