Closed mfripp closed 1 year ago
After looking a little more deeply, I realized that (I think) the TypeError
code is meant to support the case where a rule doesn't accept the index as an argument, and instead returns a dict-like object. The Param
code uses this to support exactly the type of single-pass scanning that I described above (just return the dictionary instead of caching it and subsequently returning elements). However, the current approach has a few problems:
Param
doesn't accept an index, but raises a TypeError
before it can return the initialization dict, the user ends up getting a TypeError
that their rule is being passed an index argument that it doesn't accept. No TypeError
is raised at the location of the real error in their code.So my recommendation would be something like this:
apply_indexed_rule
should use rule.__code__.co_argcount
to determine in advance how many arguments the rule requires (similar to pyomo.core.base.sets
). If the rule needs an index, apply_indexed_rule
should provide it. If the rule doesn't need an index, apply_indexed_rule
should not provide it. (Alternatively, apply_indexed_rule
should closely analyze any TypeError
s it receives, to make sure they really were the result of passing an unwanted index in this particular line of code; if so, try again without the index; if not, raise the exception.)apply_indexed_rule
should make sure that the return value from the rule is dict-like if an index was available but not accepted by the rule. In other cases, apply_indexed_rule
should make sure that the return value from the rule is not dict-like. Ideally, the 'dict-like' status should then be conveyed to the caller in some way, but that may be too much to ask.Param
, other components that can be initialized with a dictionary should also accept a dictionary returned by the initialization rule. (This should at least be available for set arrays (IndexedSet
), which I often construct by scanning other sets or parameters.)These changes would prevent masking errors in users' rules or running the rules twice, and would also streamline creation of components that need to be aggregated from other components at run time. (In general, returning a dictionary with all the data is much faster than scanning the underlying data once for each index term.)
I could put together a pull request that would implement these suggestions, if that would be helpful.
Pull requests are always welcome
I just tested this example on the latest Pyomo release (6.5.0) and it looks like this was fixed at some point so closing this.
The
pyomo.core.base.misc.apply_indexed_rule()
function includes some code which is wrapped intry/catch
clauses to catchTypeError
exceptions. If my own rule code accidentally generates aTypeError
, this code can potentially mask it. This function includes fallback code that will retry the rule without the wrappers, to re-generate the exception. However, in some of my rules, the first call has side effects which short-circuit execution during the second call, so that theTypeError
in my code is silently masked. This made it very hard to debug my code.I think the best solution would be to streamline the
apply_indexed_rule()
so that it doesn't wrap my rule in atry/catch
clause. However, I don't know which code is supposed to be wrapped by theTypeError
catching code, so I can't propose a specific fix.This is the case which causes problems for me: I often want to initialize indexed parameters by aggregating data from some other part of the model. It is possible to do this piecemeal from the initialization rule. However it is often more efficient to create the aggregated data during a single pass through the underlying data, then cache those results and extract aggregated data as needed to initialize the aggregate parameter. I can do this in a standard initialization rule as follows (1) check whether a cache has been created, (2) create and fill the cache if it hasn't been created, (3) pop the requested result from the cache. The problem is this: If step (2) generates a
TypeError
, theapply_indexed_rule()
function will catch it, then call my function again. However on the second pass, the check in step (1) will succeed, step (2) will be skipped, and I'll never get an error message connected to my bad code.Here's an example of code that has this problem.
This code should report a
TypeError
on theval = ...
line, but instead it falls through to the end, where it reports that the cache is empty. This is fairly tough to debug, since the offending line is never mentioned in a traceback. (I have to do a bisection search through my code, inserting trace points until I find the first line that is never executed.)As I mentioned, I think that tidying up the
try/catch
code inpyomo.core.base.misc.apply_indexed_rule()
is the best way to avoid this problem. Barring that, users should be warned that their rules may be called twice if they contain an error, so the rules should not have any side effects if they fail due to an error (e.g., thecache_dict
should be kept as a local variable until it is completely created, then tacked onto the model object).