Set multiple parameters from a single value

[kalekundert]

One thing that BYOC doesn't naturally handle well are config values that specify multiple parameters. For example:

class Time:
    hour = byoc.param()
    minute = byoc.param()

# config.toml
time = '12:30'

Here, we need to do some string parsing on the config value to separate the hour and the minute. The result of this parsing, though, needs to be saved to two different parameters. There are three ways to handle this currently:

• Just don't have two separate parameters. This is what I most often do, but I don't like that the structure of the config dictates the structure of the code. Often it makes more sense for the parameters to be separate, and this issue is the only reason they're combined.

• Do the parsing twice. This works, but is inefficient.

• Save the parsed value to a private parameter, then derive the public parameters from the private one. This also works, but requires that a temporary value be permanently stored in the object. In other words, it's not memory efficient. It's also pretty verbose.

Today I was thinking about a better way to handle this. My idea is to have cast functions that are shared between multiple parameters. Whenever the function is evaluated for one parameter, all of the others (if stale) are updated as well. Here's what it might look like:

# As a cast function:
class Time:
    hour = byoc.param(
            Key(DocoptConfig, '--time', cast=shared(parse_time, 0)),
    )
    minute = byoc.param(
            Key(DocoptConfig, '--time', cast=shared(parse_time, 1)),
    )

# As a getter:
class Time:
    hour = byoc.param(
            SharedKey(DocoptConfig, '--time', cast=parse_time, index=0),
    )
    minute = byoc.param(
            SharedKey(DocoptConfig, '--time', cast=parse_time, index=1),
    )

def parse_time(time_str):
    return time_str.split(':')

Some thoughts:

• I'm not really sure if this should be a getter or a cast function.
  • Cast function:
  • It's a bit weird that the Config and key need to be specified twice. It wouldn't make sense to share values between different configs/keys: the whole idea is that the function will be called once, which means the inputs must be the same for each parameter.
  • Getter:
  • It would surely be a subclass of Key. But it's conceivable that I'd want to this sort of feature with Method or Func. Maybe I'd end up doing some sort of mixin thing, although I'd just start with Key.
  • You could use this without a cast function, although there's no real reason to.
  • I'm thinking that a getter would be the way to go.
• In this case, I'm imagining that the two parameters would be linked by the identity of the parse_time function. The 0 and 1 arguments specify which element of the tuple each parameter will get. These arguments could also be callables, e.g. attrgetter('x').
• Implementing this feature would require the parameters to know about each other, which is something I've avoided so far.
  • I might avoid the need to track every parameter, by having a class-level registry of shared keys:
  • Each getter could calculate a key from its config, key, and cast function, then insert itself into a class-level registry with this key. All linked getters would end up together.
  • This would have to happen in param.__set_name__(): it's the only place that (i) is guaranteed to be called for evey param and (ii) can access the class.
  • I'd have to manually reconcile registries from different subclasses, the complexity of which is one of the main reasons I want to avoid tracking parameters in the first place.
  • I would have to store the registry for the duration of the class. I kinda feel that I do to much caching as it is, and this would be another step in that direction.
  • Perhaps users would expect that they could get a list of all the parameters for an object/class anyways, so this wouldn't be a terrible feature to implement.
  • I do feel like BYOC does too much caching, leading to heavy objects that carry around more data that they'll ever likely use. This would be another step in that direction. I'm becoming more partial to the philosophy that recalculating parameters is expected to be a heavy operation, so it's not necessary to cache every possible thing.
• What would the internal algorithm be?
  • When binding the getters, work out which shared keys go together based on their config class, key object, and cast function. Each bound getter will store a list of all its partners.
  • The bound getter API currently has two methods: iter_values() and cast_value(). I'd want to modify the latter to update all the linked params, in such a way that the cast function is not called again. Maybe these bound getters could have a with_value() context manager that temporarily overrides the value to use.
  • Have to think about how this would work for configs that generate multiple values.
  • There's no way to iterate through the getters simultaneously, because the parameters could have different pickers.
  • I could cache the inputs and share the cache between the bound getters. The nice thing is that would guard against getting the wrong answer in weird cases, e.g. non-pure cast functions. How would the cache be reset, though? I might need to add a cleanup() method to the bound getters that gets called after the value is generated.
  • I could also try to replay the values: each time a value is iterated, it is added to a list. When a shared config needs a value beyond what the list contains, it calculates it and adds it to the list. Unlike a cache, this wouldn't be robust to misbehaving cast functions. But a big benefit is that it doesn't require hashable inputs. This list would have to be reset in the same way as the cache, as discussed above.
  • Unless I add the aforementioned cleanup() method, I'd have to refresh all the linked params before yielding the new value, because once ValuesIter yields, it's not guaranteed to be called again. This leads to a bit of weirdness in that all the linked parameters will get values before the original parameter, but I suppose that's ok. It is possible for parameters to depend on other parameters, so the order in which things are loaded could matter, but practically I don't think it would be a big deal.
• I can't quite decide if this is worth doing. It would be complex, but not insane. I do seem to run into this issue with some frequency, but it hasn't been a show-stopper. I'm not sure...

[kalekundert]

Recently I've been thinking that it was a mistake to cache bound getters, but this is a case where I need them to be cached (and the cache isn't working right). Maybe this is a good opportunity to change the way I cache bound getters:

• Bound getters need to be recalculated on loads, because of implicit keys.

• My idea is to clear the bound getter cache every time a new value is calculated:

  • This does allow the possibility that the cache could get stale, but only if bound getters are loaded without calculating a new value. There's no reason this should ever happen. It just means that callers of param._load_bound_getters() need to be careful.

  • This also means that the object won't store the bound getters in perpetuity, which is nice.

  • This is only inefficient for dynamic params, but most params aren't dynamic, and even for those that are, it's not a huge expense. I think it's worth the reduced memory footprint.

• Alternatively, I could not cache the getters at all, and instead make them an optional argument to the param._update_value() method. The default path would calculate them on the fly, while special paths like SharedKey could precalculate them...

  Wait, no. If two shared keys need to operate on the same param, the second key needs to build on the bound getters prepared by the first. This requires having a short-term cache.