There are two kind of parameters. They both can return a value and have dependents and a name.
Dependent (CompositeParameter)
The dependents of this parameter are Parameters. They can't have limits and aren't changeable but act as a "wrapped" tensor.
Independent (Parameter)
These are trainable and assignable. There dependents are {}, as they are independent. They get added to the tf-collection "zfit_independent".
Specs
[x] be able to create compound parameters
API
add and multiply return tensors.
Notes:
Postponed work, wait until TF variables stabilized
change shape
Changing from a single value to n values (usable for integration/MC error):
tf.assign(var_to_change, new_value, validate_shape=False)
# Probably, validate_shape also has to be falsified on instance creation as
# .shape probably is not updated correctly. Seems to work now though.
update var
use load
dependent vars
variables depending on each other must be carefully treated with their initialization.
Our use-case for the inheritance of tf.Variable: we build a fitting library using probability density functions and minimize a negative log-likelihood.
Variables for us are therefore not just some t unable parameter out of millions but we use up to maximal 100 (often also only 5-10) and they have a lot of "meaning".
This goes as follows: create a loss function, minimize it, (use it in another loss function, minimize again,...). We don't need, for example, distributed training (one GPU per minimization is sufficient).
What we need that is not in tf.Variable:
setting limits easily (yes, constraints is there, but we reach for a slightly more "user-friendly" (for using it as fit parameters) API)
add errors (uncertainities) to the parameters as attributes (as they are an essential part of our fitting parameters).
"blind" them: change their value unpredictably if a certain flag is set (sounds odd, anyway needs some addition to the variable)
There are a few more things (like making easy compositions).
@alextp what do you think? I think ResourceVariable could fit here well.
Can you do this with a wrapper class around Variable? To make your wrapper object usable as a tf tensor you just need to call tf.register_tensor_conversion_function on it, and you can define your own assign methods and whatnot.
I think this should work since I assume you're not using tf optimizers as those don't fit this use case well.
@alextp That's an interesting thought, so I took a look at it. I realised though that we still need to be very close to what a tf.Variable does, (like all the overloaded operators etc) and don't require to do too much more (basically, a fancy constraint function could handle all that). So say multiplying two Variables should work out of the box.
And with the optimizers: we don't need them currently as they are first order algorithms and we can afford higher order (like BFGS), but we definitely need optimizers and would like to stay compatible (hoping for example for tfp.optimizers to become tf.Optimizer compatible).
Do you think it is unreasonable to inherit from ResourceVariable? Or are there any dangers with that?
Besides, is there somewhere good documentation on the Variables mechanics, as I have searched a lot but except of the official docs (which don't say anything about that lower-level handling), I did not find anything.
We have a plan to open up a subclass-variable API which will not rely on _underscored private stuff. For now though implementing convert_to_tensor and the operator overloads yourself will get you ~most of what you described.
We tried again with that, but unfortunately there are too many things to get stuck in like: tfp.Distributions use a common_dtype function to determine the dtype of the objects and it does not check for a dtype attribute directly but only if it's the instance of a known type (which our Variable of course is not) -> common_dtype does not return the real dtype of our Variable.
Just one example, in the end it seems too cumbersome. But inheriting from ResourceVariable (relying only on public fields/adding our own attrs) seems to work out quite well. Do you see any danger in that?
Only problem we are currently facing (your guess: does this may come from the direct instantiation vs using the get_variable factory?): randomly (depending on machine and luck) appearing PreConditionErrors ("Error while reading resource variable param1s from Container: localhost. This could..."), even directly initializing the variable after it's instantiation does not help. But we'll try to systematically break that down and find the problem.
We tried again with that, but unfortunately there are too many things to
get stuck in like: tfp.Distributions use a common_dtype function to
determine the dtype of the objects and it does not check for a dtype
attribute directly but only if it's the instance of a known type (which our
Variable of course is not) -> common_dtype does not return the real dtype
of our Variable.
Just one example, in the end it seems too cumbersome. But inheriting from
ResourceVariable (relying only on public fields/adding our own attrs)
seems to work out quite well. Do you see any danger in that?
Only problem we are currently facing (your guess: does this may come from
the direct instantiation vs using the get_variable factory?): randomly
(depending on machine and luck) appearing PreConditionErrors ("Error
while reading resource variable param1s from Container: localhost. This
could..."), even directly initializing the variable after it's
instantiation does not help. But we'll try to systematically break that
down and find the problem.
@mayou36 commented on Mon Oct 08 2018
Documentation of Parameters
There are two kind of parameters. They both can return a value and have dependents and a name.
Dependent (
CompositeParameter)The dependents of this parameter are
Parameters. They can't have limits and aren't changeable but act as a "wrapped" tensor.Independent (
Parameter)These are trainable and assignable. There dependents are
{}, as they are independent. They get added to the tf-collection "zfit_independent".Specs
API
Notes:
Postponed work, wait until TF variables stabilized
change shape
Changing from a single value to n values (usable for integration/MC error):
update var
use
loaddependent vars
variables depending on each other must be carefully treated with their initialization.
@mayou36 commented on Thu Nov 08 2018
Our use-case for the inheritance of tf.Variable: we build a fitting library using probability density functions and minimize a negative log-likelihood. Variables for us are therefore not just some t unable parameter out of millions but we use up to maximal 100 (often also only 5-10) and they have a lot of "meaning".
This goes as follows: create a loss function, minimize it, (use it in another loss function, minimize again,...). We don't need, for example, distributed training (one GPU per minimization is sufficient).
What we need that is not in tf.Variable:
There are a few more things (like making easy compositions).
@alextp what do you think? I think ResourceVariable could fit here well.
@alextp commented on Thu Nov 08 2018
Can you do this with a wrapper class around Variable? To make your wrapper object usable as a tf tensor you just need to call tf.register_tensor_conversion_function on it, and you can define your own assign methods and whatnot.
I think this should work since I assume you're not using tf optimizers as those don't fit this use case well.
@mayou36 commented on Wed Nov 14 2018
@alextp That's an interesting thought, so I took a look at it. I realised though that we still need to be very close to what a tf.Variable does, (like all the overloaded operators etc) and don't require to do too much more (basically, a fancy
constraintfunction could handle all that). So say multiplying two Variables should work out of the box.And with the optimizers: we don't need them currently as they are first order algorithms and we can afford higher order (like BFGS), but we definitely need optimizers and would like to stay compatible (hoping for example for
tfp.optimizersto becometf.Optimizercompatible).Do you think it is unreasonable to inherit from
ResourceVariable? Or are there any dangers with that?Besides, is there somewhere good documentation on the Variables mechanics, as I have searched a lot but except of the official docs (which don't say anything about that lower-level handling), I did not find anything.
@alextp commented on Mon Nov 19 2018
We have a plan to open up a subclass-variable API which will not rely on _underscored private stuff. For now though implementing convert_to_tensor and the operator overloads yourself will get you ~most of what you described.
@mayou36 commented on Tue Nov 20 2018
We tried again with that, but unfortunately there are too many things to get stuck in like: tfp.Distributions use a
common_dtypefunction to determine thedtypeof the objects and it does not check for adtypeattribute directly but only if it's the instance of a known type (which our Variable of course is not) ->common_dtypedoes not return the realdtypeof our Variable.Just one example, in the end it seems too cumbersome. But inheriting from
ResourceVariable(relying only on public fields/adding our own attrs) seems to work out quite well. Do you see any danger in that?Only problem we are currently facing (your guess: does this may come from the direct instantiation vs using the
get_variablefactory?): randomly (depending on machine and luck) appearingPreConditionErrors("Error while reading resource variable param1s from Container: localhost. This could..."), even directly initializing the variable after it's instantiation does not help. But we'll try to systematically break that down and find the problem.@alextp commented on Tue Nov 20 2018
Inheriting from ResourceVariable should be fine, I just want to make it easier and less error-prone.
This error shouldn't happen if initialization is being done properly, it'll take a reproducible case and some debugging to see what's going on.
On Tue, Nov 20, 2018 at 6:23 AM Jonas Eschle notifications@github.com wrote:
--
@mayou36 commented on Tue Nov 20 2018
Yes, a subclassable
Variablewould be great!Yep, of course. thx a lot