Open dksim opened 6 years ago
Thanks for your bug report @dksim. Clearly the diagnostics
module is not on par with the rest of the library.
Regarding your points:
Operator.norm
interface is relatively new, see #1065, so OperatorTest
may not be up to date with that. In any case, can you give some more details?Related issues: #551, #717
Thanks for the response @kohr-h. Upon further inspection and based on your comments, here is an updated status
ValueError Traceback (most recent call last)
in () ----> 1 optest = odl.diagnostics.OperatorTest(ray_trafo) ~\Documents\odl\odl\diagnostics\operator.py in __init__(self, operator, operator_norm, verbose, tol) 51 self.verbose = False 52 if operator_norm is None: ---> 53 self.operator_norm = self.norm() 54 else: 55 self.operator_norm = float(operator_norm) ~\Documents\odl\odl\diagnostics\operator.py in norm(self) 86 operator_norm = max(power_method_opnorm(self.operator, maxiter=2, 87 xstart=x) ---> 88 for name, x in samples(self.operator.domain) 89 if name != 'Zero') 90 ~\Documents\odl\odl\diagnostics\operator.py in (.0) 87 xstart=x) 88 for name, x in samples(self.operator.domain) ---> 89 if name != 'Zero') 90 91 self.log('Norm is at least: {}'.format(operator_norm)) ~\Documents\odl\odl\operator\oputils.py in power_method_opnorm(op, xstart, maxiter, rtol, atol, callback) 248 x_norm = x.norm() 249 if x_norm == 0: --> 250 raise ValueError('reached ``x=0`` after {} iterations'.format(i)) 251 if not np.isfinite(x_norm): 252 raise ValueError('reached nonfinite ``x={}`` after {} iterations' ValueError: reached ``x=0`` after 0 iterations
derivative
is implemented in the operator, in which case the default implementation just returns self
, and the condition is True
.
This is the correct behavior for 99 % of the cases.derivative
of a linear operator that does not return self
. I have a hard time imagining what that reason might be. In this case, the condition evaluates to False
and the derivative is being tested.Yes, adding some documentation or a warning would be useful. In my understanding the OperatorTest class serves to help a user check if the implementation of an operator derivative, adjoint etc. is correct. If this is not reliable (even if this is due to external libraries), then this probably causes confusion instead of fulfilling its purpose and the user should be at least warned.
I agree to your general sentiment that failing diagnostics can be confusing if they just reflect expected behavior (from the implementation point of view).
Let me just add that the typical user who makes use of that functionality is probably an advanced user who likes to get some quick correctness test for a custom operator they implemented. In that scenario, the diagnostics typically give a much larger error initially if there's a real issue in the implementation, more like in the order of 1. After fixing the bugs, the error may go down to single precision error. Since the user likely has a good understanding of the implementation details, they can tell if a failure is more likely due to loss of precision in one of the steps, or due to another, more subtle bug.
The diagnostics are less useful as a tool for somebody who browses the library and tests some random operator from the shelf. That said, we should definitely clarify the intention of this subpackage, and explain how to interpret results.
Also note that the linearity failure you observed was in an internal method that would not have been called by the public OperatorTest.derivative
method.
If you run the adjointness tests you will see similar things with the ray transform. Adjointness is much harder to get correct, and with external libraries there's just nothing you can do but hope that they fix the issue for you.
It appears the odl.diagnostics.OperatorTest class has a few bugs
Examples:
1. Failure to estimate norm (raises ValueError)
2. Linear operator fails derivative and adjoint tests
3. For non-linear operators there shouldn't be test for adjoints or calculation of norm
4. The following check may fail in some cases where it shouldn't (
deriv is self.operator
will only work if both are referencing the same object in memory)