Closed sparth95 closed 6 years ago
guykatzz
commented
6 years ago I'm getting the following error when I try to run the example:
/pprojects/marabou/maraboupy$ python3 debugsmall.py
Traceback (most recent call last):
File "debugsmall.py", line 23, in
@ShantanuThakoor, @clazarus, do you know why this is happening?
kjulian3
commented
6 years ago Line 169 "self.getValues[i]" should be self.getValues(i)", since self.getValues is a function. However, after this fix the example still produces errors later in line 171 because op.node_def does not have a field "inputs". I'm not sure how lines 169-172 are supposed to work, and looking at the file history this part of the code has not been changed since the original commit. The Reshape layer probably has not been tested/debugged yet.
I think it would be good if @ShantanuThakoor could take a look.
sparth95
commented
6 years ago if op.node_def.op in ['Reshape']: prevValues = [self.getValues(i) for i in input_ops] shape = prevValues[1] return np.reshape(prevValues[0], shape)
There was some bug with the reshape operator. This should cover it.
kjulian3
commented
6 years ago That fixes the issue for me. Can you submit a pull request?
sparth95
commented
6 years ago Sure, I will do it after the meeting today.
guykatzz
commented
6 years ago So, this appears to be a numerical stability issue.
During the initialization step Marabou performs Gaussian elimination on the basis matrix, which is just a matrix comprised of some of the coefficients of the input equations. In every step of the elimination process it looks at a single column of the matrix and picks the largest element in that column as the pivot element, by which the rest of the column is divided. Unfortunately, in one of the iterations even the largest element in the column is very small, and so dividing by it leads to large and inaccurate numbers. This is exacerbated in the next iterations, until you eventually see the huge numbers that you do.
I've looked at the input equations and they look fine to me, so I don't think there's anything wrong in the input.
I think the solution will involve smarter basis factorization techniques, that can avoid these small pivots. I am working on that part of the code anyway and will add this, but it will likely take me a while. In the meantime, maybe you can try making changes to your input problem and see if that works - different coefficients, different variable ordering, etc.
guykatzz
commented
6 years ago Hi Parth, I tried re-running your example and on the master branch, and I'm getting some Python errors. Can you have a look?
./debugsmall.py
/usr/lib/python3/dist-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "./debugsmall.py", line 25, in
ShantanuThakoor
commented
6 years ago Actually, a similar bug occurs with the Reshape operator as the Identity operator fix I pushed in the last PR. We will submit a PR for fixing this bug by tonight.
sparth95
commented
6 years ago I submitted a pull request earlier for the reshape operator change, I will update the code based on Shantanu's recent changes and submit another one.
sparth95
commented
6 years ago Guy, did you make any recent changes to fix this issue?
I also submitted a pull request to update reshape operator. I tried running the issue with this but it is throwing an error with the latest commits. I checked out an earlier commit and the error reappears then.
guykatzz
commented
6 years ago Hi Parth, Can you check if the problem persists after my latest changes?
sparth95
commented
6 years ago it is working for me too. I will test it out on bigger convolution networks.
I am trying to run an example with convolutions in Marabou, but most of the times I run into high degradation error. I have tried to recreate the error on a smaller model.
Model: Input(10 10 1) -> Convolution(Filter size(3 3 1 2)) -> Output(8 8 * 2) -> Reshape(128)
While running marabou on this small model, I run into very high degradation values, even immediately after preprocessing. I looked into this and I think the initial assignment of values to basic variables seems a bit off. I have attached output which contains the initial assigned value to each variable and some of these values go in the order of 10^10. The degradation value with the small number of variables is also 64,000.
I verified the network using forward pass and the difference in values from Marabou and tensorflow is negligible, so I think the equations are set up correctly. I also checked them out on a small example. I have attached the testing script, the example protobuf and the output which contains the initial assignment to all the variables.
git_issue.zip