Memory Access Violation

[dlshriver]

When running the NNMNISTAccuracy tools, they crash after about 1 hour of running with the following exception:

Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at SolverFoundation.Plugin.Gurobi.GurobiNativeInterface.GRBoptimize(IntPtr model)
   at SolverFoundation.Plugin.Gurobi.GurobiNativeInterface.Solve(IntPtr _lp)
   at SolverFoundation.Plugin.Gurobi.GurobiSolver.Solve(ISolverParameters param)
   at NNAnalysis.Utils.LPSolver.SolveLowLevelLP() in C:\Users\jggrn\Documents\Research\NeuralNetworkAnalysis\src\NNAnalysis\Solver.cs:line 201
   at NNAnalysis.Utils.Robustness.SynthesizeCounterexample(NeuralNet nn, LPSTerm[] inputs, LPSTerm epsilon, LabelWithConfidence imageLab, NNInstrumentation instr, Int32 realLabel, Int32 rowSize, Int32 colSize, Boolean isColor) in C:\Users\jggrn\Documents\Research\NeuralNetworkAnalysis\src\NNAnalysis\Robustness.cs:line 495
   at NNAnalysis.Utils.Robustness.SynthesizeCounterexamples(NeuralNet nn, ImageDataset ds, Action`1 snapshot) in C:\Users\jggrn\Documents\Research\NeuralNetworkAnalysis\src\NNAnalysis\Robustness.cs:line 282
   at NNAnalysis.Utils.Robustness.SynthesizeCounterexamplesAndStore(NeuralNet nn, ImageDataset ds, Action`1 snapshot) in C:\Users\jggrn\Documents\Research\NeuralNetworkAnalysis\src\NNAnalysis\Robustness.cs:line 214
   at Program.Main(String[] args) in C:\Users\jggrn\Documents\Research\NeuralNetworkAnalysis\src\NNAnalysis\NNMNISTAccuracy\Program.cs:line 129

I ran the tool with the following command:

.\NNMNISTAccuracy.exe --nnet ..\NeuralNetworkAnalysis\protobuf\mnist.mldiffmodel --datafile ..\mnist\t10k-images-idx3-ubyte --labelfile ..\mnist\t10k-labels-idx1-ubyte --registry=testRegistry

It seems to be related to a memory leak somewhere in the program. A similar problem occurs with NNCIFARAccuracy. The memory steadily increases, until it gets to 2G, then the program crashes.

Is there a work around or simple fix?

[dimitriv]

It's been a long time but I had noticed similar problems in the past, and IIRC it had to do with the native/managed memory interface in the guts of the Microsoft Solver Foundation, of which I have no control really, and it's not documented in a lot of detail. Also, I guess you are not building for Release/64bit, have you tried that? I am pretty sure I'd left such processes running for hours without any crashes for the paper experiments, and without observing growing working sets, so this smells to me like some Configuration issue when building the project.

Now, I think that actually best path forward is to remove the dependency on the Solver Foundation and replace it with some other available solver.

As a short-term solution, what I've done in situations like this was to simply split the input files in smaller chunks and do them in separate processes. It's not great but should get you going if you cant get through the memory leak issue.

[dlshriver]

Thanks for the quick response! I did build in Release x64 mode, but there may be some other configuration option I overlooked. Splitting the data into chunks seems like the best solution for now.

[sebastianuh]

I'm using the C++ interface of the Gurobi optimizer version 8.1, and I also get a memory leak in the GRBoptimize method. This might be caused by Gurobi.