john-hen
commented
1 year ago Hi. Thanks for the detailed report. I'm afraid though the insight I can provide will be rather limited.
First off, I hope you have a way to resume the optimization runs. When I did this in the past, also with hundreds of iterations, I would "cache" the input parameters and output results (like the .csv files in your case). So when it crashed, I wouldn't have to rerun the whole thing again.
And there were crashes. I don't remember the specifics, maybe these access violations also happened occasionally. But there are many ways in which Comsol could crash. Like, if the disk fills up with temporary files.
Memory leaks must not necessarily be JPype's fault, that could also be Comsol. In fact, in past issues here, Comsol was the culprit more often than not. Keep in mind that what you're doing is a rare use case: running that many solves within the same session. Comsol is arguably not battle-tested for this scenario. Most people do a few solves, whether that's in the GUI or in their custom Java-based application that uses Comsol as a back-end.
I've also attempted resetting the client (and hopefully the JPype connection?) each iteration with both client.clear() and a more agressive sequence of delete, garbage collect, and client restart. Neither has worked.
How do you restart the client? I don't think that's possible. We cannot restart the Java VM within the Python process. So the only way to restart the client is to terminate the session, i.e. Python must exit.
I have also attempted to tweak the
-XX:MaxMetaspaceSize -Xss -Xms -Xmxparameters located in the comsol.ini
That's a good point, I've never thought about this. (Also because these details are not well documented on Comsol's side.) But since we start the Java VM ourselves (via JPype), the settings in comsol.ini do not have any effect. I just tested that. We could of course parse the file and set the parameters. It's an option I will consider. But there are also settings in there which we should ignore, so I'm not sure how robust and future-proof that would be.
If you want to play around with the memory settings, you can start the session in client-server mode. That's something you should try anyway, maybe that alone solves it. Since you're on Windows, the default mode of operation is a stand-alone client. You can switch to client-server mode withmph.option('session', 'client-server'). Then the relevant configuration file is comsolmphserver.ini. And since we start the server as an external process, it should just read that file like it always does, when started from the command line.
I'm no expert on this, but I don't think memory is the issue here. Note that the metaspace is not nearly maxed out. What you see there is the "capacity", but that capacity should be dynamically increased if more is needed. The maximum metaspace size you also find in the crash log. It's an astronomical 17179869184.00 GB.
I don't know what "virtual space" is. Again, not an expert on the Java VM. And yeah, the 99% (in both cases) is strange. Looks like the garbage collector should do something about that. But I don't see how that leads to an access violation. Which seems to occur in native code, and thus outside the Java VM.
AlecEmser
I've written some code which uses an SHGO optimizer to feedback on the .csv output of a COMSOL file. The program is fairly simple: a COMSOL model is modified with parameters passed by the SHGO optimizer and then solved. A probe table is saved to a csv, which is read and analyzed to return a cost to the SHGO optimizer. This all runs very smoothly and converges without problem for small iteration numbers.
However, after several hours of running the program (~several hundred calls to
model.solve()) I receive the following error which kills the python kernel:The full output of this log is attached: hs_err_pid22828.log I suspect that the issue is related to Java running out of non-class metaspace, as the allocations are always full when failure occurs:
I have ensured everything in my Python distribution is up to date and have run the code with both Spyder and VSCode Jupyter yet the issue persists. Similarly, I have tried both COMSOL 6.0 and 6.1. Assuming that the issue might be some memory leak associated with JPype, I've also attempted resetting the client (and hopefully the JPype connection?) each iteration with both client.clear() and a more agressive sequence of delete, garbage collect, and client restart. Neither has worked.
I have also attempted to tweak the
-XX:MaxMetaspaceSize -Xss -Xms -Xmxparameters located in the comsol.ini file underCOMSOL\COMSOL61\Multiphysics\bin\win64\but these values seem uncorrelated to the virtual space described in the log. The machine I am running on has 128G of RAM so I am not afraid of allocating significantly more memory to the metaspace that appears to be filling up, if only I knew how.Anyway, thank you for any insight into this problem. I am happy to share any more information which may be helpful.