search

radiasoft / sirepo

Sirepo is a framework for scientific cloud computing. Try it out!
https://sirepo.com
Apache License 2.0
63 stars 31 forks source link

error in lattice gate 3D execution #3389

Open ncook882 opened 3 years ago

ncook882 commented 3 years ago

Upon completing the simulation execution, the following error is given. The Impact Density and Particle Trace Reports are not produced.

Screen Shot 2021-03-10 at 2 53 30 PM

Link to Simulation: https://beta2.sirepo.com/warpvnd#/visualization/QBuMmxcP

moellep commented 3 years ago

If you run the same simulation with Execution "Serial" it completes successfully. I noticed the field animation also fails for STL simulation when run in parallel mode. I'll create a simple python script which reproduces the problem.

moellep commented 3 years ago

Attached is a warp python script which crashes when run with a single CPU. I tested on sirepo dev and on jupyter - both result in a seg fault during the call to scipy_interp1d(). I'm guessing the bug isn't in scipy, but something else is corrupting memory prior to the call.

warp-crash.zip

moellep commented 3 years ago

Top of the gdb stack trace warp toppy:

#0  0x00007eff6ab7f686 in dtbsv_ ()
   from /home/vagrant/.pyenv/versions/py3/lib/python3.7/site-packages/warp/toppy.cpython-37m-x86_64-linux-gnu.so
#1  0x00007eff6845ab81 in dgbtrs_ ()
   from /home/vagrant/.pyenv/versions/py3/lib/python3.7/site-packages/scipy/special/../../scipy.libs/libopenblasp-r0-085ca
80a.3.9.so
cchall commented 3 years ago

I wouldn't claim to know what is really going on here, but moving the scipy.interpolate.interp1d before warp is imported seems to allow the script to complete successfully for me.

moellep commented 3 years ago

Thanks @cchall - that is a good work-around for now.

ncook882 commented 3 years ago

I'm still seeing what appears to be slow performance and/or the simulation not completing at the final frame. @cchall are you seeing this as well?

https://www.sirepo.com/warpvnd#/visualization/DzjcEZit

moellep commented 3 years ago

I'm running the same simulation on alpha now and it is also taking a really long time on the last step. The last step is when it calculates the impact density. That step is usually the longest for STL simulations, but in this case it seems to take longer than normal.