Open pkienzle opened 5 years ago
Trac update at 2017/10/27 10:09:33
: butler changed description from:
The following OpenCL memory error was reported from SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.
{{{ 2017-10-23 17:39:12 : Traceback (most recent call last): File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in call File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/init.py", line 1088, in kernel_call File "
", line 169, in enqueue_knl_cylinder_Iq LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT === Steps: 161 of 159 chisq: 70.4 ETA: -86s M2.p2_radius: 71.5 | M2.p2_sld_shell: -0.108 | M2.p2_thickness: 75.8
M1.p1_scale: 0.000499 | M1.p1_sld_solvent: 1.38 | M1.p2_radius: 136
M1.p2_radius.width: 0.851 | M1.p2_scale: 0.0181 | M1.p2_sld_core: 2.08
M1.p2_thickness: 14.1 | M4.p2_radius: 64.8 | M4.p2_sld_shell: -0.0451
M4.p2_thickness: 79.6 | M3.p2_radius: 69 | M3.p2_sld_shell: -0.199
M3.p2_thickness: 83.7 | M3.p2_thickness.width: 0.794 |
2017-10-23 17:42:59 : Traceback (most recent call last): File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT2017-10-23 17:43:49 : (' 0.357', ' 0.697') 2017-10-23 17:43:50 : (' 0.357', ' 0.697') 2017-10-23 17:43:51 : (' 0.505', ' 15') 2017-10-23 17:43:51 : (' 0.505', ' 15') 2017-10-23 17:43:55 : (' 0.0847', ' 0.268') 2017-10-23 17:43:55 : (' 0.0847', ' 0.268') 2017-10-23 17:45:05 : Computation completed! 2017-10-23 17:45:07 : Computation completed! 2017-10-23 17:45:20 : Computation completed! 2017-10-23 17:45:21 : Computation completed! 2017-10-23 17:45:22 : Computation completed! 2017-10-23 17:45:25 : Computation completed! 2017-10-23 17:45:26 : Computation completed! 2017-10-23 17:45:27 : Computation completed! 2017-10-23 17:45:29 : Computation completed! 2017-10-23 17:45:40 : Computation completed! 2017-10-23 17:45:45 : Computation completed! 2017-10-23 17:46:15 : Computation completed! 2017-10-23 17:48:42 : Fitting is in progress... 2017-10-23 17:48:42 : Fitting is in progress... 2017-10-23 17:48:42 : Traceback (most recent call last): File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps AttributeError: 'FitDriver' object has no attribute 'fitter'
None }}}
to:
The following OpenCL memory error was reported from !SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.
{{{ 2017-10-23 17:39:12 : Traceback (most recent call last): File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr File "sas/sascalc/fit/BumpsFitting.pyc", line 289, in fit File "sas/sascalc/fit/BumpsFitting.pyc", line 202, in theory File "sas/sascalc/fit/BumpsFitting.pyc", line 192, in _recalculate File "sas/sascalc/fit/AbstractFitEngine.pyc", line 242, in residuals File "/Users/lionel/.sasview/plugin_models/Cyl_CS_sphere.py", line 272, in evalDistribution File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 582, in evalDistribution File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 617, in calculate_Iq File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/sasview_model.py", line 638, in _calculate_Iq File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/sasmodels/kernelcl.py", line 567, in call File "/Applications/SasView 4.1.app/Contents/Resources/lib/python2.7/pyopencl/init.py", line 1088, in kernel_call File "
", line 169, in enqueue_knl_cylinder_Iq LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT === Steps: 161 of 159 chisq: 70.4 ETA: -86s M2.p2_radius: 71.5 | M2.p2_sld_shell: -0.108 | M2.p2_thickness: 75.8
M1.p1_scale: 0.000499 | M1.p1_sld_solvent: 1.38 | M1.p2_radius: 136
M1.p2_radius.width: 0.851 | M1.p2_scale: 0.0181 | M1.p2_sld_core: 2.08
M1.p2_thickness: 14.1 | M4.p2_radius: 64.8 | M4.p2_sld_shell: -0.0451
M4.p2_thickness: 79.6 | M3.p2_radius: 69 | M3.p2_sld_shell: -0.199
M3.p2_thickness: 83.7 | M3.p2_thickness.width: 0.794 |
2017-10-23 17:42:59 : Traceback (most recent call last): File "sas/sascalc/data_util/calcthread.pyc", line 268, in _run LogicError: when processing argument #4 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT2017-10-23 17:43:49 : (' 0.357', ' 0.697') 2017-10-23 17:43:50 : (' 0.357', ' 0.697') 2017-10-23 17:43:51 : (' 0.505', ' 15') 2017-10-23 17:43:51 : (' 0.505', ' 15') 2017-10-23 17:43:55 : (' 0.0847', ' 0.268') 2017-10-23 17:43:55 : (' 0.0847', ' 0.268') 2017-10-23 17:45:05 : Computation completed! 2017-10-23 17:45:07 : Computation completed! 2017-10-23 17:45:20 : Computation completed! 2017-10-23 17:45:21 : Computation completed! 2017-10-23 17:45:22 : Computation completed! 2017-10-23 17:45:25 : Computation completed! 2017-10-23 17:45:26 : Computation completed! 2017-10-23 17:45:27 : Computation completed! 2017-10-23 17:45:29 : Computation completed! 2017-10-23 17:45:40 : Computation completed! 2017-10-23 17:45:45 : Computation completed! 2017-10-23 17:46:15 : Computation completed! 2017-10-23 17:48:42 : Fitting is in progress... 2017-10-23 17:48:42 : Fitting is in progress... 2017-10-23 17:48:42 : Traceback (most recent call last): File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 82, in compute File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 14, in map_apply File "sas/sasgui/perspectives/fitting/fit_thread.pyc", line 11, in map_getattr File "sas/sascalc/fit/BumpsFitting.pyc", line 275, in fit File "sas/sascalc/fit/BumpsFitting.pyc", line 375, in run_bumps AttributeError: 'FitDriver' object has no attribute 'fitter'
None }}}
Trac update at 2017/10/27 16:17:27
: butler changed milestone from "SasView 4.2.0" to "SasView 4.3.0"
Trac update at 2018/10/23 21:45:02
: pkienzle commented:
The resulting error occurs also when a model is displayed on the fit page then you test a new OpenCL device:
LogicError: when processing argument #233 (1-based): clsetkernelarg failed: INVALID_MEM_OBJECT
In this case, I believe it is because the program was compiled in a different OpenCL context than the one used to allocate the q vectors. We need to keep all references to the buffers and kernels in the context object and retrieve them from when running. That way when the context is reset we can force a new set of buffers to be allocated and recompile the kernels.
Note: may want to store the compiled model as a class variable in the SasviewModel wrapper so that it doesn't get recompiled each time you select a model on the fit page. At the top of SasviewModel._calculate_Iq
use self.__class__._model = core.build_model(self._model_info)
. The model info attribute is already stored at the class level in _generate_model_attributes
, but the build model step was being done lazily.
Trac update at 2018/10/24 14:23:28
: pkienzle commented:
The work-around is to tell the user to reselect the model.
Trac update at 2019/03/28 14:56:34
: wojciech commented:
I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.
Trac update at 2019/03/29 00:07:14
: pkienzle commented:
Replying to [comment:5 wojciech]:
I also experience this when switching between models and categories (cylinder and ellipsoids). I've been stress testing the GPU Options interface for 5.0.
Are you working from sasmodels/master or a tagged revision?
There was some additional cleanup that didn't get merged until the beta_approx branch was merged, though it's been too long for me to remember the details.
@pkienzle and @wpotrzebowski , to what extent is this still an issue? and if it is, is a SasView issue or a sasmodels issue? I think for now I will move this to 5.1 pending feedback on whether to close? Given it has been this way for a while I don't see it being something to spend resources on for a final 4.x release
The following OpenCL memory error was reported from !SasView. This may be because sasview_model.py (calculate_Iq) is allocating a new q vector each time the model is evaluated rather than reusing the same q vector across all fits. Restructuring the calculator so that the q values are held for the life of the model may fix the error.
Migrated from http://trac.sasview.org/ticket/1015