calibration not terminating. MemoryError

[Jakob-We]

Version for dmlib-0.0.21 for the calibration. Data aquisiton like 2 month old version with 3 Steps. Using a uEye and alpao DM97-15 with 97 actuators. PC specs: AMD Ryzen Threadripper 1920X 12-Core, 128 GB Ram using Python 3.7.4(Anaconda) Run: Python.exe calibration.py in anaconda prompt. Just removed the .decode('utf-8') in line 30 since it was already a String and therefore raising an error and changed finame and radius_um. Get the following output: Computing Zernike polynomials ... Computing masks ... Computing phases 00.00% ... 24 times Computing Zernike polynomials ...

Then the ram is slowly filling with cpu at 100%

then with full ram: a few times "Computing masks ..." and then multiple times MemoryError Computing masks ... Traceback (most recent call last): File "", line 1, in File "C:\Users\xx\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\xx\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\xx\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\xx\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main__") File "C:\Users\xx\Anaconda3\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\xx\Anaconda3\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\xx\Anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "T:\xxx\dmlib\dmlib\examples\calibration.py", line 61, in status_cb=print) File "C:\Users\xx\Anaconda3\lib\site-packages\dmlib\calibration.py", line 156, in calibrate zfA1, zfA2, mask = self._make_zfAs() File "C:\Users\xx\Anaconda3\lib\site-packages\dmlib\calibration.py", line 107, in _make_zfAs zfA1 = np.zeros((self.zfm.sum(), self.cart.nk))

and it seems like it was loading the .h5 file again. And next I got "Computing phases 00.00% ..." followed by the error multipl times:

RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

here I killed the process after some time. I tried this with other amount of steps and older versions but it never terminated within a reasonable time sometime I even let it run over night.

Is this normal and I just have not enough ram? Do you have any ideas how to solve this problem?

[jacopoantonello]

Can you run the original calibration.py using the example data? Just fix the string error, without changing anything else. If so, then probably you have something wrong with your modifications, data, or parameters.

You should run calibration.py from a terminal (Anaconda Prompt), not from within an IDE (Spyder, PyCharm, ...), otherwise there can be issues spawning new processes.

Perhaps your own HDF5 file is huge? Like you're using a camera with a huge resolution? A cheap 1024x1280 camera is more than sufficient. The higher the camera resolution the unnecessarily slower this will be. Yet because your PC is so powerful, perhaps this is not even a problem.

I can run calibration.py with the example data on a cheap laptop. If you can as well in another laptop for comparison, then maybe there's something wrong with spawning processes in Anaconda / Python and your powerful PC.

[Jakob-We]

Thanks for the quick response. And sry for the late reply I had to wait before I could use the PC and had other stuff to do aswell. The HDF5 file is about 600 MB for 3 Steps and the image resolution is 1216 x 1936. I did run the calibration from terminal (Anaconda Prompt).

I tried to run the example data with the unmodified calibration.py and I still got the RuntimeError from above but the RAM did not overflow with this data.

Now the strange thing so someone else I know is using a slightly different PC (also 128 GB RAM and a differend Threadripper) and an older version of dmlib to calibrate a dm and he uses Spyder to run his code witch is modified a bit on the side how he loads the images. But the rest is basicly the same. And when I tried to run the example data on the same PC where he is working on it did not work with Spyder neither with Anaconda Prompt. And my own data didnt work aswell.

So you think its worth trying to run the calibration on a PC with a more "normal" CPU?

[jacopoantonello]

There was a missing if __name__ == '__main__': in calibration.py. Should be fixed now.

[Jakob-We]

Thanks it looks like its working now just run one calibration successful.