Errors when loading meshes from .cdb files when using multiprocessing

maximepollet27 commented 2 years ago

Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

Description of the bug

Hi,

For my work, I need to run many FE analyses (several thousand), and I have been using pyansys (I love it btw :)). As time matters, I have been exploring various methods for optimizing the speed. In particular, I am loading pre-made meshes from .cdb files and running several analyses simultaneously using the multiprocessing module.

The script fea.py is as follows:

from multiprocessing import Pool
from ansys.mapdl.core import launch_mapdl
from pathlib import Path
import tqdm

def fea_analysis(sample):

    mapdl = launch_mapdl(exec_file='C:/Program Files/ANSYS Inc/v212/ansys/bin/winx64/ANSYS212.exe')
    mesh_path = Path().cwd().joinpath(sample)
    mapdl.cdread('db', mesh_path)

if __name__ == '__main__':

    samples = ['1.cdb', '2.cdb']

    with Pool(2) as pool:
        results = list(tqdm.tqdm(pool.imap(fea_analysis, samples), total=len(samples)))

where tqdm is a module that allows me to display a progress bar, fea_analysis is my analysis function in which mapdl instances are created and terminated, and samples is an indexing list.

When loading the .cdb mesh files, some of the analyses (not all of them) fail with the following error:

Traceback (most recent call last):
  File "C:\Users\mp2333\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "E:\Maxime\ansys_analysis\fea.py", line 17, in fea_analysis
    mapdl.cdread('db', mesh_path)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 1134, in cdread
    self.input(fname, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 116, in wrapper
    out = func(*args, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 1293, in input
    with open(os.path.join(local_path, tmp_name), "w") as f:
PermissionError: [Errno 13] Permission denied: 'C:/Users/mp2333/AppData/Local/Temp/ansys_wtkojixrqs\\_input_tmp_.inp'

I also had the same window pop-up error as here: #181 a few times (I couldn't find a general rule for predicting which of the two errors gets raised).

These errors only occur when the analyses are performed simultaneously using Pool. When I run only one, it works fine. Because of this behavior and of the error messages, I think that there is maybe some sort of conflict occurring when writing and reading files. It might also have some connection with what is discussed here #844 as the path PermissionError: [Errno 13] Permission denied: 'C:/Users/mp2333/AppData/Local/Temp/ansys_wtkojixrqs\\_input_tmp_.inp' has a mixture of / and \.

Please let me know if you have any idea of where this might come from!

Steps To Reproduce

Rename 1.txt and 2.txt as 1.cdb amd 2.cdb (Github won't let me import .cdb files)
Move to a directory with fea.py, 1.cdb and 2.cdb.
Run: python fea.py
As mentioned, running the exact same script several times can trigger different errors, so several trials might be needed to reproduce the error. 1.txt 2.txt

Which Operating System are you using?

Windows

Which Python version are you using?

3.7

Installed packages

aiohttp==3.8.1 aiosignal==1.2.0 ansys-api-mapdl==0.5.1 ansys-corba==0.1.0 ansys-grpc-mapdl==0.2.0 ansys-mapdl-core==0.61.6 ansys-mapdl-reader==0.51.11 appdirs==1.4.4 async-timeout==4.0.2 asynctest==0.13.0 attrs==21.4.0 cachetools==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 colorama==0.4.4 commonmark==0.9.1 COMPAS==1.15.1 contextlib2==21.6.0 cycler==0.11.0 Cython==0.29.28 fonttools==4.32.0 frozenlist==1.3.0 geomdl==5.3.1 gmsh==4.10.0 google-api-core==2.7.1 google-api-python-client==2.43.0 google-auth==2.6.3 google-auth-httplib2==0.1.0 googleapis-common-protos==1.56.0 grpcio==1.44.0 httplib2==0.20.4 idna==3.3 imageio==2.16.2 importlib-metadata==4.11.3 importlib-resources==5.7.1 jsonschema==4.4.0 kiwisolver==1.4.2 matplotlib==3.0.3 meshio==5.3.4 mpmath==1.2.1 multidict==6.0.2 networkx==2.6.3 numpy==1.21.5 packaging==21.3 pandas==1.3.5 parallelbar==0.1.19 Pillow==9.1.0 protobuf==3.20.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycollada==0.7.2 Pygments==2.12.0 pyiges==0.2.1 pyparsing==3.0.8 pyrsistent==0.18.1 python-dateutil==2.8.2 pytz==2022.1 pyvista==0.34.0 requests==2.27.1 rich==12.3.0 rsa==4.8 schema==0.7.5 scikit-spatial==6.4.0 scipy==1.7.3 scooby==0.5.12 six==1.16.0 slackclient==2.9.3 sympy==1.10.1 tqdm==4.64.0 typing_extensions==4.1.1 uritemplate==4.1.1 urllib3==1.26.9 vtk==9.1.0 watchdog==2.1.7 wslink==1.5.2 yarl==1.7.2 zipp==3.8.0

germa89 commented 2 years ago

Hi @maximepollet27

Thank you very much for posting this issue.

It is indeed a permission error. When using mapdl.cdread PyMAPDL resources to mapdl.input which generates a file _input_tmp_.inp in the Python Working directory where it writes some commands (hence the "w" in that line you mentioned). Since you have two pools, and both with the same Python working directory, both processes are attempting to write the same file, hence the error.

I will work on a solution for this. On your side, without editing the code, I do not think there is much you can do. Since the file name is hardcoded.

maximepollet27 commented 2 years ago

Hi @germa89

Thank you for being so quick to reply!

maximepollet27 commented 2 years ago

Hi @germa89

Sorry for reopening this issue so long after you found the first solution, I was occupied by other things. After installing the latest update, I resumed trying to implement this script that combines multiprocessing and loading .CDB mesh files. Unfortunately, I keep running in issues when i run the script. The errors are not always the same and seem to still be due to temporary files reading and writing. Here are some examples of the errors I run into:

Traceback (most recent call last):
  File "E:\Maxime\fe_debug\fea.py", line 11, in fea_analysis
    mapdl.cdread('db', mesh_path)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 1183, in cdread
    self.input(fname, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 116, in wrapper
    out = func(*args, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 1358, in input
    with open(os.path.join(local_path, tmp_out)) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/mp2333/AppData/Local/Temp/ansys_hrooifpbup\\_input_tmp_yeqwmuykmg_.out'

or

Traceback (most recent call last):
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1488, in launch_mapdl
    **start_parm,
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 344, in __init__
    self._multi_connect()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 392, in _multi_connect
    timeout=attempt_timeout, set_no_abort=set_no_abort
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 533, in _connect
    self._set_no_abort()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\misc.py", line 283, in wrapper
    out = func(*args, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 666, in _set_no_abort
    self.nerr(abort=-1, mute=True)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\_commands\session\run_controls.py", line 423, in nerr
    return self.run(command, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl.py", line 2572, in run
    text = self._run(command, verbose=verbose, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 729, in _run
    response = self._send_command(cmd, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 131, in wrapper
    raise MapdlExitedError("MAPDL server connection terminated") from None
ansys.mapdl.core.errors.MapdlExitedError: MAPDL server connection terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Maxime\fe_debug\fea.py", line 9, in fea_analysis
    mapdl = launch_mapdl(exec_file='C:/Program Files/ANSYS Inc/v212/ansys/bin/winx64/ANSYS212.exe')
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1496, in launch_mapdl
    lic_check.check()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\licensing.py", line 449, in check
    raise LicenseServerConnectionError("\n".join(self._license_file_msg))
ansys.mapdl.core.errors.LicenseServerConnectionError: Exceeded 10 seconds while waiting for C:\Users\mp2333\AppData\Local\Temp\.ansys\licdebug.FEAT_ANSYS.181.out to exist.

or

Traceback (most recent call last):
  File "E:\Maxime\fe_debug\fea.py", line 11, in fea_analysis
    mapdl.cdread('db', mesh_path)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 1183, in cdread
    self.input(fname, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 131, in wrapper
    raise MapdlExitedError("MAPDL server connection terminated") from None
ansys.mapdl.core.errors.MapdlExitedError: MAPDL server connection terminated

There are also times when the script run successfully. Do you have an idea of where this could come from ?

To produce these issues, I am still using the fea.py script from above, which I changed slightly:

from multiprocessing import Pool
from ansys.mapdl.core import launch_mapdl
from pathlib import Path
import tqdm
import traceback

def fea_analysis(sample):
    try:
        mapdl = launch_mapdl(exec_file='C:/Program Files/ANSYS Inc/v212/ansys/bin/winx64/ANSYS212.exe')
        mesh_path = Path().cwd().joinpath(sample)
        mapdl.cdread('db', mesh_path)
        mapdl.exit()
    except Exception:
        traceback.print_exc()

if __name__ == '__main__':

    samples = ['1.cdb', '2.cdb']

    with Pool(2) as pool:
        #results = list(tqdm.tqdm(pool.imap(fea_analysis, samples), total=len(samples)))
        results = list(pool.imap(fea_analysis, samples))

The packages I have installed are: aiohttp 3.8.1 aiosignal 1.2.0 ansys-api-mapdl 0.5.1 ansys-api-platform-instancemanagement 1.0.0b3 ansys-corba 0.1.0 ansys-mapdl-core 0.62.1 ansys-mapdl-reader 0.51.14 ansys-platform-instancemanagement 1.0.2 appdirs 1.4.4 async-timeout 4.0.2 asynctest 0.13.0 attrs 21.4.0 cachetools 5.0.0 certifi 2021.10.8 charset-normalizer 2.0.12 colorama 0.4.4 commonmark 0.9.1 COMPAS 1.15.1 contextlib2 21.6.0 cycler 0.11.0 Cython 0.29.28 fonttools 4.32.0 frozenlist 1.3.0 geomdl 5.3.1 gmsh 4.10.0 google-api-core 2.7.1 google-api-python-client 2.43.0 google-auth 2.6.3 google-auth-httplib2 0.1.0 googleapis-common-protos 1.56.0 grpcio 1.44.0 httplib2 0.20.4 idna 3.3 imageio 2.16.2 importlib-metadata 4.11.3 importlib-resources 5.7.1 jsonschema 4.4.0 kiwisolver 1.4.2 matplotlib 3.0.3 meshio 5.3.4 mpmath 1.2.1 multidict 6.0.2 networkx 2.6.3 numpy 1.21.5 packaging 21.3 pandas 1.3.5 parallelbar 0.1.19 Pillow 9.1.0 pip 22.0.4 protobuf 3.20.0 protoc-gen-swagger 0.1.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycollada 0.7.2 Pygments 2.12.0 pyiges 0.2.1 pyparsing 3.0.8 pyrsistent 0.18.1 python-dateutil 2.8.2 pytz 2022.1 pyvista 0.34.0 requests 2.27.1 rich 12.3.0 rsa 4.8 schema 0.7.5 scikit-spatial 6.4.0 scipy 1.7.3 scooby 0.5.12 setuptools 61.0.0 six 1.16.0 slackclient 2.9.3 sympy 1.10.1 tqdm 4.64.0 typing_extensions 4.1.1 uritemplate 4.1.1 urllib3 1.26.9 vtk 9.1.0 watchdog 2.1.7 wheel 0.37.1 wslink 1.5.2 yarl 1.7.2 zipp 3.8.0

germa89 commented 2 years ago

Hi @maximepollet27

Before I try something, can you use license_server_check=True at launch_mapdl??

maximepollet27 commented 2 years ago

Hi @germa89

Thanks a lot for the quick reply. I used license_server_check=True and ran the script two times. Here are the resulting errors for the two trials:

Traceback (most recent call last):
  File "E:\Maxime\fe_debug\fea.py", line 9, in fea_analysis
    mapdl = launch_mapdl(exec_file='C:/Program Files/ANSYS Inc/v212/ansys/bin/winx64/ANSYS212.exe', license_server_check=True)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1498, in launch_mapdl
    raise exception
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1488, in launch_mapdl
    **start_parm,
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 344, in __init__
    self._multi_connect()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 392, in _multi_connect
    timeout=attempt_timeout, set_no_abort=set_no_abort
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 533, in _connect
    self._set_no_abort()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\misc.py", line 283, in wrapper
    out = func(*args, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 666, in _set_no_abort
    self.nerr(abort=-1, mute=True)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\_commands\session\run_controls.py", line 423, in nerr
    return self.run(command, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl.py", line 2572, in run
    text = self._run(command, verbose=verbose, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 729, in _run
    response = self._send_command(cmd, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 131, in wrapper
    raise MapdlExitedError("MAPDL server connection terminated") from None
ansys.mapdl.core.errors.MapdlExitedError: MAPDL server connection terminated

and

Traceback (most recent call last):
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1488, in launch_mapdl
    **start_parm,
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 344, in __init__
    self._multi_connect()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 392, in _multi_connect
    timeout=attempt_timeout, set_no_abort=set_no_abort
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 533, in _connect
    self._set_no_abort()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\misc.py", line 283, in wrapper
    out = func(*args, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 666, in _set_no_abort
    self.nerr(abort=-1, mute=True)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\_commands\session\run_controls.py", line 423, in nerr
    return self.run(command, **kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl.py", line 2572, in run
    text = self._run(command, verbose=verbose, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 729, in _run
    response = self._send_command(cmd, mute=mute)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\errors.py", line 131, in wrapper
    raise MapdlExitedError("MAPDL server connection terminated") from None
ansys.mapdl.core.errors.MapdlExitedError: MAPDL server connection terminated

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Maxime\fe_debug\fea.py", line 9, in fea_analysis
    mapdl = launch_mapdl(exec_file='C:/Program Files/ANSYS Inc/v212/ansys/bin/winx64/ANSYS212.exe', license_server_check=True)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1496, in launch_mapdl
    lic_check.check()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\licensing.py", line 449, in check
    raise LicenseServerConnectionError("\n".join(self._license_file_msg))
ansys.mapdl.core.errors.LicenseServerConnectionError: Exceeded 10 seconds while waiting for C:\Users\mp2333\AppData\Local\Temp\.ansys\licdebug.FEAT_ANSYS.181.out to exist.

germa89 commented 2 years ago

I will have a look soon (ping me if you dont get news in several days)

maximepollet27 commented 2 years ago

Great, thanks!

maximepollet27 commented 2 years ago

Hi @germa89

In the meantime I have been trying to use the LocalMapdlPool described here: https://mapdldocs.pyansys.com/api/_autosummary/ansys.mapdl.core.pool.LocalMapdlPool.html#ansys.mapdl.core.pool.LocalMapdlPool

It is a lot more stable, but an issue similar to the ones I described above happened at some point (after more than 500 analyses).

Creating Pool: 100%|███████████████████████████████████████████████████████████████████| 15/15 [00:11<00:00,  1.30it/s]
MAPDL Running:  39%|███████████████████████                                    | 585/1500 [6:09:27<13:13:32, 52.04s/it]Exception in thread Instance 10:
Traceback (most recent call last):
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1488, in launch_mapdl
    **start_parm,
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 344, in __init__
    self._multi_connect()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\mapdl_grpc.py", line 407, in _multi_connect
    f"Unable to connect to MAPDL gRPC instance at {self._channel_str}"
OSError: Unable to connect to MAPDL gRPC instance at 127.0.0.1:50721

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mp2333\AppData\Local\Programs\Python\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Users\mp2333\AppData\Local\Programs\Python\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\pool.py", line 573, in _spawn_mapdl
    run_location=run_location, port=port, **self._spawn_kwargs
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\launcher.py", line 1496, in launch_mapdl
    lic_check.check()
  File "E:\Maxime\venv\ansys_analysis\lib\site-packages\ansys\mapdl\core\licensing.py", line 449, in check
    raise LicenseServerConnectionError("\n".join(self._license_file_msg))
ansys.mapdl.core.errors.LicenseServerConnectionError: Exceeded 10 seconds while waiting for C:\Users\mp2333\AppData\Local\Temp\.ansys\licdebug.FEAT_ANSYS.181.out to exist.

It seems like the two issues have a similar root. Hopefully, that's a useful hint. Thanks for your help.

ansys / pymapdl