Simulation with Resultfile or Flags Fail to execute

[NilsDressler]

Description

When calling OMPython.simulaute(resultfile='/home/user/modelica/testfile0.mat') with a result file (and most likely also with other flags, the following log message is shown

Failed to simulate simulation: [Errno 2] No such file or directory: '/tmp/tmp37u5_obx/Package.Simulation -r=/home/user/modelica/testfile0.mat'

Steps to reproduce

Minimal Example that fails

from os.path import expanduser
from OMPython import ModelicaSystem

home_dir = expanduser("~")

model_path = home_dir + '/modelica/resultfilebug/'
model_file_name = 'BouncingBall.mo'
model_name = 'BouncingBall' 
mod = ModelicaSystem(model_path + model_file_name, model_name )

mod.buildModel()

result_file_path = home_dir + '/modelica/resultfilebug/resultfiles/'
result_file_name = 'resultfile_0.mat'

mod.simulate(resultfile=result_file_path+result_file_name)

Throws followingexception:

Exception has occurred: FileNotFoundError (note: full exception trace is shown but execution is paused at: _run_module_as_main) [Errno 2] No such file or directory: '/tmp/tmpj5k_1p6k/BouncingBall -r=~/modelica/resultfilebug/resultfiles/resultfile_0.mat'

Even though the file exists in my file system

ls /tmp/tmpqmbz5d57/BouncingBall
/tmp/tmpqmbz5d57/BouncingBall

Running

user@machine:~$ /tmp/tmpqmbz5d57/BouncingBall -r=/home/user/modelica/resultfilebug/resultfiles/resultfile_0.mat

LOG_ASSERT | debug | simulation_input_xml.c: Error: can not read file BouncingBall_init.xml as setup file to the generated simulation code. Execution failed!

While

user@machine:/tmp/tmpq5rla9at$ ./BouncingBall -r=/home/user/modelica/resultfilebug/resultfiles/resultfile_0.mat

LOG_SUCCESS | info | The initialization finished successfully without homotopy method. LOG_SUCCESS | info | The simulation finished successfully.

Expected behavior

The model should just build and return the following log messages

LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.

Screenshots

If applicable, add screenshots to help explain your problem.

Version and OS

• Python Version: Python 3.10.12
• OMPython Version: 3.5.1
• OpenModelica Version: 1.22.3
• OS: Windows 11 64 bit - Code run from WSL :Ubuntu 22.04.4 LTS

Additional context

When digging into the problem and debugging I came across this:

The error message suggests that the subprocess is unable to find the executable file /tmp/tmpqmbz5d57/BouncingBall -r=/home/user/modelica/testfile0.mat

The issue here is that the provided cmd is being interpreted as a single command with spaces in its name. Instead, you should separate the executable path and the command-line arguments into separate elements of the cmd list.

So the problem is created within

p = subprocess.Popen(cmd, env=my_env)

A solution to the problem is how cmd is created. If it is a list with

• getExeFile
• override
• csvinput
• r
• simflags

as seperate list entries, it works with my system configuration

[NilsDressler]

I also found a working solution that made the code work for my system.

Before I make a pull request I'm wondering the following. What format are the simFlags expected to be ? A long string containing the flags like mod.simulate(simflags='-flag1 -flag2 -...') or a list containing the flags as sting mod.simulate(simflags=['-flag1', '-flag2' '-...'])

I couldn't really find an answer to that in the documentation. I can make a solution that works for both cases, but that might be unnecessary complicated if just on of them is a valid input.

[InigoGastesi]

Hi, I have the same problem, I have found out that OMPython creates a command like this

cmd = '/tmp/tmpltwurgv3/variable_torque -overrideFile=/tmp/tmpltwurgv3/variable_torque_override.txt -r=variable_torque_size_50.mat'  
subprocess.Popen(cmd)

But doing the following test works nice

import subprocess
cmd = [
    "/tmp/tmpltwurgv3/variable_torque",
    "-overrideFile=/tmp/tmpltwurgv3/variable_torque_override.txt",
    "-r=variable_torque_size_50.mat"
]
p = subprocess.Popen(cmd)

So I suppose that the format of simFlags are like this mod.simulate(simflags=['-flag1', '-flag2' '-...'])

[j-emils]

I have the same problem.

[NilsDressler]

@InigoGastesi Yes this was also the solution i found and how i fixed i in the commit i made. If you give the subprocess a python list containing the path to the executable at index 0 and then all the flags in the subsequent indices, it works. Compared to the current implementation you only have to remove the initial blank space from the flags.

[InigoGastesi]

@NilsDressler I using your commit now in my project, thank you

[arun3688]

@NilsDressler the issue is fixed with this commit fa56e30212ca194133060a47280bf8de00b5c8e6 and thanks for @NilsDressler for fixing the issue

[InigoGastesi]

Hi @arun3688, I think that this issue is not fixed in that commit. In that commit the only think that is fixed is the problem with the subprocess

[arun3688]

@InigoGastesi Yes that is true, but still the issue fixes the subprocess problem in linux. @NilsDressler It is not allowed to provide result file path in the simulate() API, you can only provide the name for the result file and the result file will always be generated in the temp directory. see the docs here https://openmodelica.org/doc/OpenModelicaUsersGuide/1.22/ompython.html#simulation

The model can be simulated using the simulate API in the following ways,

1. without any arguments
2. resultfile (keyword argument) - (only filename is allowed and not the location)
3. simflags (keyword argument) - runtime simulationflags supported by OpenModelica

An example of how to use the simflags is also provided in the documentation

[InigoGastesi]

@arun3688 I'm checking your last comment and in the following code

filename = f'{model_name}_size_{size}'
print(filename) # voltage_torqueVariable_size_50
mod.simulate(resultfile=f'{filename}.mat') #Para simular el modelo, empleo el método ".simulate()"

I have this error

[Errno 2] No such file or directory: '/tmp/tmp9y_av4ml/voltage_torqueVariable -overrideFile=/tmp/tmp9y_av4ml/voltage_torqueVariable_override.txt -r=voltage_torqueVariable_size_50.mat'

I checked it and the file exists. Also, I tried do the same without .mat extension and the same problem appears. As you can see in the print there is only filename

[arun3688]

@InigoGastesi what happens if you directly use mod.simulate(resultfile="voltage_torqueVariable_size_50.mat")

[InigoGastesi]

Same error :(

[arun3688]

@InigoGastesi strange what happens if you just run mod.simulate()

[InigoGastesi]

Same error again

[arun3688]

@InigoGastesi can you show your script with the model attached

[InigoGastesi]

@arun3688 Here is the code that load the model

sizes = [50, 150, 500, 2000]
model_name = "voltage_torqueVariable"
model_path_full = model_path + model_name + ".mo"

model_path_full = "/home/igastesi/Documentos/HEGAPS/dlsm-trainer/OM_models/voltage_torqueVariable.mo"

exec_times = []
for s in sizes:
    torque = get_random_torque(s, step_size, std_dev=0.8)
    voltaje = get_random_voltaje(s, step_size, std_dev=0.8)
    model_to_file(torque, voltaje, model_path_full)
    omc = OMCSessionZMQ()
    mod = ModelicaSystem(model_path_full, model_name)

    # Configuración simulación
    mod.setSimulationOptions(["startTime=0.0",f'stopTime={s}',f'stepSize={step_size}',"tolerance=1e-06"])
    start_time = time.time()
    simulate(model_name, s, mod, plot_values, True)
    end_time = time.time()

    exec_times.append(end_time - start_time)

Code of simulatefunction

def simulate(model_name, size, mod, export_params, export_to_file = False):
    # Simulación
    filename = f'{model_name}_size_{size}'

    #mod.setParameters(f"{variable}={v}")
    mod.simulate(resultfile=f'{filename}.mat') #Para simular el modelo, empleo el método ".simulate()"

    data = {}
    data["time"] = mod.getSolutions(["time"],resultfile=f'{filename}.mat')[0]
    print(data)
    # Configure Plot
    print(f"size = {size}")
    plt.figure(figsize=(20, 6))
    plt.grid(True)
    #plt.yscale('log')

    # Plot parameters
    for param in export_params:
        # Get data
        data[param] = mod.getSolutions(param, resultfile=f'{filename}.mat')[0]
        # Plot data
        plt.plot(data["time"], data[param], label=param)

    plt.legend()
    plt.show()

[NilsDressler]

@arun3688, thanks for the explanation regarding the syntax for the flags., I have missed that. In that case the proposed solution in e24c067 could be simplified further.

And i tried just with running with

resultfile (keyword argument) - (only filename is allowed and not the location)

in mind. And the problem still remains the same

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpyuvymk0r/BouncingBall -r=/home/dressler/modelica/resultfilebug/resultfiles/resultfile_0.mat -logFormat=xml -nlsInfo'

[arun3688]

@InigoGastesi @NilsDressler I can reproduce the issue, I will fix it today.

[arun3688]

@NilsDressler @InigoGastesi the issue is fixed with this commit 69a502ea2e2c85d148e540015a69253513e24375
Actually it is possible to provide result file path in the as said in the documentation simulate(resultfile="C:/resultFile/test.mat") API, see the example below, update your package using python -m pip install -U https://github.com/OpenModelica/OMPython/archive/master.zip

An example usage is

Usage

>>> from OMPython import ModelicaSystem
>>> mod = ModelicaSystem("C:/BouncingBall.mo", "BouncingBall")
>>> mod.simulate(resultfile="C:/OPENMODELICAGIT/BouncingBall_res.mat") // path provided by user
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
>>> mod.resultfile
'C:/OPENMODELICAGIT/BouncingBall_res.mat'
>>> mod.getSolutions()
('der(h)', 'der(v)', 'e', 'flying', 'foo', 'g', 'h', 'impact', 'time', 'v', 'v_new')
>>> mod.simulate(resultfile="test.mat") // providing only the file name 
LOG_SUCCESS       | info    | The initialization finished successfully without homotopy method.
LOG_SUCCESS       | info    | The simulation finished successfully.
>>> mod.getSolutions()
('der(h)', 'der(v)', 'e', 'flying', 'foo', 'g', 'h', 'impact', 'time', 'v', 'v_new')

[casella]

@NilsDressler please check with the nightly build and close the ticket if it works for you. Thanks!

[j-emils]

I'm still experiencing issues with this:

self = <Popen: returncode: 255 args: '/tmp/tmpu48lhhg0/myTestPackage...>
args = ['/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt']
executable = b'/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt'
preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = None, creationflags = 0, shell = False, p2cread = -1
p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1, errwrite = -1
restore_signals = True, gid = None, gids = None, uid = None, umask = -1
start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""

        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)

        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable

        if executable is None:
            executable = args[0]

        sys.audit("subprocess.Popen", executable, args, cwd, env)

        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return

        orig_executable = executable

        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()

                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)

            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)

            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)

        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass

            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1 -overrideFile=/tmp/tmpu48lhhg0/myTestPackage.systems.myModel1_override.txt'

[arun3688]

@j-emils I can see that you are directly using the subprocess to execute the executable and the error message says it cannot find the executable in that location, also i see you are not using ModelicaSystem constructor, It would be good if you can provide a minimal example to reproduce the issue, I suggest you to change to the executable directory before using the subprocess command

[j-emils]

@arun3688, we are using the ModelicaSystem constructor, it's just that we have created a wrapper around it. Maybe there's a dependency on the OpenModelica version? It's failing with version 1.22.0.

[arun3688]

@j-emils please uninstall your OMPython package and install it with the latest master,

pip uninstall OMPython and then pip install git+https://github.com/OpenModelica/OMPython.git/@master

[j-emils]

@arun3688, I'm still getting the same error. If I change the call with subprocess to use a list instead of a string it works, analogous to the proposal from @NilsDressler . Will try to create an example for you to test.

[NilsDressler]

@casella If the latest nightly referred to https://github.com/OpenModelica/OMPython/commit/891528c8009348881019c81d8d7420c1a8d3ee14 then it still does not work for my minimal example. I will make a pull request for the solution provided by me

[casella]

@arun3688 can you please follow this up? Thanks!

[arun3688]

@casella the issue is fixed with the this commit https://github.com/OpenModelica/OMPython/pull/214

[casella]

Great, thanks @arun3688!