`test_c_code` fails on NixOS

[balodja]

Packing the FMPy for NixOS at the moment. tests/test_c_code fails because of minor type mismatches in Dymola-generated FMU's sprintf. The log is here:

pytest logs

``` $ pytest ============================================================================================== test session starts ============================================================================================== platform linux -- Python 3.8.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 rootdir: /home/balodja/projects/nix/builds/source collected 62 items test_c_code.py FF [ 3%] test_command_line.py ... [ 8%] test_common_functions.py . [ 9%] test_cswrapper.py . [ 11%] test_cvode.py . [ 12%] test_examples.py ...s [ 19%] test_extracted_fmu.py . [ 20%] test_fmi3.py s [ 22%] test_fmu_container.py s [ 24%] test_fmu_info.py .... [ 30%] test_get_directional_derivative.py s [ 32%] test_get_start_values.py . [ 33%] test_input.py ..... [ 41%] test_jupyter_notebook.py . [ 43%] test_output_grid.py .. [ 46%] test_read_csv.py . [ 48%] test_reference_fmus.py ..... [ 56%] test_remoting.py ssss [ 62%] test_serialize_fmu_state.py . [ 64%] test_ssp.py sssssssssssssss [ 88%] test_start_value_units.py ssss [ 95%] test_type_definitions.py . [ 96%] test_validation.py .. [100%] =================================================================================================== FAILURES ==================================================================================================== _____________________________________________________________________________________________ CCodeTest.test_cmake ______________________________________________________________________________________________ self = def test_cmake(self): """ Create a CMake project """ from subprocess import check_call import shutil from fmpy.util import visual_c_versions try: # check if CMake is installed check_call(['cmake']) cmake_available = True except: cmake_available = False for fmu in self.fmus: filename = download_file(self.url + fmu) model_name, _ = os.path.splitext(filename) # clean up if os.path.isdir(model_name): shutil.rmtree(model_name) # create the CMake project create_cmake_project(filename, model_name) if not cmake_available: continue # skip compilation # generate the build system cmake_args = ['cmake', '.'] vc_versions = visual_c_versions() if os.name == 'nt': if 160 in vc_versions: cmake_args += ['-G', 'Visual Studio 16 2019', '-A', 'x64'] elif 150 in vc_versions: cmake_args += ['-G', 'Visual Studio 15 2017 Win64'] elif 140 in vc_versions: cmake_args += ['-G', 'Visual Studio 14 2015 Win64'] elif 120 in vc_versions: cmake_args += ['-G', 'Visual Studio 12 2013 Win64'] elif 110 in vc_versions: cmake_args += ['-G', 'Visual Studio 11 2012 Win64'] check_call(args=cmake_args, cwd=model_name) # run the build system > check_call(args=['cmake', '--build', '.'], cwd=model_name) test_c_code.py:81: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ popenargs = (), kwargs = {'args': ['cmake', '--build', '.'], 'cwd': 'IntegerNetwork1'}, retcode = 2, cmd = ['cmake', '--build', '.'] def check_call(*popenargs, **kwargs): """Run command with arguments. Wait for command to complete. If the exit code was zero then return, otherwise raise CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute. The arguments are the same as for the call function. Example: check_call(["ls", "-l"]) """ retcode = call(*popenargs, **kwargs) if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] > raise CalledProcessError(retcode, cmd) E subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 2. /nix/store/zv167zw7gwr9h0gpkfcikdfqnscrd0q6-python3-3.8.9/lib/python3.8/subprocess.py:364: CalledProcessError --------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------- Usage cmake [options] cmake [options] cmake [options] -S -B Specify a source directory to (re-)generate a build system for it in the current working directory. Specify an existing build directory to re-generate its build system. Run 'cmake --help' for more information. Downloading https://github.com/modelica/fmi-cross-check/raw/master/fmus/2.0/cs/c-code/MapleSim/2016.2/Rectifier/Rectifier.fmu -- The C compiler identification is GNU 10.3.0 -- The CXX compiler identification is GNU 10.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /nix/store/74kv08wjf06ifgk4dxrnra4qhzr5s1w4-gcc-wrapper-10.3.0/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /nix/store/74kv08wjf06ifgk4dxrnra4qhzr5s1w4-gcc-wrapper-10.3.0/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /home/balodja/projects/nix/builds/source/tests/Rectifier Scanning dependencies of target Rectifier [ 50%] Building C object CMakeFiles/Rectifier.dir/sources/Rectifier.c.o [100%] Linking C shared library binaries/linux64/Rectifier.so binaries binaries/linux64 binaries/linux64/Rectifier.so binaries/win32 binaries/win32/Rectifier.dll binaries/win32/Rectifier.lib binaries/win32 binaries/win32/Rectifier.dll binaries/win32/Rectifier.lib binaries/win32/Rectifier.dll binaries/win32/Rectifier.lib data data/simRecord.m data/simRecord.m documentation documentation/_main.html documentation/_main.html modelDescription.xml sources sources/fmuTemplate.h sources/Rectifier.c sources/fmuTemplate.h sources/Rectifier.c binaries binaries/linux64 binaries/linux64/Rectifier.so binaries/win32 binaries/win32/Rectifier.dll binaries/win32/Rectifier.lib [100%] Built target Rectifier Downloading https://github.com/modelica/fmi-cross-check/raw/master/fmus/2.0/cs/c-code/Dymola/2017/IntegerNetwork1/IntegerNetwork1.fmu -- The C compiler identification is GNU 10.3.0 -- The CXX compiler identification is GNU 10.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /nix/store/74kv08wjf06ifgk4dxrnra4qhzr5s1w4-gcc-wrapper-10.3.0/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /nix/store/74kv08wjf06ifgk4dxrnra4qhzr5s1w4-gcc-wrapper-10.3.0/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1 Scanning dependencies of target IntegerNetwork1 [ 50%] Building C object CMakeFiles/IntegerNetwork1.dir/sources/all.c.o --------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------- In file included from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/dsmodel.c:4475, from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/all.c:22: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/dymf2c.c: In function ‘DymosimMessageIntSev_’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/dymf2c.c:157:34: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘int’ [-Wformat=] 157 | sprintf(&textline[ic], " %ld", *i); | ~~^ ~~ | | | | | int | long int | %d In file included from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/dsmodel.c:4476, from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/all.c:22: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c: In function ‘Integer2String’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c:3579:20: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 5 has type ‘int’ [-Wformat=] 3579 | sprintf(buf,"%*.*ld",(int)minwidth,(int)precision,(int)x); | ~~~~~^ ~~~~~~ | | | | long int int | %*.*d /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c: In function ‘Integer2String2’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c:3600:33: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘int’ [-Wformat=] 3600 | sprintf(buf,leftJustified?"%-*ld":"%*ld",(int)minwidth,(int)x); | ~~~~^ ~~~~~~ | | | | long int int | %-*d /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c:3600:40: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘int’ [-Wformat=] 3600 | sprintf(buf,leftJustified?"%-*ld":"%*ld",(int)minwidth,(int)x); | ~~~^ ~~~~~~ | | | | long int int | %*d In file included from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/matrixop.c:3746, from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/dsmodel.c:4476, from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/all.c:22: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/localeless.cpp: In function ‘sscanfCx2_underscore’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/localeless.cpp:315:25: warning: format ‘%lx’ expects argument of type ‘long unsigned int *’, but argument 3 has type ‘unsigned int *’ [-Wformat=] 315 | return sscanf(line, "%lx_%lx", a,b); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %x /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/localeless.cpp:315:29: warning: format ‘%lx’ expects argument of type ‘long unsigned int *’, but argument 4 has type ‘unsigned int *’ [-Wformat=] 315 | return sscanf(line, "%lx_%lx", a,b); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %x /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/localeless.cpp: In function ‘sscanfClu’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/localeless.cpp:326:25: warning: format ‘%lu’ expects argument of type ‘long unsigned int *’, but argument 3 has type ‘unsigned int *’ [-Wformat=] 326 | return sscanf(line, "%lu", a); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %u In file included from /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/all.c:40: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/cvode.c: In function ‘CVProcessError’: /home/balodja/projects/nix/builds/source/tests/IntegerNetwork1/sources/cvode.c:4175:5: error: format not a string literal and no format arguments [-Werror=format-security] 4175 | fprintf(stderr, msgfmt); | ^~~~~~~ cc1: some warnings being treated as errors make[2]: *** [CMakeFiles/IntegerNetwork1.dir/build.make:82: CMakeFiles/IntegerNetwork1.dir/sources/all.c.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/IntegerNetwork1.dir/all] Error 2 make: *** [Makefile:103: all] Error 2 ____________________________________________________________________________________________ CCodeTest.test_compile _____________________________________________________________________________________________ self = def test_compile(self): """ Compile the platform binary """ for fmu in self.fmus: filename = download_file(self.url + fmu) > compile_platform_binary(filename) test_c_code.py:26: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /nix/store/zk6qh4jspm8pi2dm40rprbgmb7ra6kki-python3-3.8.9-env/lib/python3.8/site-packages/fmpy/util.py:865: in compile_platform_binary binary = compile_dll(model_description=model_description, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ model_description = ModelDescription(fmiVersion='2.0', modelName='IntegerNetwork1', coSimulation=CoSimulation(modelIdentifier='IntegerNetwork1'), modelExchange=None, scheduledExecution=None) sources_dir = '/run/user/1000/tmpsk0htif3/sources', compiler = 'gcc', target_platform = 'linux64' def compile_dll(model_description, sources_dir, compiler=None, target_platform=None): """ Compile the shared library Parameters: sources_dir directory that contains the FMU's source code compiler compiler to use (None: use Visual C on Windows, GCC otherwise) platform platform to compile for the binary for """ from . import platform, system import subprocess if target_platform is None: target_platform = platform if model_description.fmiVersion == '1.0': raise Exception("FMI 1.0 source FMUs are currently not supported") if compiler is None: if target_platform in ['win32', 'win64', 'x86_64-windows']: compiler = 'vc' else: compiler = 'gcc' include_dir = os.path.join(os.path.dirname(__file__), 'c-code') preprocessor_definitions = [] source_files = [] if len(model_description.buildConfigurations) == 0: raise Exception("No build configuration found.") build_configuration = model_description.buildConfigurations[0] if len(build_configuration.sourceFileSets) > 1: raise Exception("More than one SourceFileSet is not supported.") source_file_set = build_configuration.sourceFileSets[0] source_files += source_file_set.sourceFiles for definition in source_file_set.preprocessorDefinitions: literal = definition.name if definition.value is not None: literal += '=' + definition.value preprocessor_definitions.append(literal) if len(source_files) == 0: raise Exception("No source files specified in the model description.") print('Compiling platform binary...') sources = ' '.join(source_files) model_identifier = build_configuration.modelIdentifier if compiler == 'vc': target = model_identifier + '.dll' vc_versions = visual_c_versions() toolset = 'x86_amd64' if target_platform in ['win64', 'x86_64-windows'] else 'x86' definitions = ' '.join(f' /D{d}' for d in preprocessor_definitions) if len(vc_versions) == 0: raise Exception("No VisualStudio found") # use the latest version vc_version = vc_versions[-1] if vc_version < 150: command = rf'call "%%VS{vc_version}COMNTOOLS%%\..\..\VC\vcvarsall.bat" {toolset}' else: installation_path = visual_studio_installation_path() command = rf'call "{installation_path}\VC\Auxiliary\Build\vcvarsall.bat" {toolset}' command += f' && cl /LD /I. /I"{include_dir}" {definitions} /Fe{model_identifier} shlwapi.lib {sources}' elif compiler == 'gcc': definitions = ' '.join(f' -D{d}' for d in preprocessor_definitions) if target_platform in ['linux64', 'x86_64-linux']: cc = 'gcc' target = model_identifier + '.so' if system == 'windows': cc = 'wsl ' + cc output = subprocess.check_output(f"wsl wslpath -a '{include_dir}'") include_dir = output.decode('utf-8').strip() command = f'{cc} -c -I. -I{include_dir} -fPIC {definitions} {sources}' command += f' && {cc} -static-libgcc -shared -o{target} *.o -lm' elif target_platform in ['darwin64', 'x86_64-darwin']: target = model_identifier + '.dylib' command = f'gcc -c -I. -I{include_dir} {definitions} {sources}' command += f' && gcc -shared -o{target} *.o -lm' else: raise Exception("Unsupported target platform for selected compiler: '%s'" % compiler) else: raise Exception("Unsupported compiler: '%s'" % compiler) cur_dir = os.getcwd() os.chdir(sources_dir) status = os.system(command) os.chdir(cur_dir) print(sources_dir) print(command) dll_path = os.path.join(sources_dir, target) if status != 0 or not os.path.isfile(dll_path): > raise Exception('Failed to compile shared library') E Exception: Failed to compile shared library /nix/store/zk6qh4jspm8pi2dm40rprbgmb7ra6kki-python3-3.8.9-env/lib/python3.8/site-packages/fmpy/util.py:841: Exception --------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------- Downloading https://github.com/modelica/fmi-cross-check/raw/master/fmus/2.0/cs/c-code/MapleSim/2016.2/Rectifier/Rectifier.fmu Compiling platform binary... /run/user/1000/tmp4koozdtu/sources gcc -c -I. -I/nix/store/zk6qh4jspm8pi2dm40rprbgmb7ra6kki-python3-3.8.9-env/lib/python3.8/site-packages/fmpy/c-code -fPIC Rectifier.c && gcc -static-libgcc -shared -oRectifier.so *.o -lm Downloading https://github.com/modelica/fmi-cross-check/raw/master/fmus/2.0/cs/c-code/Dymola/2017/IntegerNetwork1/IntegerNetwork1.fmu Compiling platform binary... /run/user/1000/tmpsk0htif3/sources gcc -c -I. -I/nix/store/zk6qh4jspm8pi2dm40rprbgmb7ra6kki-python3-3.8.9-env/lib/python3.8/site-packages/fmpy/c-code -fPIC all.c && gcc -static-libgcc -shared -oIntegerNetwork1.so *.o -lm --------------------------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------------------------- In file included from dsmodel.c:4475, from all.c:22: dymf2c.c: In function ‘DymosimMessageIntSev_’: dymf2c.c:157:34: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘int’ [-Wformat=] 157 | sprintf(&textline[ic], " %ld", *i); | ~~^ ~~ | | | | | int | long int | %d In file included from dsmodel.c:4476, from all.c:22: matrixop.c: In function ‘Integer2String’: matrixop.c:3579:20: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 5 has type ‘int’ [-Wformat=] 3579 | sprintf(buf,"%*.*ld",(int)minwidth,(int)precision,(int)x); | ~~~~~^ ~~~~~~ | | | | long int int | %*.*d matrixop.c: In function ‘Integer2String2’: matrixop.c:3600:33: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘int’ [-Wformat=] 3600 | sprintf(buf,leftJustified?"%-*ld":"%*ld",(int)minwidth,(int)x); | ~~~~^ ~~~~~~ | | | | long int int | %-*d matrixop.c:3600:40: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘int’ [-Wformat=] 3600 | sprintf(buf,leftJustified?"%-*ld":"%*ld",(int)minwidth,(int)x); | ~~~^ ~~~~~~ | | | | long int int | %*d In file included from matrixop.c:3746, from dsmodel.c:4476, from all.c:22: localeless.cpp: In function ‘sscanfCx2_underscore’: localeless.cpp:315:25: warning: format ‘%lx’ expects argument of type ‘long unsigned int *’, but argument 3 has type ‘unsigned int *’ [-Wformat=] 315 | return sscanf(line, "%lx_%lx", a,b); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %x localeless.cpp:315:29: warning: format ‘%lx’ expects argument of type ‘long unsigned int *’, but argument 4 has type ‘unsigned int *’ [-Wformat=] 315 | return sscanf(line, "%lx_%lx", a,b); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %x localeless.cpp: In function ‘sscanfClu’: localeless.cpp:326:25: warning: format ‘%lu’ expects argument of type ‘long unsigned int *’, but argument 3 has type ‘unsigned int *’ [-Wformat=] 326 | return sscanf(line, "%lu", a); | ~~^ ~ | | | | | unsigned int * | long unsigned int * | %u In file included from all.c:40: cvode.c: In function ‘CVProcessError’: cvode.c:4175:5: error: format not a string literal and no format arguments [-Werror=format-security] 4175 | fprintf(stderr, msgfmt); | ^~~~~~~ cc1: some warnings being treated as errors =============================================================================================== warnings summary ================================================================================================ tests/test_input.py::InputTest::test_discrete_signal_events /home/balodja/projects/nix/builds/source/tests/test_input.py:78: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations dtype=np.dtype([('time', np.float64), ('x', np.int)])) -- Docs: https://docs.pytest.org/en/stable/warnings.html ============================================================================================ short test summary info ============================================================================================ FAILED test_c_code.py::CCodeTest::test_cmake - subprocess.CalledProcessError: Command '['cmake', '--build', '.']' returned non-zero exit status 2. FAILED test_c_code.py::CCodeTest::test_compile - Exception: Failed to compile shared library ============================================================================= 2 failed, 33 passed, 27 skipped, 1 warning in 40.33s ============================================================================== ```

system details

``` > nix-shell -p nix-info --run "nix-info -m" - system: `"x86_64-linux"` - host os: `Linux 5.10.48, NixOS, 21.05.1493.1f4e50f03c5 (Okapi)` - multi-user?: `yes` - sandbox: `yes` - version: `nix-env (Nix) 2.3.12` - channels(root): `"nixos-21.05.1493.1f4e50f03c5"` - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos` ```

Should I present more technical information?

P.S. I expect the issue to remain on other Linux'es as well.

[d94pn]

The source code seems to be quite old, for Dymola 2017 I assume. The file Cvode.c was updated from Sundials soon after that, so you could either try to fetch and merge in the updated function below, or simply comment out the failing line (should be non-critical):

SUNDIALS_EXPORT void cvProcessError(CVodeMem cv_mem, int error_code, const char module, const char fname, const char *msgfmt, ...) { va_list ap; char msg[256];

/ Initialize the argument pointer variable (msgfmt is the last required argument to cvProcessError) /

va_start(ap, msgfmt);

/ Compose the message /

vsprintf(msg, msgfmt, ap);

if (cv_mem == NULL) { / We write to stderr /

ifndef NO_FPRINTF_OUTPUT

fprintf(stderr, "\n[%s ERROR]  %s\n  ", module, fname);
fprintf(stderr, "%s\n\n", msg);

endif

} else { / We can call ehfun / ehfun(error_code, module, fname, msg, eh_data); }

/ Finalize argument processing / va_end(ap);

return; }