OAT installation failing with python 3.9 and python 3.10

[Prateek1410]

Greetings!

During installation of OAT it fails at building wheel for oxDNA analysis tools. The following error comes up.

Building wheel for oxDNA-analysis-tools (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for oxDNA-analysis-tools (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [93 lines of output]
      running bdist_wheel
      running build
      running build_py
      running egg_info
      writing src/oxDNA_analysis_tools.egg-info/PKG-INFO
      writing dependency_links to src/oxDNA_analysis_tools.egg-info/dependency_links.txt
      writing entry points to src/oxDNA_analysis_tools.egg-info/entry_points.txt
      writing requirements to src/oxDNA_analysis_tools.egg-info/requires.txt
      writing top-level names to src/oxDNA_analysis_tools.egg-info/top_level.txt
      reading manifest file 'src/oxDNA_analysis_tools.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      writing manifest file 'src/oxDNA_analysis_tools.egg-info/SOURCES.txt'
      /tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'oxDNA_analysis_tools.cython_utils' as data is deprecated, please list it in `packages`.
          !!

          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'oxDNA_analysis_tools.cython_utils' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'oxDNA_analysis_tools.cython_utils' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'oxDNA_analysis_tools.cython_utils' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.

      !!

        check.warn(importable)
      copying src/oxDNA_analysis_tools/cython_utils/get_confs.c -> build/lib.linux-x86_64-cpython-39/oxDNA_analysis_tools/cython_utils
      running build_ext
      building 'oxDNA_analysis_tools.UTILS.get_confs' extension
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/numpy/core/include -I/mnt/c/oxDNA/venv_oat/include -I/usr/include/python3.9 -c src/oxDNA_analysis_tools/cython_utils/get_confs.c -o build/temp.linux-x86_64-cpython-39/src/oxDNA_analysis_tools/cython_utils/get_confs.o
      In file included from /tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1948,
                       from /tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                       from /tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:5,
                       from src/oxDNA_analysis_tools/cython_utils/get_confs.c:759:
      /tmp/pip-build-env-jl32vqw7/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
         17 | #warning "Using deprecated NumPy API, disable it with " \
            |  ^~~~~~~
      src/oxDNA_analysis_tools/cython_utils/get_confs.c: In function ‘__pyx_pf_20oxDNA_analysis_tools_5UTILS_9get_confs_cget_confs’:
      src/oxDNA_analysis_tools/cython_utils/get_confs.c:2405:16: warning: passing argument 1 of ‘fread’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       2405 |   (void)(fread(__pyx_v_chunk, __pyx_v_chunk_size, 1, __pyx_v_traj_file));
            |                ^~~~~~~~~~~~~
      In file included from /usr/include/stdio.h:867,
                       from /usr/include/python3.9/Python.h:25,
                       from src/oxDNA_analysis_tools/cython_utils/get_confs.c:19:
      /usr/include/x86_64-linux-gnu/bits/stdio2.h:284:25: note: expected ‘void * restrict’ but argument is of type ‘const char *’
        284 | fread (void *__restrict __ptr, size_t __size, size_t __n,
            |        ~~~~~~~~~~~~~~~~~^~~~~
      src/oxDNA_analysis_tools/cython_utils/get_confs.c:2490:77: warning: passing argument 1 of ‘__pyx_f_20oxDNA_analysis_tools_5UTILS_9get_confs_parse_conf’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       2490 |     __pyx_t_4 = __pyx_f_20oxDNA_analysis_tools_5UTILS_9get_confs_parse_conf(__pyx_v_chunk, (__pyx_v_conf_starts[__pyx_t_9]), (__pyx_v_sizes[__pyx_t_13]), __pyx_v_nbases); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 50, __pyx_L1_error)
            |                                                                             ^~~~~~~~~~~~~
      src/oxDNA_analysis_tools/cython_utils/get_confs.c:1780:18: note: expected ‘char *’ but argument is of type ‘const char *’
       1780 | static PyObject *__pyx_f_20oxDNA_analysis_tools_5UTILS_9get_confs_parse_conf(char *, int, int, int); /*proto*/
            |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      src/oxDNA_analysis_tools/cython_utils/get_confs.c:2530:8: warning: passing argument 1 of ‘free’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       2530 |   free(__pyx_v_chunk);
            |        ^~~~~~~~~~~~~
      In file included from /usr/include/python3.9/Python.h:34,
                       from src/oxDNA_analysis_tools/cython_utils/get_confs.c:19:
      /usr/include/stdlib.h:565:25: note: expected ‘void *’ but argument is of type ‘const char *’
        565 | extern void free (void *__ptr) __THROW;
            |                   ~~~~~~^~~~~
      src/oxDNA_analysis_tools/cython_utils/get_confs.c: In function ‘__pyx_f_20oxDNA_analysis_tools_5UTILS_9get_confs_parse_conf’:
      src/oxDNA_analysis_tools/cython_utils/get_confs.c:2847:24: warning: passing argument 1 of ‘strtok’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       2847 |   __pyx_v_ptr = strtok(__pyx_v_ptr, ((char const *)"t = "));
            |                        ^~~~~~~~~~~
      In file included from /usr/include/python3.9/Python.h:30,
                       from src/oxDNA_analysis_tools/cython_utils/get_confs.c:19:
      /usr/include/string.h:336:14: note: expected ‘char * restrict’ but argument is of type ‘const char *’
        336 | extern char *strtok (char *__restrict __s, const char *__restrict __delim)
            |              ^~~~~~
      x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-39/src/oxDNA_analysis_tools/cython_utils/get_confs.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-39/oxDNA_analysis_tools/UTILS/get_confs.cpython-39-x86_64-linux-gnu.so
      installing to build/bdist.linux-x86_64/wheel
      running install
      running install_lib
      copying build/lib.linux-x86_64-cpython-39/oxDNA_analysis_tools/cython_utils/get_confs.c -> build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools/cython_utils
      copying build/lib.linux-x86_64-cpython-39/oxDNA_analysis_tools/UTILS/get_confs.cpython-39-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools/UTILS
      running install_egg_info
      removing 'build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools-2.0.3-py3.9.egg-info' (and everything under it)
      Copying src/oxDNA_analysis_tools.egg-info to build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools-2.0.3-py3.9.egg-info
      running install_scripts
      error: [('build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools-2.0.3-py3.9.egg-info', 'build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools-2.0.3.dist-info', "[Errno 13] Permission denied: 'build/bdist.linux-x86_64/wheel/oxDNA_analysis_tools-2.0.3.dist-info'")]
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for oxDNA-analysis-tools
Failed to build oxDNA-analysis-tools
ERROR: Could not build wheels for oxDNA-analysis-tools, which is required to install pyproject.toml-based projects

Interestingly, OAT gets installed without any issue when I use python 3.8. Could you please help me with this? I posted here since I thought after the updates, all oxDNA related queries go here. Thank you for your time and consideration.

Warm Regards Prateek

[Prateek1410]

Dear all,

Update: I managed to install OAT after giving full control (permissions) to authenticated users in the security panel of the properties of the oxDNA folder on my PC, using python 3.9. Running oat config shows everything as fine except it's unable to find the package oxpy. This is despite the fact that my oxDNA installation had run smoothly and that make test_oxpy shows all tests to have been passed. This shouldn't affect the running of the scripts since oxpy is required only if I do analysis from python, right?

Warm Regards Prateek

[ErikPoppleton]

Hi Prateek,

It is likely that the oxpy binary installed itself somewhere off the PYTHONPATH. I believe it defaults to ~/.local/bin/ which is not a place Python checks. You either need to add it to your PYTHONPATH or move oxpy to a directory where Python will look. There is more info on installation issues here and in the last section of this.

As far as what it's used for, you need oxpy for any oat analysis which involves calculating the oxDNA energy function. This includes output_bonds, bond_analysis and duplex_finder

[Prateek1410]

Hi Erik,

The issue got resolved! The problem was I had compiled oxDNA using python3.8 while I had separately installed OAT and oxpy using python3.9. The error was:

ImportError: Python version mismatch: module was compiled for version 3.8, while the interpreter is running version 3.9.

So, I uninstalled OAT and deleted oxDNA and started from the beginning this time passing DPYTHON_INCLUDE_DIRS=/usr/include/python3.9 and -DPYTHON_EXECUTABLE=/usr/bin/python3.9 as cmake options as suggested by you all.

I ran all the tests listed on the page and all of them were passed except tests.sh. When I ran it using bash command, I got the following error:

Testing align...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing backbone_flexibility...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing bond_analysis...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing mean and deviations...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing centroid with indexing...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing contact_map...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing distance and clustering (this one takes a while because of the plot)...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing duplex_finder...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing duplex_angle_plotter...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing generate_force...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing minify...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing multidimensional_scaling_mean...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing output_bonds...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing pca
SyntaxError: invalid syntax
AN ERROR OCCURED
Testing subset_trajectory...
SyntaxError: invalid syntax
AN ERROR OCCURED

Testing superimpose...
SyntaxError: invalid syntax
AN ERROR OCCURED

What could this be? Once again, thanks a lot for your help!

Warm Regards Prateek

[RodenLuo]

Hi,

My best guess is you are invoking analysis/tests/test.sh in a shell environment that uses python2 as the default python interpreter.

By typing cat analysis/tests/test.sh, you can see a handful of commands like this one:

python ../src/oxDNA_analysis_tools/align.py minitraj.dat aligned.dat 2>&1 >/dev/null

You can cd analysis/tests and then run the command above. You should see the full error message.

You may be able to bypass this error using the following command.

/usr/bin/python3.9 ../src/oxDNA_analysis_tools/align.py minitraj.dat aligned.dat 2>&1 >/dev/null

[ErikPoppleton]

Roden's suggestion is the most likely answer. You can double check your Python interpreter with which python and python --version

[Prateek1410]

Thank you Roden and Erik! Yes, it was because the default python version on my WSL is 2.7. I changed the python invocations in the test file to python3.9 and it worked. Everything came out to be ok. Weirdly though, when I changed the invocations to python3 (which is set to python3.9 itself) it didn't work as OAT and oxpy weren't found.

[Prateek1410]

Also, the earlier version of bond_analysis script used to print the number of native bonds and misbonds for each configuration while it processed the trajectory to the stdout. But the current version only gives the avg number over the entire trajectory. How can I obtain per conf numbers?

[RodenLuo]

For the first, if you type which python3.9 and which python3, are the output the same?

For the second, you may want to take a look at #45. It's not exactly the same, but very close to what you need. But it also requires a bit more work to prepare the input data.

[Prateek1410]

1. No, it gives different output: /usr/bin/python3.9 and /usr/bin/python3 respectively.

2. Thanks, I used the script you offered there and have posted a query regarding the same. Please have a look .

[RodenLuo]

1. This goes beyond oxDNA stuff but more about Linux and python. No need to focus on it if you have a way to run it. But if you are interested:

   You need to further verify if they are the same or not. You may type ls -l /usr/bin/python3.9 and ls -l /usr/bin/python3. In some cases, one might see that python3 is actually pointing to python3.9. In your case, I believe there is either no pointing, or python3 is actually pointing to some other version, say, python3.8. Anyhow, you should be able to type alias python="/usr/bin/python3.9" in your terminal, and then invoking python should work fine from there. To avoid typing this line every time in the terminal, you can put it inside your ~/.bashrc.

   The root of all this is, OAT and oxpy is installed somewhere in your system. In my case, it is ~/.local/lib/python3.10/site-packages/oxpy and ~/.local/bin/oat. Your shell needs to be able to find oat without prefixing it with the whole path. Add ~/.local/bin in your PATH environment variable should do the job. As for oxpy, you need to invoke a python that has ~/.local/lib/python3.10/site-packages in the include dir. There are several ways doing so. Google for PYTHONPATH, sys.path should give you more details.

2. Will take a look and respond later in the other thread.