Ninja error

[angshuman-parashar]

Ninja error reported by @suyashbakshi in Timeloop issue https://github.com/NVlabs/timeloop/issues/130.

Original comment: https://github.com/NVlabs/timeloop/issues/130#issuecomment-1100439685

[suyashbakshi]

Hello, I just wanted to give a quick update on this issue. I'm still running into issues while installing timeloop-python. But the issue is rather different now. I have upgraded to a Ubuntu 22.04 VM compared to my earlier version 20.04.

I have built timeloop and accelergy by following the instructions on https://timeloop.csail.mit.edu/. Then I cloned timeloop-python repo and set the TIMELOOP_INCLUDE_PATH and TIMELOOP_LIB_PATH to point to the 'include' and 'lib' dirs of timeloop.

Now when I execute sudo pip3 install -e . for timeloop-python, I get the following error complaining about the TIMELOOP_INCLUDE_PATH env variable.

I'm not entirely sure where am I going wrong with the build process. Any help would be greatly appreciated.

suyash@suyash-virtual-machine:~/work/timeloop-dev/timeloop-python$ sudo pip3 install -e .
Obtaining file:///home/suyash/work/timeloop-dev/timeloop-python
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from pytimeloop==0.0.1) (5.4.1)
Installing collected packages: pytimeloop
  Running setup.py develop for pytimeloop
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [66 lines of output]
        running develop
        /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
          warnings.warn(
        /usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
          warnings.warn(
        running egg_info
        writing manifest file 'pytimeloop.egg-info/SOURCES.txt'
        running build_ext
        -- The C compiler identification is GNU 11.2.0
        -- The CXX compiler identification is GNU 11.2.0
        -- Detecting C compiler ABI info
        -- Detecting C compiler ABI info - done
        -- Check for working C compiler: /usr/bin/cc - skipped
        -- Detecting C compile features
        -- Detecting C compile features - done
        -- Detecting CXX compiler ABI info
        -- Detecting CXX compiler ABI info - done
        -- Check for working CXX compiler: /usr/bin/c++ - skipped
        -- Detecting CXX compile features
        -- Detecting CXX compile features - done
        CMake Error at CMakeLists.txt:8 (message):
          Please point TIMELOOP_INCLUDE_PATH to your Timeloop headers.

        -- Configuring incomplete, errors occurred!
        See also "/home/suyash/work/timeloop-dev/timeloop-python/build/temp.linux-x86_64-3.10/CMakeFiles/CMakeOutput.log".
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/home/suyash/work/timeloop-dev/timeloop-python/setup.py", line 106, in <module>
            setup(
          File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
            return distutils.core.setup(**attrs)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
            return run_commands(dist)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
            dist.run_commands()
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
            self.run_command(cmd)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
            cmd_obj.run()
          File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run
            self.install_for_development()
          File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 114, in install_for_development
            self.run_command('build_ext')
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
            self.distribution.run_command(command)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
            cmd_obj.run()
          File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 79, in run
            _build_ext.run(self)
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
            self.build_extensions()
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
            self._build_extensions_serial()
          File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
            self.build_extension(ext)
          File "/home/suyash/work/timeloop-dev/timeloop-python/setup.py", line 97, in build_extension
            subprocess.check_call(
          File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['cmake', '/home/suyash/work/timeloop-dev/timeloop-python', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/suyash/work/timeloop-dev/timeloop-python/build/lib.linux-x86_64-3.10/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DVERSION_INFO=0.0.1', '-DCMAKE_BUILD_TYPE=Release', '-GNinja']' returned non-zero exit status 1.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [66 lines of output]
    running develop
    /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
      warnings.warn(
    /usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
      warnings.warn(
    running egg_info
    writing manifest file 'pytimeloop.egg-info/SOURCES.txt'
    running build_ext
    -- The C compiler identification is GNU 11.2.0
    -- The CXX compiler identification is GNU 11.2.0
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working C compiler: /usr/bin/cc - skipped
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Check for working CXX compiler: /usr/bin/c++ - skipped
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    CMake Error at CMakeLists.txt:8 (message):
      Please point TIMELOOP_INCLUDE_PATH to your Timeloop headers.

    -- Configuring incomplete, errors occurred!
    See also "/home/suyash/work/timeloop-dev/timeloop-python/build/temp.linux-x86_64-3.10/CMakeFiles/CMakeOutput.log".
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/suyash/work/timeloop-dev/timeloop-python/setup.py", line 106, in <module>
        setup(
      File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/usr/lib/python3/dist-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/usr/lib/python3/dist-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/suyash/work/timeloop-dev/timeloop-python/setup.py", line 97, in build_extension
        subprocess.check_call(
      File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '/home/suyash/work/timeloop-dev/timeloop-python', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/suyash/work/timeloop-dev/timeloop-python/build/lib.linux-x86_64-3.10/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DVERSION_INFO=0.0.1', '-DCMAKE_BUILD_TYPE=Release', '-GNinja']' returned non-zero exit status 1.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[angshuman-parashar]

I'm sure you've double-checked this already, but just for our reference could you paste the contents of your $TIMELOOP_INCLUDE_PATH?

[suyashbakshi]

Sure.

suyash@suyash-virtual-machine:~/work/timeloop-dev/timeloop-python$ echo $TIMELOOP_INCLUDE_PATH 
/home/suyash/work/timeloop-dev/timeloop/include
suyash@suyash-virtual-machine:~/work/timeloop-dev/timeloop-python$ echo $TIMELOOP_LIB_PATH 
/home/suyash/work/timeloop-dev/timeloop/lib/

[suyashbakshi]

Hello, is there any possible solution to this? I would really like to use Timeloop-python for my research work.

If there are any known "environments" (operating system, dependency versions etc.) where Timeloop-python is working for others, I'd be interested in knowing that too and will try to setup a similar environment.

[angshuman-parashar]

Reproduced the ninja issue on Ubuntu 20.04. We are working on it. Please bear with us.

[angshuman-parashar]

On Ubuntu 20.04 export the following env variable:

export CMAKE_GENERATOR="Unix Makefiles"

Then torpedo everything, do a fresh git clone (submodules and all) and try the pip3 install again. It should work.

[suyashbakshi]

Thank you, I will try this and get back to you.

[suyashbakshi]

Hello, thank you! The command you suggested seems to have fixed the issue. The pip installation finished successfully:

~/work/timeloop-dev/timeloop-python$ pip3 install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/suyash/work/timeloop-dev/timeloop-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from pytimeloop==0.0.1) (5.4.1)
Building wheels for collected packages: pytimeloop
  Building wheel for pytimeloop (pyproject.toml) ... done
  Created wheel for pytimeloop: filename=pytimeloop-0.0.1-cp310-cp310-linux_x86_64.whl size=365771 sha256=3c4d6137575463985c90c96b2c4c355f60ec01f7f1f94270309dd80d6ad155b2
  Stored in directory: /home/suyash/.cache/pip/wheels/68/f4/1d/31df7c8878faa3da9b844d2c39a8870a604874d379c053f770
Successfully built pytimeloop
Installing collected packages: pytimeloop
Successfully installed pytimeloop-0.0.1

However, now when I run the "timeloop-model.py" tool with the "sample.cfg" from "timeloop-dev/timeloop/configs/model", it gives the following error. Is the config file format different for pyTimeloop? Or am I missing something?

~/work/timeloop-dev/timeloop-python$ timeloop-model.py ~/work/timeloop-dev/timeloop/configs/model/sample.cfg 
free(): invalid pointer
Aborted (core dumped)

[suyashbakshi]

Hello, so from the tutorials, I realized that the timeloop-model.py and timeloop-mapper.py requires 3 config files as input: arch, prob and mapper/model settings (which was causing the segfault in my previous post). However, for a mapper configuration file that works without any issue with the native (C++) timeloop tool, the same config file results in the below mentioned error for me when running with timeloop-python (timeloop-mapper.py). Could you please help me understand the issue with this?

I have attached the arch, map (for mapper) and prob config files (converted them to "txt" format to be able to upload on github but I provide them in yaml format as input param to timeloop-mapper.py).

test_arch.txt test_map.txt test_prob.txt

Following is the error that I get:

$ timeloop-mapper.py test_arch.yaml test_prob.yaml test_map.yaml 
[INFO] 2022-05-28 14:08:57,745 - pytimeloop.accelergy_interface.invoke_accelergy - execute:/home/suyash/.local/bin/accelergy tmp-accelergy.yaml --oprefix timeloop-mapper. -o ./ > timeloop-mapper.accelergy.log 2>&1

MESSAGE: attempting to read problem shape from file: /home/suyash/work/timeloop-dev/timeloop/problem-shapes/cnn-layer.yaml
[INFO] 2022-05-28 14:08:57,780 - pytimeloop.mapspace.MapSpace - Initializing Index Factorization subspace.
  Factorization options along problem dimension R = 6
  Factorization options along problem dimension S = 6
  Factorization options along problem dimension P = 6
  Factorization options along problem dimension Q = 6
  Factorization options along problem dimension C = 1287
  Factorization options along problem dimension K = 4752
  Factorization options along problem dimension N = 1
Mapspace Dimension [IndexFactorization] Size: 7926107904
Mapspace Dimension [LoopPermutation] Size: 128024064000
Mapspace Dimension [Spatial] Size: 64
Mapspace Dimension [DatatypeBypass] Size: 512

Mapspace split! Per-split Mapping Dimension [IndexFactorization] Size: 7926107904 Residue: 0
['energy']
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
Aborted (core dumped)

I would greatly appreciate any help!

[gilbertmike]

@suyashbakshi Thanks for reporting the issue. This was actually fixed last week. Could you try doing a git pull and reinstall?

Also, PyTimeloop's focus right now is for programatically using Timeloop's features, not as replacement for the timeloop-model and timeloop-mapper binaries. The output formatting was low in my priority so it isn't implemented. PyTimeloop won't print the results in timeloop-mapper.stats.txt. That will be coming in the future.

[suyashbakshi]

@gilbertmike Thank you for the update. No problem, I only wish to use the basic functionality of the "model" and "mapper" from my python scripts, which as I understand are being supported.

On another note, it appears that the examples have not been updated in the "examples/" dir. Since naively running the example results in a segfault, it may be tricky for a new user to find the problem. I'm adding my findings from debugging here just for reference. For instance, the example: https://github.com/Accelergy-Project/timeloop-python/blob/f415a2dae9691fa95e827d206af00ac37dce8595/examples/model.py if executed as it is, resulted in a segfault.

I understand following changes are required to correct them:

1. Change the following line: https://github.com/Accelergy-Project/timeloop-python/blob/f415a2dae9691fa95e827d206af00ac37dce8595/examples/model.py#L6 to: from pytimeloop.app import model

2. following line: https://github.com/Accelergy-Project/timeloop-python/blob/f415a2dae9691fa95e827d206af00ac37dce8595/examples/model.py#L35 to: app = model.ModelApp(config, '.')

3. Also directly printing eval_stats here: https://github.com/Accelergy-Project/timeloop-python/blob/f415a2dae9691fa95e827d206af00ac37dce8595/examples/model.py#L37 only returns an object of type "bindings.model.EvaluationResult" To get the results, one would need to print the stats as follows:

   print("Utilization: ",eval_stats.utilization)
   print("Energy: ",eval_stats.energy)
   print("Algorithmic computes: ",eval_stats.algorithmic_computes)
   print("Actual compute: ",eval_stats.actual_computes)
   print("Cycles: ",eval_stats.cycles)
   print("Last level accesses: ",eval_stats.last_level_accesses)

I hope this would be helpful.

[gilbertmike]

@suyashbakshi Thanks for pointing out the bug. This was written very early in PyTimeloop development. They're quite outdated, as you pointed out. I think I will just delete those and point users to the bin/ folder since that one will always be up to date.

[gilbertmike]

The `CMAKE_GENERATOR="Unix Makefiles" workaround is no longer needed. I updated the build system to just use Make on non Windows platforms. I'll close the issue.

Please comment if you think the issue hasn't been resolved.