search

NVlabs / timeloop

Timeloop performs modeling, mapping and code-generation for tensor algebra workloads on various accelerator architectures.
https://timeloop.csail.mit.edu/
BSD 3-Clause "New" or "Revised" License
325 stars 101 forks source link

Using timeloop-model from python script #130

Closed suyashbakshi closed 2 years ago

suyashbakshi commented 2 years ago

Hello, I'm interested in using the timeloop-model to iteratively evaluate different mappings from my python script that generates new mappings (which is sort of a mapper). Is there any easier way to do this, than first writing the mapping to a config file and then invoking timeloop-model as a subprocess from within the python script?

nellie-wu commented 2 years ago

We have developed a python frontend for timeloop that uses pybind to interact with timeloop library. This interface should run faster than subprocess calls. Please find more information about the open-sourced project here: https://github.com/Accelergy-Project/timeloop-python

suyashbakshi commented 2 years ago

Thank you nellie-wu! I will check it out.

suyashbakshi commented 2 years ago

Hello, I tried to install the said project and ran into the below shown error. I have built timeloop earlier and set its include and lib paths (TIMELOOP_INCLUDE_PATH and TIMELOOP_LIB_PATH) as mentioned in the installation steps.

The error seems to be originating from some issue with ninja. I installed the "ninja-build" using apt-get and still got the same error. I'm using Ubuntu 20.04.3.

I'd appreciate any help to fix this issue. Thanks! -- Suyash

$ pip3 install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/suyash/my/timeloop-python
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from pytimeloop==0.0.1) (5.3.1)
Installing collected packages: pytimeloop
  Running setup.py develop for pytimeloop
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [74 lines of output]
        running develop
        /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        WARNING: The user site-packages directory is disabled.
        Checking .pth file support in /home/suyash/.local/lib/python3.8/site-packages
        /usr/bin/python3 -E -c pass
        TEST PASSED: /home/suyash/.local/lib/python3.8/site-packages appears to support .pth files
        running egg_info
        writing pytimeloop.egg-info/PKG-INFO
        writing dependency_links to pytimeloop.egg-info/dependency_links.txt
        writing requirements to pytimeloop.egg-info/requires.txt
        writing top-level names to pytimeloop.egg-info/top_level.txt
        reading manifest file 'pytimeloop.egg-info/SOURCES.txt'
        adding license file 'LICENSE'
        writing manifest file 'pytimeloop.egg-info/SOURCES.txt'
        running build_ext
        CMake Error at CMakeLists.txt:2 (project):
          Running

           '/tmp/pip-build-env-65pt7nc9/overlay/bin/ninja' '--version'

          failed with:

           No such file or directory

        -- Configuring incomplete, errors occurred!
        See also "/home/suyash/my/timeloop-python/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log".
        /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
          warnings.warn(
        /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
          warnings.warn(
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/home/suyash/my/timeloop-python/setup.py", line 106, in <module>
            setup(
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
            return distutils.core.setup(**attrs)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
            return run_commands(dist)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
            dist.run_commands()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
            self.run_command(cmd)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
            super().run_command(command)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
            self.install_for_development()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 114, in install_for_development
            self.run_command('build_ext')
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
            self.distribution.run_command(command)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
            super().run_command(command)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
            cmd_obj.run()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
            _build_ext.run(self)
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
            self.build_extensions()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
            self._build_extensions_serial()
          File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
            self.build_extension(ext)
          File "/home/suyash/my/timeloop-python/setup.py", line 97, in build_extension
            subprocess.check_call(
          File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['cmake', '/home/suyash/my/timeloop-python', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/suyash/my/timeloop-python/build/lib.linux-x86_64-cpython-38/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DVERSION_INFO=0.0.1', '-DCMAKE_BUILD_TYPE=Release', '-GNinja']' returned non-zero exit status 1.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [74 lines of output]
    running develop
    /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    Checking .pth file support in /home/suyash/.local/lib/python3.8/site-packages
    /usr/bin/python3 -E -c pass
    TEST PASSED: /home/suyash/.local/lib/python3.8/site-packages appears to support .pth files
    running egg_info
    writing pytimeloop.egg-info/PKG-INFO
    writing dependency_links to pytimeloop.egg-info/dependency_links.txt
    writing requirements to pytimeloop.egg-info/requires.txt
    writing top-level names to pytimeloop.egg-info/top_level.txt
    reading manifest file 'pytimeloop.egg-info/SOURCES.txt'
    adding license file 'LICENSE'
    writing manifest file 'pytimeloop.egg-info/SOURCES.txt'
    running build_ext
    CMake Error at CMakeLists.txt:2 (project):
      Running

       '/tmp/pip-build-env-65pt7nc9/overlay/bin/ninja' '--version'

      failed with:

       No such file or directory

    -- Configuring incomplete, errors occurred!
    See also "/home/suyash/my/timeloop-python/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log".
    /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
      warnings.warn(
    /tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
      warnings.warn(
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/suyash/my/timeloop-python/setup.py", line 106, in <module>
        setup(
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
        super().run_command(command)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1214, in run_command
        super().run_command(command)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/tmp/pip-build-env-akrea82k/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/suyash/my/timeloop-python/setup.py", line 97, in build_extension
        subprocess.check_call(
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '/home/suyash/my/timeloop-python', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/suyash/my/timeloop-python/build/lib.linux-x86_64-cpython-38/', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DVERSION_INFO=0.0.1', '-DCMAKE_BUILD_TYPE=Release', '-GNinja']' returned non-zero exit status 1.
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
gilbertmike commented 2 years ago

Yeah, this is a weird bug. First, could you try clearing the build directory? A simple rm -rf build will do.

If that doesn't fix it, could you try installing in a virtual environment (I've seen issues when installed in the system env)?

suyashbakshi commented 2 years ago

Thanks for the reply michael. I have tried doing both and no success still.

I also tried rebuilding timeloop (executing "scons" in timeloop dir) and then performed the install instructions, but no success either.

Po0ria commented 2 years ago

Looking at your errors : '/tmp/pip-build-env-65pt7nc9/overlay/bin/ninja' '--version' It is searching for ninja not ninja-build. An easy fix might be to symlink "ninja-build" to "ninja". Take a look at this post

suyashbakshi commented 2 years ago

Hello, I tried the solutions suggested in the link you mentioned, but it still gives the same error.

I executed the two commands mentioned in the following link https://zoomadmin.com/HowToInstall/UbuntuPackage/ninja-build, and I can see both ninja and ninja-build in /usr/bin.

angshuman-parashar commented 2 years ago

Opened new issue at timeloop-python repo: https://github.com/Accelergy-Project/timeloop-python/issues/9.

Closing issue here since it is not related to main Timeloop codebase.