[BUG]: tensornvme installation is incomplete in official docker images

MEllis-github commented 1 year ago

🐛 Describe the bug

Description

The official docker images run the TensorNVME install commands, however at runtime, executing cd TensorNVMe && tensornvme check (or running the training demos depending on tensornvme) produces ImportError: libaio.so.1: cannot open shared object file: No such file or directory.

A few key observations from container runtime:

LD_LIBRARY_PATH is not correctly updated in the image build. At runtime it is LD_LIBRARY_PATH =/usr/local/nvidia/lib:/usr/local/nvidia/lib64.
Running find / -type d -iname ".tensornvme" -ls does not locate the install directory .tensornvme.
The last command of the image build does not include WITH_ROOT=1, so the install directory is ~/.tensornvme versus /usr, and ~ or $HOME evaluates to / in the container runtime.

Attempting to install it at container runtime confirms the attempted install location is /.tensornvme:

$ cd TensorNVMe 
$ pip install -v --no-cache-dir .
Using pip 21.2.4 from /opt/conda/lib/python3.9/site-packages/pip (python 3.9)
Processing /workspace/TensorNVMe
DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Running command python setup.py egg_info
running egg_info
creating /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info
writing /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/dependency_links.txt
writing entry points to /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/entry_points.txt
writing requirements to /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file '/tmp/pip-pip-egg-info-ag4spu7n/tensornvme.egg-info/SOURCES.txt'
Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from tensornvme==0.1.0) (23.0)
Requirement already satisfied: click in /opt/conda/lib/python3.9/site-packages (from tensornvme==0.1.0) (8.1.3)
Requirement already satisfied: torch in /opt/conda/lib/python3.9/site-packages (from tensornvme==0.1.0) (1.12.1)
Requirement already satisfied: typing_extensions in /opt/conda/lib/python3.9/site-packages (from torch->tensornvme==0.1.0) (4.4.0)
Building wheels for collected packages: tensornvme
Running command /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tqkdcb5z
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-r7tnomax/setup.py", line 120, in <module>
  setup_dependencies()
File "/tmp/pip-req-build-r7tnomax/setup.py", line 105, in setup_dependencies
  os.makedirs(backend_install_dir, exist_ok=True)
File "/opt/conda/lib/python3.9/os.py", line 225, in makedirs
  mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.tensornvme'
Building wheel for tensornvme (setup.py) ... error
ERROR: Failed building wheel for tensornvme
Running setup.py clean for tensornvme
Running command /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
running clean
'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.9' does not exist -- can't clean it
Failed to build tensornvme
Installing collected packages: tensornvme
Attempting uninstall: tensornvme
Found existing installation: tensornvme 0.1.0
Uninstalling tensornvme-0.1.0:
  Removing file or directory /opt/conda/bin/tensornvme
  Removing file or directory /opt/conda/lib/python3.9/site-packages/tensornvme-0.1.0.dist-info/
  Removing file or directory /opt/conda/lib/python3.9/site-packages/tensornvme/
  Successfully uninstalled tensornvme-0.1.0
Running command /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-q5nfdluf/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.9/tensornvme
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-req-build-r7tnomax/setup.py", line 120, in <module>
    setup_dependencies()
  File "/tmp/pip-req-build-r7tnomax/setup.py", line 105, in setup_dependencies
    os.makedirs(backend_install_dir, exist_ok=True)
  File "/opt/conda/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.tensornvme'
Running setup.py install for tensornvme ... error
Rolling back uninstall of tensornvme
Moving to /opt/conda/bin/tensornvme
from /tmp/pip-uninstall-0rqpd__w/tensornvme
Moving to /opt/conda/lib/python3.9/site-packages/tensornvme-0.1.0.dist-info/
from /opt/conda/lib/python3.9/site-packages/~ensornvme-0.1.0.dist-info
Moving to /opt/conda/lib/python3.9/site-packages/tensornvme/
from /opt/conda/lib/python3.9/site-packages/~ensornvme
ERROR: Command errored out with exit status 1: /opt/conda/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-r7tnomax/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-q5nfdluf/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.9/tensornvme Check the logs for full command output.

Environment

docker.io/hpcaitech/colossalai:0.2.7

MEllis-github commented 1 year ago

I confirmed setting WITH_ROOT=1 in WITH_ROOT=1 pip install -v --no-cache-dir . in the image build succeeds and causes the tensornvme check to pass. Is there any reason to not include this in the official image build?

MEllis-github commented 1 year ago

While the above enables tensornvme check to pass, the pytest ./tests for backend uring still fail (output below). Checking the build, it appears uring availability is checked, not found, and yet the build does not proceed to install it. The environment was docker.io/hpcaitech/colossalai:0.2.7 with a reinstallation of tensornvme as described above.

============================= test session starts ==============================
platform linux -- Python 3.9.12, pytest-7.3.1, pluggy-1.0.0
rootdir: /workspace/TensorNVMe
collected 9 items

tests/test_adam.py F                                                     [ 11%]
tests/test_disk_offloader.py F.F.F.F.                                    [100%]

=================================== FAILURES ===================================
__________________________________ test_adam ___________________________________

    @torch.no_grad()
    def test_adam():
        params = list(gpt2_toy().cpu().parameters())
        for _, p in enumerate(params):
            if p.grad is None and p.requires_grad:
                p.grad = torch.ones_like(p.data, dtype=torch.float) * 0.12345

        params_gt = copy.deepcopy(params)
        for _, p in enumerate(params_gt):
            if p.grad is None and p.requires_grad:
                p.grad = torch.ones_like(p.data, dtype=torch.float) * 0.12345
        optimizer = Adam(params_gt, 1e-3)
        optimizer.step()

        test_config = [
            {'n_entries': 1, 'backend': None, 'prefetch': 0, 'vecio': False},

            {'n_entries': 1, 'backend': 'uring', 'prefetch': 0, 'vecio': False},
            {'n_entries': 8, 'backend': 'uring', 'prefetch': 2, 'vecio': False},

            {'n_entries': 1, 'backend': 'uring', 'prefetch': 0, 'vecio': True},
            {'n_entries': 8, 'backend': 'uring', 'prefetch': 2, 'vecio': True},

            {'n_entries': 1, 'backend': 'aio', 'prefetch': 0, 'vecio': False},
            {'n_entries': 8, 'backend': 'aio', 'prefetch': 2, 'vecio': False},

            {'n_entries': 1, 'backend': 'aio', 'prefetch': 0, 'vecio': True},
            {'n_entries': 8, 'backend': 'aio', 'prefetch': 2, 'vecio': True},
        ]

        for i, cfg in enumerate(test_config):
            params_test = copy.deepcopy(params)
            for _, p in enumerate(params_test):
                if p.grad is None and p.requires_grad:
                    p.grad = torch.ones_like(p.data, dtype=torch.float) * 0.12345
            if cfg['backend'] is None:
                offloader = None
            else:
>               offloader = DiskOffloader(
                    '.', cfg['n_entries'], backend=cfg['backend'])

tests/test_adam.py:221: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensornvme.offload.DiskOffloader object at 0x7f2c77f2c720>
dir_name = '.', n_entries = 1, backend = 'uring'

    def __init__(self, dir_name: str, n_entries: int = 16, backend: str = 'uring') -> None:
>       assert backend in get_backends(
        ), f'Unsupported backend: {backend}, please install tensornvme with this backend'
E       AssertionError: Unsupported backend: uring, please install tensornvme with this backend

/opt/conda/lib/python3.9/site-packages/tensornvme/offload.py:10: AssertionError
_____________________________ test_sync_io[uring] ______________________________

backend = 'uring'

    @pytest.mark.parametrize('backend', ['uring', 'aio'])
    def test_sync_io(backend):
        x = torch.rand(2, 2)
        x_copy = x.clone()
>       of = DiskOffloader('.', backend=backend)

tests/test_disk_offloader.py:10: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensornvme.offload.DiskOffloader object at 0x7f2c77f2c9f0>
dir_name = '.', n_entries = 16, backend = 'uring'

    def __init__(self, dir_name: str, n_entries: int = 16, backend: str = 'uring') -> None:
>       assert backend in get_backends(
        ), f'Unsupported backend: {backend}, please install tensornvme with this backend'
E       AssertionError: Unsupported backend: uring, please install tensornvme with this backend

/opt/conda/lib/python3.9/site-packages/tensornvme/offload.py:10: AssertionError
_____________________________ test_async_io[uring] _____________________________

backend = 'uring'

    @pytest.mark.parametrize('backend', ['uring', 'aio'])
    def test_async_io(backend):
        x = torch.rand(2, 2)
        x_copy = x.clone()
>       of = DiskOffloader('.', backend=backend)

tests/test_disk_offloader.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensornvme.offload.DiskOffloader object at 0x7f2c77ed47c0>
dir_name = '.', n_entries = 16, backend = 'uring'

    def __init__(self, dir_name: str, n_entries: int = 16, backend: str = 'uring') -> None:
>       assert backend in get_backends(
        ), f'Unsupported backend: {backend}, please install tensornvme with this backend'
E       AssertionError: Unsupported backend: uring, please install tensornvme with this backend

/opt/conda/lib/python3.9/site-packages/tensornvme/offload.py:10: AssertionError
___________________________ test_sync_vec_io[uring] ____________________________

backend = 'uring'

    @pytest.mark.parametrize('backend', ['uring', 'aio'])
    def test_sync_vec_io(backend):
        x = torch.rand(2, 2)
        y = torch.rand(2, 2, 2)
        x_copy = x.clone()
        y_copy = y.clone()
>       of = DiskOffloader('.', backend=backend)

tests/test_disk_offloader.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensornvme.offload.DiskOffloader object at 0x7f2c77edc0e0>
dir_name = '.', n_entries = 16, backend = 'uring'

    def __init__(self, dir_name: str, n_entries: int = 16, backend: str = 'uring') -> None:
>       assert backend in get_backends(
        ), f'Unsupported backend: {backend}, please install tensornvme with this backend'
E       AssertionError: Unsupported backend: uring, please install tensornvme with this backend

/opt/conda/lib/python3.9/site-packages/tensornvme/offload.py:10: AssertionError
___________________________ test_async_vec_io[uring] ___________________________

backend = 'uring'

    @pytest.mark.parametrize('backend', ['uring', 'aio'])
    def test_async_vec_io(backend):
        x = torch.rand(2, 2)
        y = torch.rand(2, 2, 2)
        x_copy = x.clone()
        y_copy = y.clone()
>       of = DiskOffloader('.', backend=backend)

tests/test_disk_offloader.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tensornvme.offload.DiskOffloader object at 0x7f2c77edc630>
dir_name = '.', n_entries = 16, backend = 'uring'

    def __init__(self, dir_name: str, n_entries: int = 16, backend: str = 'uring') -> None:
>       assert backend in get_backends(
        ), f'Unsupported backend: {backend}, please install tensornvme with this backend'
E       AssertionError: Unsupported backend: uring, please install tensornvme with this backend

/opt/conda/lib/python3.9/site-packages/tensornvme/offload.py:10: AssertionError
=========================== short test summary info ============================
FAILED tests/test_adam.py::test_adam - AssertionError: Unsupported backend: u...
FAILED tests/test_disk_offloader.py::test_sync_io[uring] - AssertionError: Un...
FAILED tests/test_disk_offloader.py::test_async_io[uring] - AssertionError: U...
FAILED tests/test_disk_offloader.py::test_sync_vec_io[uring] - AssertionError...
FAILED tests/test_disk_offloader.py::test_async_vec_io[uring] - AssertionErro...
========================= 5 failed, 4 passed in 2.21s ==========================

hpcaitech / ColossalAI