v1.0.5 AISFileLister ModuleNotFoundError

[austinmw]

After installing v1.0.5, I get the following error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[43], line 7
      4 image_prefix = ["ais://cifar10/"]
      6 # Listing all files starting with these prefixes on AIStore
----> 7 dp_urls = AISFileLister(url="http://localhost:51080/", source_datapipe=image_prefix)
      9 # list obj urls
     10 print(list(dp_urls))

File [~/anaconda3/envs/pytorch2/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:82](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/~/anaconda3/envs/pytorch2/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:82), in AISFileListerIterDataPipe.__init__(self, source_datapipe, url, length)
     81 def __init__(self, source_datapipe: IterDataPipe[str], url: str, length: int = -1) -> None:
---> 82     _assert_aistore()
     83     _assert_aistore_version()
     84     self.source_datapipe: IterDataPipe[str] = source_datapipe

File [~/anaconda3/envs/pytorch2/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:34](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/~/anaconda3/envs/pytorch2/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:34), in _assert_aistore()
     32 def _assert_aistore() -> None:
     33     if not HAS_AIS:
---> 34         raise ModuleNotFoundError(
     35             "Package `aistore` (>=1.0.2) is required to be installed to use this datapipe."
     36             "Please run `pip install --upgrade aistore` or `conda install aistore` to install the package"
     37             "For more info visit: https://github.com/NVIDIA/aistore/blob/master/sdk/python/"
     38         )

ModuleNotFoundError: Package `aistore` (>=1.0.2) is required to be installed to use this datapipe.Please run `pip install --upgrade aistore` or `conda install aistore` to install the packageFor more info visit: https://github.com/NVIDIA/aistore/blob/master/sdk/python/

[aaronnw]

Thanks for bringing this up. It looks like there was an error with the version import for v1.0.5.

There is also an issue with compatibility between the pytorch integration and the newer sdk versions. See this updated README: https://github.com/NVIDIA/aistore/blob/master/python/aistore/pytorch/README.md#compatibility-issue

Please install 1.0.6 and give the patch above a shot.

[aaronnw]

Ok, we have patched the Python SDK with a new version, 1.0.7, which does not require any additional workarounds to work with torchdata. Please install that version for full compatibility and let us know if you have any issues!

[austinmw]

Thanks!

[austinmw]

@aaronnw Hi, I've installed aistore==1.0.7, but I still get this error:

ModuleNotFoundError                       Traceback (most recent call last)
[/home/ec2-user/SageMaker/torchdata/TorchData.ipynb](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/TorchData.ipynb) Cell 109 in ()
      4 image_prefix = ["ais://cifar10/"]
      6 # Listing all files starting with these prefixes on AIStore
----> 7 dp_urls = AISFileLister(url="http://localhost:51080/", source_datapipe=image_prefix)
      9 # list obj urls
     10 print(list(dp_urls))

File [~/anaconda3/envs/torchdata/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:82](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/~/anaconda3/envs/torchdata/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:82), in AISFileListerIterDataPipe.__init__(self, source_datapipe, url, length)
     81 def __init__(self, source_datapipe: IterDataPipe[str], url: str, length: int = -1) -> None:
---> 82     _assert_aistore()
     83     _assert_aistore_version()
     84     self.source_datapipe: IterDataPipe[str] = source_datapipe

File [~/anaconda3/envs/torchdata/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:34](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/~/anaconda3/envs/torchdata/lib/python3.8/site-packages/torchdata/datapipes/iter/load/aisio.py:34), in _assert_aistore()
     32 def _assert_aistore() -> None:
     33     if not HAS_AIS:
---> 34         raise ModuleNotFoundError(
     35             "Package `aistore` (>=1.0.2) is required to be installed to use this datapipe."
     36             "Please run `pip install --upgrade aistore` or `conda install aistore` to install the package"
     37             "For more info visit: https://github.com/NVIDIA/aistore/blob/master/sdk/python/"
     38         )

ModuleNotFoundError: Package `aistore` (>=1.0.2) is required to be installed to use this datapipe.Please run `pip install --upgrade aistore` or `conda install aistore` to install the packageFor more info visit: https://github.com/NVIDIA/aistore/blob/master/sdk/python/

[austinmw]

Also I get an error with from aistore.botocore_patch import botocore:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[/home/ec2-user/SageMaker/torchdata/TorchData.ipynb](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/TorchData.ipynb) Cell 117 in ()
      1 import boto3
----> 2 from aistore.botocore_patch import botocore

File [~/anaconda3/envs/torchdata/lib/python3.8/site-packages/aistore/botocore_patch/botocore.py:29](https://vscode-remote+ssh-002dremote-002bvscode.vscode-resource.vscode-cdn.net/home/ec2-user/SageMaker/torchdata/~/anaconda3/envs/torchdata/lib/python3.8/site-packages/aistore/botocore_patch/botocore.py:29), in 
      4 """
      5 Allow use of Amazon's boto3 and botocore libraries with AIStore.
      6 
   (...)
     25 is set.
     26 """
     28 import inspect
---> 29 import wrapt
     32 # pylint: disable=unused-argument
     33 def _ais_redirect_wrapper(wrapped, instance, args, kwargs):

ModuleNotFoundError: No module named 'wrapt'

[aaronnw]

For your first comment, I'm not able to replicate that with 1.0.7 and AISFileLister. Can you verify your version by importing aistore and checking aistore.__version__?

For your second comment, you'll need to install aistore with the botocore requirements, e.g.

pip install aistore[botocore]

as shown here in the botocore section of the README

[austinmw]

(restarted kernel to make sure)

[aaronnw]

Thanks for the feedback. Looks like I made a mistake and missed including the module patch in the latest version, so torchdata could not find it without the manual patch. I've updated it yet again to 1.0.8 and it should (finally) work. Sorry for the trouble!