Twenkid
commented
1 year ago I managed to fix it and run the whole clip-retrieval notepad example locally. The import now completes.
>>> import clip_retrieval
>>> dir(clip_retrieval)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'clip_back', 'clip_end2end', 'clip_filter', 'clip_front', 'clip_index', 'clip_inference', 'ivf_metadata_ordering', 'load_clip']The solution: I moved to Python 3.10 (maybe not strictly needed, IDK) and upgraded pip. Then there was another error during trying to run clip-retrieval command, during importing _pustil, "partial import", which happened to be a conflict in the installation, maybe mixing sudo apt-get install and pip3 install / python3.10 -m pip install, because the error trace included references to two different directories.
File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 95, in <module>
from . import _pslinux as _psplatform
File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 26, in <module>
from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib/python3/dist-packages/psutil/__init__.py)There were problems with removing and reinstalling the conflicting libraries, but manually deleting with sudo rm -r ... of the folders with the modules psutil and then reinstalling it with pip finally resolved it. (There were two, which added to the confusion, one with .egg-info sudo rm -r /usr/lib/python3/dist-packages/psutil-5.5.1 sudo rm -r /usr/lib/python3/dist-packages/psutil-5.5.1.egg-info
img2dataset --url_list=myimglist.txt --output_folder=image_folder --thread_count=4 --image_
size=256
Starting the downloading of this file
Sharding file number 1 of 1 called /mnt/z/myimglist.txt
0it [00:00, ?it/s]File sharded in 0 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!
0it [00:00, ?it/s]myimglist.txt
https://placekitten.com/200/305 https://placekitten.com/200/304 https://placekitten.com/200/303 https://placekitten.com/200/302 https://placekitten.com/150/235 https://placekitten.com/120/120
When calling clip-retrieval, it downloads
clip-retrieval inference --input_dataset image_folder --output_folder embedding_folder
The number of samples has been estimated to be 6
Starting the worker
dataset is 12
Starting work on task 0
100%|███████████████████████████████████████| 354M/354M [01:35<00:00, 3.72MiB/s]
/home/xxxx/.local/lib/python3.10/site-packages/clip/clip.py:160: FutureWarning: 'torch.onnx._patch_torch._node_getitem' is deprecated in version 1.13 and will be removed in version 1.14. Please Internally use '_node_get' in symbolic_helper instead..
if "value" in node.attributeNames() and str(node["value"]).startswith("cuda"):
/home/xxxx/.local/lib/python3.10/site-packages/clip/clip.py:186: FutureWarning: 'torch.onnx._patch_torch._node_getitem' is deprecated in version 1.13 and will be removed in version 1.14. Please Internally use '_node_get' in symbolic_helper instead..
if inputs[i].node()["value"] == 5:
warming up with batch size 256 on cpu
done warming up in 56.720762968063354s
/home/tosh/.local/lib/python3.10/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
sample_per_sec 1 ; sample_count 6...
Windows
If that info would be useful for anyone, I realized I was trying to import clip_retrieval in CLI (on WSL I was doing the same initially), while the example usage from the sample notebook is to call it as a command. img2dataset succeeds:
<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject
Starting the downloading of this file
Sharding file number 1 of 1 called z:/list.txt
0it [00:00, ?it/s]File sharded in 1 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!
<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject
<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject
1it [00:07, 7.66s/it]
worker - success: 1.000 - failed to download: 0.000 - failed to resize: 0.000 - images per sec: 8 - count: 18
total - success: 1.000 - failed to download: 0.000 - failed to resize: 0.000 - images per sec: 8 - count: 18Calling clip-retrieval properly now returned another error: JAX requirement and it seems it has to be built from the source. So probably it may be just JAX, rather than OpenMP and BLAS, IDK yet.
clip-retrieval
<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject
Traceback (most recent call last):
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\site-packages\jax\_src\lib\__init__.py", line 38, in <module>
import jaxlib as jaxlib
ModuleNotFoundError: No module named 'jaxlib'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\Scripts\clip-retrieval.exe\__main__.py", line 4, in <module>
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\__init__.py", line 6, in <module>
from .clip_inference.main import main as clip_inference
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\clip_inference\__init__.py", line 3, in <module>
from .main import main as clip_inference
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\clip_inference\main.py", line 10, in <module>
from clip_retrieval.clip_inference.distributor import PysparkDistributor, SequentialDistributor
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\clip_inference\distributor.py", line 5, in <module>
from .worker import worker
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\clip_inference\worker.py", line 15, in <module>
from clip_retrieval.clip_inference.mapper import ClipMapper
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\clip_retrieval\clip_inference\mapper.py", line 5, in <module>
from sentence_transformers import SentenceTransformer
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\sentence_transformers\__init__.py", line 3, in <module>
from .datasets import SentencesDataset, ParallelSentencesDataset
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\sentence_transformers\datasets\__init__.py", line 3, in <module>
from .ParallelSentencesDataset import ParallelSentencesDataset
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\sentence_transformers\datasets\ParallelSentencesDataset.py", line 4, in <module>
from .. import SentenceTransformer
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\sentence_transformers\SentenceTransformer.py", line 11, in <module>
import transformers
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\transformers\__init__.py", line 30, in <module>
from . import dependency_versions_check
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\transformers\dependency_versions_check.py", line 17, in <module>
from .utils.versions import require_version, require_version_core
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\transformers\utils\__init__.py", line 34, in <module>
from .generic import (
File "C:\Users\toshb\AppData\Roaming\Python\Python39\site-packages\transformers\utils\generic.py", line 36, in <module>
import jax.numpy as jnp
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\site-packages\jax\__init__.py", line 35, in <module>
from jax import config as _config_module
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\site-packages\jax\config.py", line 17, in <module>
from jax._src.config import config
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\site-packages\jax\_src\config.py", line 29, in <module>
from jax._src import lib
File "C:\Users\toshb\AppData\Local\Programs\Python\Python39\lib\site-packages\jax\_src\lib\__init__.py", line 40, in <module>
raise ModuleNotFoundError(
ModuleNotFoundError: jax requires jaxlib to be installed. See https://github.com/google/jax#installation for installation instructions.
rom1504
Any tips for resolving this error? As far as I checked all dependencies are installed and are within the ranges (see below), also I tried uninstall-install for the suspect problematic libs (flask, requests). I didn't use venv so far though if that could be a problem with some of the libs, flask in particular which breaks during importing) and I don't know if this particular Py version could be a problem.
On Windows, where I first tired, the error on WSL2 did not happen and manual importing flask and HTTPAdapter run, i.e.
from flask import Flaskandfrom requests.adapters import HTTPAdapterreturn fine. In WSL,>>> import requestsreturns, butfrom requests.adapters import HTTPAdapteris erroneous as in the clip_retrieval error.(On Win the final obstacle were (perhaps as far as I discovered) BLAS and OpenMP missing binaries/proper installation/connection/build (
from . import _swigfaiss ...). I tried to fix it for a while, even manually copying .dlls (libblas.dll, libiomp5md.dll, liblapack.dll"), but the error didn't change, then I gave up and tried WSL.https://github.com/rom1504/clip-retrieval/blob/main/requirements.txt