search

DIMAthe47 / cbir_framework

0 stars 2 forks source link

Not work InvertedMultiIndexSearcherTest. #1

Open heinrichI opened 4 years ago

heinrichI commented 4 years ago

I run all test, but InvertedMultiIndexSearcherTest not execute. image More precisely on the line 

py_imi = pyimi.PyInvertedMultiIndexBuilder().buildInvertedMultiIndex(x_ids, x_pqcodes,
                                                                                 cluster_centers)

.pyd files are in the folder image

DIMAthe47 commented 4 years ago

I have just successfully run test_inverted_multi_index_searcher.

Steps to find the reason of failure:

>>venv\Scripts\activate
>>cd cbir_framework\core\search\inverted_multi_index
>>python
>>import py_inverted_multi_index

This gave me an error "ImportError: DLL load failed". With help of DependenciesGui.exe (https://github.com/lucasg/Dependencies) (be sure to run it from your virtualenv's command prompt) find what is missing in py_inverted_multi_index library. In my case it was python35.dll, libgfortran-3.dll, libopenblass.dll.

Steps I have made to fix it. Create virtual env with python 3.5. Download openblas, add it's bin directory to path. Download and install mingw-w64 x86_64-6.1.0-win32-seh-rt_v5-rev1, add it's bin folder to path (or another mingw64 with libgfortran-3.dll).

After these steps "import py_inverted_multi_index" had success. And test_inverted_multi_index_searcher.py run had success too.

DIMAthe47 commented 4 years ago

My recommendations on project. I tried to make stream-like loading, computing, transformation, saving. I even tried to encode these steps in json-configs: https://github.com/DIMAthe47/cbir_47/tree/master/model_generators. But it is all in raw state and in big mess.

I think most valuable part of this project are c++ modules, cython wraps, and examples of how to use them. But if your main app is in python and you need c++ only for perfomance I would advise to use https://numba.pydata.org/ instead.

For storing data I would advise: numpy arrays: .npy, .npz, or .hdf5. images: file hierarchy inside .zip(zip is not for compression because most images are already compressed, but for grouping it inside one file). I think I have good examples for storing: https://gitlab.com/Digipathology/Pathadin/-/tree/master/src/main/python/ndarray_persist.

heinrichI commented 4 years ago

Thanks, I already guessed that the problem is in the libraries. Added them. image But still the test is aborted without errors on

        searcher_ = imis.InvertedMultiIndexSearcher(x=base_vectors, x_ids=ids_ndarray,
                                                    cluster_centers=cluster_centers)

image I think, reason in python_wrap. Create another Issue: https://github.com/DIMAthe47/inverted_multi_index/issues/1