OpenCV/Numpy Installation Issue on Conda Python 3.10 (MacOS)

wjbmattingly commented 2 months ago

Search before asking

[X] I have searched the Multimodal Maestro issues and found no similar bug report.

Bug

First of all, thank you for this package! I am excited to test it out. I have a slight issue, however. I created a fresh environment to test out the package.

When trying to import modules from the maestro package, there's a NumPy version compatibility error. The error suggests that a module compiled with NumPy 1.x is being run with NumPy 2.1.1, which can cause crashes.

Code to reproduce:

from maestro.trainer.common import MeanAveragePrecisionMetric
from maestro.trainer.models.florence_2 import train, TrainingConfiguration

Error message:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need t
...
ImportError: numpy.core.multiarray failed to import

Environment:

Operating System: macOS
Python version: 3.10
Maestro version: 0.1.0

Environment

Environment:

Operating System: macOS
Python version: 3.10
Maestro version: 0.1.0

Minimal Reproducible Example

Code to reproduce:

from maestro.trainer.common import MeanAveragePrecisionMetric
from maestro.trainer.models.florence_2 import train, TrainingConfiguration

Additional

No response

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

onuralpszr commented 2 months ago

Hello @wjbmattingly I was setting up CI and other configuration I decided to take look this problem and

[x] Tested python 3.12 OSX "worked" ✅ - no problem detected
[x] Tested python 3.11 OSX "worked" ✅ - no problem detected
[x] Tested python 3.10 OSX "worked" ✅ - no problem detected
[x] Tested python 3.9 OSX "worked" ✅ - no problem detected

I assumed mac you used apple silicon and I used brew to install all of the pythons(3.6 to 3.13) all of them tested under new virtualenv envs and no-cache installs

OSX version: Sonama 14.6

onuralpszr commented 2 months ago

@wjbmattingly related PR https://github.com/roboflow/multimodal-maestro/pull/43 I worked for ci and package dependencies (I tested on this PR branch)

onuralpszr commented 2 months ago

@wjbmattingly We just merged changes I made so could you please re-try this to verify is it work or not, thank you.

wjbmattingly commented 2 months ago

This is so strange. I created a new Conda env and reinstalled. I have the same error for some reason. Here is the full error:

I am in a fresh Conda env with Python 3.10. Perhaps it's an issue with Conda? I can try using virtualenv to see if that helps. All I did to install was the following

pip install --no-cache-dir maestro

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
    app.launch_new_instance()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
    app.start()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
    self.io_loop.start()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 205, in start
    self.asyncio_loop.run_forever()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue
    await self.process_one()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 534, in process_one
    await dispatch(*args)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell
    await result
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 362, in execute_request
    await super().execute_request(stream, ident, parent)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 778, in execute_request
    reply_content = await reply_content
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 449, in do_execute
    res = shell.run_cell(
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 549, in run_cell
    return super().run_cell(*args, **kwargs)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell
    result = self._run_cell(
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell
    result = runner(coro)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 128, in _pseudo_sync_runner
    coro.send(None)
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/var/folders/4f/ddlj81h90_n0_h5wwvjbd2b40000gn/T/ipykernel_60948/3770834381.py", line 1, in <module>
    from maestro.trainer.common import MeanAveragePrecisionMetric
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/maestro/__init__.py", line 9, in <module>
    from maestro.lmms.gpt4 import prompt_image
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/maestro/lmms/gpt4.py", line 3, in <module>
    import cv2
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "/Applications/anaconda3/envs/maestro2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

{
    "name": "AttributeError",
    "message": "_ARRAY_API not found",
    "stack": "---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
AttributeError: _ARRAY_API not found"
}

{
    "name": "ImportError",
    "message": "numpy.core.multiarray failed to import",
    "stack": "---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 from maestro.trainer.common import MeanAveragePrecisionMetric
      2 from maestro.trainer.models.florence_2 import train, TrainingConfiguration

File /Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/maestro/__init__.py:9
      6 except importlib_metadata.PackageNotFoundError:
      7     __version__ = \"development\"
----> 9 from maestro.lmms.gpt4 import prompt_image
     10 from maestro.markers.sam import SegmentAnythingMarkGenerator
     11 from maestro.postprocessing.mask import (
     12     compute_mask_iou_vectorized,
     13     mask_non_max_suppression,
   (...)
     18     refine_marks
     19 )

File /Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/maestro/lmms/gpt4.py:3
      1 import base64
----> 3 import cv2
      4 import numpy as np
      5 import requests

File /Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/cv2/__init__.py:181
    176             if DEBUG: print(\"Extra Python code for\", submodule, \"is loaded\")
    178     if DEBUG: print('OpenCV loader: DONE')
--> 181 bootstrap()

File /Applications/anaconda3/envs/maestro2/lib/python3.10/site-packages/cv2/__init__.py:153, in bootstrap()
    149 if DEBUG: print(\"Relink everything from native cv2 module to cv2 package\")
    151 py_module = sys.modules.pop(\"cv2\")
--> 153 native_module = importlib.import_module(\"cv2\")
    155 sys.modules[\"cv2\"] = py_module
    156 setattr(py_module, \"_native\", native_module)

File /Applications/anaconda3/envs/maestro2/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

ImportError: numpy.core.multiarray failed to import"
}

onuralpszr commented 2 months ago

@wjbmattingly It is looks like a conflict between cv2 and numpy, what I can suggest to you is remove "opencv-python-headless" and use conda version of opencv instead of pypi package, But I am pretty sure it works on normal python installations (I installed all of them via "brew" (brew install python@3.10,brew install python@3.9,brew install python@3.8)

I also mode github action that making sure it works for OSX as well. So I am pretty sure it works fine (but conda has different package source)

wjbmattingly commented 2 months ago

Ok! That did the trick, thanks so much!! (I uninstalled and used the conda version of opencv) I have a few new errors, but I think I can now debug these.

onuralpszr commented 2 months ago

Ok! That did the trick, thanks so much!! (I uninstalled and used the conda version of opencv) I have a few new errors, but I think I can now debug these.

Ok! That did the trick, thanks so much!! (I uninstalled and used the conda version of opencv) I have a few new errors, but I think I can now debug these.

I will work on conda of maestro so hopefully you can install easily, let's keep this issue open but let me change title of issue to "conda related" bug so I can work on this too as well.

cc @SkalskiP

wjbmattingly commented 2 months ago

Thanks! That works for me. If you want me to do testing when you have this ready, let me know. More than happy to test it out.

roboflow / maestro