search

prophesee-ai / openeb

Open source SDK to create applications leveraging event-based vision hardware equipment
https://www.prophesee.ai/metavision-intelligence/
164 stars 38 forks source link

Feature request: native python data loader or Mac support for MetavisionSDK #58

Closed shiba24 closed 1 year ago

shiba24 commented 2 years ago

Hi,

I'm using Prophesee Gen3 camera now, and having a feature request to you. Since I couldn't find any forum or feature request form for Metavision SDK, I'm writing here.

What I want to do in the end: load .raw data on my Mac (M1, 2020). And currently this is not possible in my understanding.

What I want to request: either of the below options:

Sincerely,

Shintaro

biphasic commented 2 years ago

@shiba24 very much supported.

lbristiel-psee commented 2 years ago

Hello @biphasic and @shiba24 ,

currently there is no support of MacOS but you can try to compile OpenEB project on MacOS by yourself. In addition, if you are looking for minimal functionnalities (like reading events from a RAW file), you don't have to compile the whole OpenEB (for example you can skip tests and Python3 bindings).

Essentially, my advices would be:

mkdir build && cd build
cmake .. -DBUILD_TESTING=OFF -COMPILE_PYTHON3_BINDINGS=OFF
cmake --build . --config Release --parallel 4

This should allow you to get the main C++ libraries and Python modules available. For example the main Open Samples and Application should be working properly.

Hope this helps!

shiba24 commented 2 years ago

@lbristiel-psee Thank you for the reply.

Hmmmm, that sounds quite a workaround and not sure the right way to go. Did you (or anyone around you) ever try to compile on Mac?

Also I don't understand why people need to install pytorch etc (with very specific version constraints) just in order to load the data.

lbristiel-psee commented 2 years ago

Did you (or anyone around you) ever try to compile on Mac?

Yes, I tried and it worked without error (some warning during compilation, but not blocking)

I don't understand why people need to install pytorch etc (with very specific version constraints) just in order to load the data

pytorch is needed if you want to use Machine Learning. You don't need it to load RAW data. So you can skip the "Machine Learning" and "Python Bindings" of the install guide.

biphasic commented 2 years ago

A Python package that can load your file formats, has a C++ or other compiled backend for speed and works on all OS would be amazing! Something like h5py, which is just really great to use.

shiba24 commented 2 years ago

@lbristiel-psee Ok, I tried below (I guess I need to enable python bindings, right?):

mkdir build && cd build
cmake .. -DBUILD_TESTING=OFF -DPYTHON3_SITE_PACKAGES=<path>/venv/lib/python3.9/site-packages

And I got this error:

CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON3_SITE_PACKAGES

How can I resolve this?

@biphasic One possibility is to make our own with referring this, if they don't have any plan.

lbristiel-psee commented 2 years ago

@shiba24 what you have is a warning, not an error. Did it prevent the compilation to work? When I tried on MaOS, I did not specify PYTHON3_SITE_PACKAGES (I used default python3). And I did not compile the Python3 bindings because I just wanted to test "reading a RAW file" (like you mentioned) for which we don't need the Python Bindings, so I launched cmake .. -DBUILD_TESTING=OFF -COMPILE_PYTHON3_BINDINGS=OFF

shiba24 commented 2 years ago

@lbristiel-psee Actually what I want is load .raw data in python (sorry if it wasn't clear to you). I am doing like the below, and the compilation works now. But still I can't import in my python environment on Mac.

> cmake .. -DBUILD_TESTING=OFF -DPYTHON3_SITE_PACKAGES=/Users/shiba24/work/venv_openeb/lib/python3.8/site-packages -DPython3_EXECUTABLE=/Users/shiba24/work/venv_openeb/bin/python3.8

-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building METAVISION_SDK_MODULES_OPEN modules : base;core;driver;core_ml;ui
-- Building METAVISION_SDK_MODULES_ADVANCED modules : analytics;calibration;cv;ml;3dview;cv3d
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libusb-1.0'
--   Found libusb-1.0, version 1.0.26
-- Found LibUSB: /opt/homebrew/Cellar/libusb/1.0.26/include/libusb-1.0  
-- Found Boost: /opt/homebrew/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: program_options filesystem timer chrono thread 
-- Found Python3: /Users/shiba24/work/venv_openeb/bin/python3.8 (found version "3.8.14") found components: Interpreter Development Development.Module Development.Embed 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Performing Test HAS_FLTO_THIN
-- Performing Test HAS_FLTO_THIN - Success
-- Found pybind11: /opt/homebrew/include (found version "2.10.0")
-- Found OpenCV: /opt/homebrew/Cellar/opencv/4.6.0 (found version "4.6.0") found components: core highgui imgproc videoio imgcodecs calib3d objdetect 
-- Found Boost: /opt/homebrew/lib/cmake/Boost-1.80.0/BoostConfig.cmake (found version "1.80.0") found components: filesystem system 
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Generating internal bias files.
--  - vdac18_8m1.calib
--  - gen3_idac.calib
--  - gen3_idac_n_thick.calib
--  - gen3_idac_n_thin.calib
--  - gen3_idac_p_thick.calib
--  - gen3_idac_p_thin.calib
--  - gen3_idac_railp.calib
-- Found OpenGL: /Library/Developer/CommandLineTools/SDKs/MacOSX13.0.sdk/System/Library/Frameworks/OpenGL.framework   
-- Found GLEW: /opt/homebrew/lib/cmake/glew/glew-config.cmake  
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/shiba24/work/build_openeb/openeb/build
> cmake --build . --config Release -- -j 4

But when I try using python library, it can not locate it.

> source utils/scripts/setup_env.sh.in
> python3.8 -c 'from metavision_core.event_io.raw_reader import RawReader'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'metavision_core'

I feel it's almost there - can you help me on this?

lbristiel-psee commented 2 years ago

Hi,

you are maybe not configuring the environment properly with the command source utils/scripts/setup_env.sh.in The script you are using seems to be utils/scripts/setup_env.sh.in whereas you should use build/utils/scripts/setup_env.sh And check the content of the file after sourcing it and make sure it was sourced properly (i.e. check the PYTHON_PATH value was properly initialized).

Hope this helps, Laurent

shiba24 commented 2 years ago

Thank you, import succeeded! Now I have another error and still cannot load .raw file 🙏 :

In [1]: from metavision_core.event_io import RawReader
In [2]: raw_reader = RawReader(<some_path>)
In [3]: raw_reader.load_n_events(100)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In [4], line 1
----> 1 raw_reader.load_n_events(100)

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:485, in RawReader.load_n_events(self, n_events)
    475 """
    476 Loads a batch of *n_events* events.
    477 
   (...)
    482     events (numpy array): structured numpy array containing the events.
    483 """
    484 n_events = int(n_events)
--> 485 self._advance(n_events)
    487 # if all events are decoded, there is only this classes buffer left.
    488 if self._decode_done:

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:168, in RawReaderBase._advance(self, n_events, delta_t, drop_events)
    165         self._seek_event = n_events
    167 while not (self._decode_done or _are_enough_ev_loaded(final_time, n_events)):
--> 168     self._run()

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:140, in RawReaderBase._run(self)
    138     return False
    139 data = self.i_events_stream.get_latest_raw_data()
--> 140 self.i_decoder.decode(data)
    142 return True

RuntimeError: NumPy type info missing for N10Metavision7EventCDE

This error happens for EventsIterator too. Also, RawReader.seek_event function crashed:

In [13]: raw_reader.seek_event(1000)
[1]    39539 segmentation fault  ipython

I'm at the head of the main branch (commit 8e57704b195d653bdd2075a0d48b6539e159d976, with minor modification on cmake/custom_functions/python3.cmake (specifying PYTHON3_DEFAULT_VERSION "3.8"). Do you know how I can fix this?

shiba24 commented 2 years ago

@lbristiel-psee Hi, could you take a look into this? Even some workaround helps me a lot. This becomes a hard blocker for my research now.

lbristiel-psee commented 2 years ago

Hello @shiba24 . I don't reproduce this problem on our supported platforms. Maybe this is something specific to your config/platform/dependencies. Maybe you should consider switching to Ubuntu and see if the same problem happens.

shiba24 commented 2 years ago

Hi @lbristiel-psee , let me clarify. You did not reproduce this issue in your Mac? (So you can use python bindings at least in your Mac environment?)

lbristiel-psee commented 2 years ago

no, I was not able to reproduce on Ubuntu. My access to MacOs is limited.

shiba24 commented 2 years ago

Hm, so probably no one has ever tested pybindings on Mac. Ubuntu works fine with OpenEB. I will take a look into this by myself..

And now, it comes to my original request of this thread. Can you prioritize the Mac OS support? At least two users (@biphasic and I) request to use Mac. Recent Mac (M1) CPU is much faster than Ubuntu. So apart from GPU and ROS, Mac is actually the first option for the development in many cases. (Because OpenEB / Prophesee cam do not require ROS.)

lbristiel-psee commented 1 year ago

Well noted @shiba24 . We have this support in our radar and will let you know when it gets included in a future release.

biphasic commented 1 year ago

@shiba24 please check out the native data loader that we released, it works without issues on Mac: https://github.com/fabhertz95/expelliarmus

shiba24 commented 1 year ago

Hey @biphasic - thank you for the awesome work :) I have a couple of ideas to contribute, I will take a look when I find time. Anyway love this open-sourcing!

Rennylex commented 1 week ago

Thank you, import succeeded! Now I have another error and still cannot load .raw file 🙏 :

In [1]: from metavision_core.event_io import RawReader
In [2]: raw_reader = RawReader(<some_path>)
In [3]: raw_reader.load_n_events(100)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In [4], line 1
----> 1 raw_reader.load_n_events(100)

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:485, in RawReader.load_n_events(self, n_events)
    475 """
    476 Loads a batch of *n_events* events.
    477 
   (...)
    482     events (numpy array): structured numpy array containing the events.
    483 """
    484 n_events = int(n_events)
--> 485 self._advance(n_events)
    487 # if all events are decoded, there is only this classes buffer left.
    488 if self._decode_done:

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:168, in RawReaderBase._advance(self, n_events, delta_t, drop_events)
    165         self._seek_event = n_events
    167 while not (self._decode_done or _are_enough_ev_loaded(final_time, n_events)):
--> 168     self._run()

File ~/work/build_openeb/openeb/sdk/modules/core/python/pypkg/metavision_core/event_io/raw_reader.py:140, in RawReaderBase._run(self)
    138     return False
    139 data = self.i_events_stream.get_latest_raw_data()
--> 140 self.i_decoder.decode(data)
    142 return True

RuntimeError: NumPy type info missing for N10Metavision7EventCDE

This error happens for EventsIterator too. Also, RawReader.seek_event function crashed:

In [13]: raw_reader.seek_event(1000)
[1]    39539 segmentation fault  ipython

I'm at the head of the main branch (commit 8e57704b195d653bdd2075a0d48b6539e159d976, with minor modification on cmake/custom_functions/python3.cmake (specifying PYTHON3_DEFAULT_VERSION "3.8"). Do you know how I can fix this?

HI! I am having the same issue RuntimeError: NumPy type info missing for N10Metavision7EventCDE, can you tell me how did you fix it on your Mac? thanks!