search

FORTH-ModelBasedTracker / MocapNET

We present MocapNET, a real-time method that estimates the 3D human pose directly in the popular Bio Vision Hierarchy (BVH) format, given estimations of the 2D body joints originating from monocular color images. Our contributions include: (a) A novel and compact 2D pose NSRM representation. (b) A human body orientation classifier and an ensemble of orientation-tuned neural networks that regress the 3D human pose by also allowing for the decomposition of the body to an upper and lower kinematic hierarchy. This permits the recovery of the human pose even in the case of significant occlusions. (c) An efficient Inverse Kinematics solver that refines the neural-network-based solution providing 3D human pose estimations that are consistent with the limb sizes of a target person (if known). All the above yield a 33% accuracy improvement on the Human 3.6 Million (H3.6M) dataset compared to the baseline method (MocapNET) while maintaining real-time performance
https://www.youtube.com/watch?v=Jgz1MRq-I-k
Other
840 stars 135 forks source link

Illegal instruction (core dumped) #79

Closed VisImage closed 2 years ago

VisImage commented 2 years ago

Just installed latest version. Ubuntu 20.4, running on CPU

MocapNET$ ./MocapNET2LiveWebcamDemo --from shuffle.web --frames 375 returns: Illegal instruction (core dumped)

MocapNET$ ./OpenCVTest --from shuffle.webm works fine.

Please advise. Thank you.

AmmarkoV commented 2 years ago

Illegal instruction means you are using a CPU that lacks specific instructions ! For example AVX Since the CMakeFile build script automatically builds all of my code for your exact CPU this is probably emitted by Tensorflow.. So what CPU are you using what Tensorflow version are you using ?

AmmarkoV commented 2 years ago

By the way I have provided a script to help you build your own Tensorflow version here

If you find it hard building tensorflow you are not alone and you should see this video "How To Make Package Managers Cry" -> https://www.youtube.com/watch?v=NSemlYagjIU#t=19m0s

VisImage commented 2 years ago

Thank you AmmarkoV, This is my CPU:
*-cpu product: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz vendor: Intel Corp. physical id: 1 bus info: cpu@0 size: 1196MHz capacity: 2801MHz width: 64 bits

AmmarkoV commented 2 years ago

Hello so yes as I imagined, your CPU is an i7 CPU, the first generation of i7 ( launched at 2009 so it is now 13 years old ) Your CPUs supported "instruction" set is SSE4.2 ( https://www.intel.com/content/www/us/en/products/sku/41316/intel-core-i7860-processor-8m-cache-2-80-ghz/specifications.html ) so that's why you get the illegal instruction error..!

This is also not a problem of my code but of Tensorflow which is developed by Google and not me :P

Tensorflow versions after 1.6 need AVX instructions by default ( see here : https://github.com/tensorflow/tensorflow/issues/19584 )

You need to either find or build your own version of Tensorflow 1 ( I have this script to help you with that ) or Tensorflow 2 ( I have this script ) or try running on a more recent CPU.. You can even use a cloud server like Google Collab

I can see that there are people that build Tensorflow 2 without AVX ( https://github.com/yaroslavvb/tensorflow-community-wheels/issues/174 ) so this is definitely possible

More information on building Tensorflow are here : https://www.tensorflow.org/install/lang_c

VisImage commented 2 years ago

Thank for the swift reply. I got the point. I am trying to try MocapNET on another PC with more recent CPU and on google Colab. I'll update with the out comes afterwards.

Thank you and stay tuned...

VisImage commented 2 years ago

Just tested on a Lenovo Y720, Ubuntu 20.4: *-cpu: product: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz

"./MocapNET2LiveWebcamDemo --from shuffle.webm --openpose --frames 375" Crashed with the following error:

terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.2.0) ../modules/highgui/src/window.cpp:376: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

Aborted (core dumped)

"./OpenCVTest --from shuffle.webm " works fine.

AmmarkoV commented 2 years ago

Hello this is a bug on my part somewhere ..

If you try ./MocapNET2LiveWebcamDemo --from shuffle.webm --openpose --frames 375 --show 3

it works ( notice the added --show 3)

AmmarkoV commented 2 years ago

Ok I found the problem, I actually introduced it today trying to fix Issue https://github.com/FORTH-ModelBasedTracker/MocapNET/issues/78

if you git pull and then rebuild it should work now with all visualizations..!

VisImage commented 2 years ago

YES, it works now :-) Thank you, Ammar.

A couple of more questions. 1)There are some tools included in your repo. Do you have any description of these tools, such as drawCSV, convertCSV3DToMocapNETInput, and so on. 2) Do you have any description on the CSV files referred in your repo? Is it for 2D keypoints, 3D keypoints, or may both? 3) We may migrate your kernel algorithm to Windows, if our validations on your algorithm are positive. Do you have any requirement for us to take into account in our implementation/migration?

AmmarkoV commented 2 years ago

1) This is an active research project so most of the utilities etc are written to accommodate a set of experiments and are self-explanatory, and deal with a specific workflow e.g. the convertCSV3DToMocapNETInput is a utility to dump from an internal conversion to files that can be consumed directly by the MocapNETFromCSV utility etc.. In any case this github repository is actually a snapshot of a private development repository and since this is a solo-maintained project there is no man-power available to properly document everything.. I am currently maintaining 3 different flavors of the same code in three different repositories and you can imagine what an effort this requires to keep everything playing smoothly at the same time.

2) CSV files are mainly used in this demo repository to encode 2D inputs that can then be fed to the MocapNET NN, if you want an example please check the mnet3 branch ( https://github.com/FORTH-ModelBasedTracker/MocapNET/tree/mnet3 ) and follow the commands given there to download and test the latest version that also regresses hands..! If you check on the intermediate CSV files etc you will quickly understand their format. Just be careful when doing the normalization of the 2D points ( read bullet 3 for more info on this )

3) The code is actually already working on windows through the WsL subsystem layer, so you don't need to migrate anything, you can already run most of it ( I am not sure if the visualization windows work on WsL though ) directly. I am currently developing a version that uses Tensorflow/ONNX/TensorRT runtimes so it will also be available on embedded devices soon for automotive purposes. Due to the low-dependency with no exotic compiler syntax commands "style" that I use for the C/C++ code it should be almost directly compilable/portable to Windows (except the experimental OpenGL stuff), so since tensorflow and OpenCV builds are available for Windows porting it should be straightforward, I just don't like windows (especially the newest versions) that's why I don't directly support it. If you want to do the port for commercial purposes please note the license of the project. Although I am its creator and developer its copyright belongs to the Foundation of Research and Technology Greece and they place restrictions on its use for non research purposes..! Finally there are a lot of caveats implementing it most of them are documented on the published papers that are listed in the main README.md, however an important consideration that I also mentioned in bullet point 2 is that very often the cause of regression quality problems is the aspect ratio of the input images/2D points. MocapNET is trained using a virtual camera that mimics a GoPro Hero4 camera @ 1920x1080 resolution, it works best on 2D joint constellations that follow this "aspect ratio distribution", since all points are normalized to (0..1,0..1) 2D points originating from let's say a camera with 640x480 (1.33 aspect ratio) have a much different "fingerprint" when they form an NSRM matrix when compared to let's say the 1920x1080 (1.77 aspect ratio), I have code that tries to embed 2D point clouds in larger frames to account for that, other than that for very good results the HCD module introduced @ ICPR2020 does a lot of work and can be configured during runtime to match different skeletons etc so there is a lot of potential for improvement, however these things are only partially exposed ( for example here )

AmmarkoV commented 2 years ago

I am also closing this issue since it is resolved, in case you have more problems/questions feel free to open a new one..! If you reply on a closed issue like this it is not guaranteed that I will see it since I get a ton of github notifications from many repositories and they often get lost