ModuleNotFoundError: No module named 'pyrealsense2'

[didi1150]

| Camera Model | { D435 } | | Platform | Raspberry Pi| | SDK Version | {2.0}
| Language | {python }

I am trying to make tensorflow lite work with the intel realsense cam d435 with raspberry pi. But I get the message

e.py --modeldir=Sample_TFLite_Model/ Traceback (most recent call last): File "TFLite_detection_intel_realsense.py", line 15, in import pyrealsense2 as rs ModuleNotFoundError: No module named 'pyrealsense2'

I tried rebuilding the viewer, and setting the python path in the bashrc file, but it didn't work

[MartyG-RealSense]

Hi @didi1150 Can you confirm please that you have built the RealSense Python wrapper, as it is not installed automatically when librealsense is installed. You have to specify for librealsense to include it when building the SDK or install it separately by building it from source code after librealsense has been installed.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/python

[didi1150]

I tried, but it didn't work, it still says the module cannot be found. @MartyG-RealSense

[MartyG-RealSense]

Could you go to the librealsense > build > wrappers > python folder on your Raspberry Pi and see whether there is a file there that starts with the name 'pyrealsense2' and ends in '.so' please like the file highlighted in the image below. The middle part of the filename will vary depending upon the Python version installed on your particular Pi.

[didi1150]

Yes i copied it next to my script and renamed it pyrealsense2.so but it didnt work either. I tried resetting the pythonpath and adding the librealsense.so file too, but it didnt work. I followed the acrobotic guide step by step when installing everything

[didi1150]

@MartyG-RealSense could I add you somewhere else, eg discord because I dont want to wait a whole day for a response

[MartyG-RealSense]

I apologize for the delay in responding further. Support at locations outside of official RealSense forums cannot be provided.

The pyrealsense2.so file should not be renamed. If you are using Python 3, you should use the one generated by the build process that has a complex long name, like the one highlighted in the image above. This file should be copy and pasted unchanged into the same folder location as your script, along with the librealsense2.so file.

[didi1150]

@MartyG-RealSense I tried this one too, it didnt work

[MartyG-RealSense]

You mention in https://github.com/IntelRealSense/librealsense/issues/9865#issuecomment-946613064 that you used the Acrobotic guide to install the SDK and the RealSense Python wrapper. You are therefore using the Raspbian OS and not Ubuntu.

When you set the pythonpath in the bashrc file, did you source the bashrc file afterwards with the command source ~/.bashrc to apply the changes?

Some RealSense users have also recommended adding the pythonpath definition to the end of the bashrc file.

[didi1150]

yes I did

[didi1150]

I did nearly everything I found on the internet, copy the files to the script, setting paths, and rebuilding the wrapper. It didn't work

[didi1150]

How do I build it from source

[MartyG-RealSense]

If you are using the Acrobotic guide to install on Raspbian then you are already installing from source code after having obtained the librealsense SDK's files from the librealsense GitHub repo.

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_raspbian.md#update-udev-rule

The next reference that I will refer you to is https://github.com/IntelRealSense/librealsense/issues/4375

[didi1150]

/usr/bin/ld: cannot open output file pyrealsense2.cpython-37m-arm-linux-gnueabihf.so.2.49.0: Too many levels of symbolic links collect2: error: ld returned 1 exit status make[2]: [wrappers/python/CMakeFiles/pyrealsense2.dir/build.make:298: wrappers/python/pyrealsense2.cpython-37m-arm-linux-gnueabihf.so.2.49.0] Error 1 make[1]: [CMakeFiles/Makefile2:1173: wrappers/python/CMakeFiles/pyrealsense2.dir/all] Error 2

[MartyG-RealSense]

I do not personally have knowledge of the subject of symbolic links on Raspberry Pi unfortunately, and this particular Too many levels of symbolic links error does not seem to be one that has occurred before in previous RealSense cases.

Looking again at the Acrobotic guide for installing on Raspbian, I know that some RealSense users have experienced problems when using the protobuf version suggested by the guide (and the even older version suggested by Intel's official guide for Pi 3b) and have had a smoother build experience when using a more modern version such as protobuf 3.14

[didi1150]

ok I rebuild everything and fixed it. I just deleted the build folder and did everything again

[MartyG-RealSense]

Is it working correctly now, please?

[didi1150]

yes. Do you know I can combine the tensorflow lite code here: https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi/blob/master/TFLite_detection_webcam.py with the distance measurement on the official intel guide? cuz im kind of new to python, and Im wayy behind my deadline @MartyG-RealSense

[MartyG-RealSense]

It may be easier for you to use the RealSense SDK's TensorFlow wrapper, which makes use of the pyrealsense2 wrapper that you have already successfully installed and provides an object detection tutorial and tutorials for other functions.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/tensorflow

Tutorial 2 of the wrapper demonstrates how to query XYZ coordinates of each detected object and to separate individual coordinates in meters. So you could conceivably read the Z coordinate to obtain the detected object's distance.

[didi1150]

@MartyG-RealSense How do I add tensorflow to windows 10? Because I want to test my code on the pc and I can never complete the https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi guide, because it always has errors on the step 2e: Building the tensorflow package. Is there another more easier way to install tensorflow?

[didi1150]

@MartyG-RealSense I found another guide, but now I get this error:

[...] from google.protobuf import descriptor as _descriptor ModuleNotFoundError: No module named 'google.protobuf'

I already have protobuf installed from the acrobotic guide, what do I do now?

[MartyG-RealSense]

The problem with protobuf may be caused when it has first been installed by one method and then installed by another method, as suggested in the discussion in the link below.

https://github.com/protocolbuffers/protobuf/issues/6135#issuecomment-546241914

In regard to installing TensorFlow on Windows, you can use the Python pip install command pip install --upgrade tensorflow according to the official TensorFlow installation guide at https://www.tensorflow.org/install/pip

[didi1150]

ok I fixed the above issue, now I get this: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:132] HadoopFileSystem load error: libhdfs.so: cannot open shared object file: No such file or directory Traceback (most recent call last): File "MobileNetSSDwithRealSenseTF.py", line 14, in import visualization_utils as vis_util File "/home/pi/MobileNet-SSDLite-RealSense-TF/visualization_utils.py", line 4, in import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements ModuleNotFoundError: No module named 'matplotlib'

[MartyG-RealSense]

Have you installed matplotlib on your computer with pip install matplotlib

https://www.tutorialspoint.com/how-to-install-matplotlib-in-python

[didi1150]

I am doing things now on raspberry again, and yes, I have installed matplotlib. @MartyG-RealSense

[MartyG-RealSense]

Are you still using this guide on Pi:

https://github.com/EdjeElectronics/TensorFlow-Lite-Object-Detection-on-Android-and-Raspberry-Pi

[didi1150]

ye but im trying to run this https://github.com/PINTO0309/MobileNet-SSDLite-RealSense-TF

[MartyG-RealSense]

I researched your question extensively but did not find a solution unfortunately. Have you made any progress?

[didi1150]

No

[didi1150]

But I have to go anyways, Ill come back later

[didi1150]

@MartyG-RealSense ok im trying to run the official tensorflow wrapper doc, but I don't understand how to generate a model: https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API#run-network-in-opencv

I have downloaded the Faster-RNN Inception-v2 file, but what do I do now? I cannot run the scripts because they say I dont have the libraries, and I dont want to search for all the libraries for 2 hours.

[MartyG-RealSense]

The instructions for Example 1 state that a file called frozen_inference_graph.pb must also be placed in your working directory. The RealSense TensorFlow wrapper's instructions refer to extraction of this file. I therefore found it located in a compressed tar.gz file downloaded when clicking on the 'Weights' link of the Faster-RCNN-Inception-V2 model.

[didi1150]

When I clicked the link, nothing happened. Nothing even downloaded. Could you send me the file?

[MartyG-RealSense]

If you right-click on the link and select Open as new window (or whatever your browser's menu option is to open a page in a new tab) then the file should hopefully start downloading on the new page. If it does not, I found that clicking the page reload icon at the top of the browser window starts the download.

I have not been able to attach the file to this GitHub message, so if the file still will not download for you then please send me an email and I will reply with the file attached.

[didi1150]

What worked for me was, copying the link, and then pasting it in another window. Just Open as a new window didn't do it for me.

[didi1150]

The question is, will the stuff in this tensorflow run on raspberry?

[didi1150]

And is it possible to install tflite_runtime on windows?

[MartyG-RealSense]

Googling for tensorflow raspberry pi shows extensive cases of TensorFlow and TensorFlow Lite being able to run on Raspberry Pi. I have not personally seen a case where the RealSense TensorFlow wrapper has been used on Pi, but that does not mean that it is not possible.

Edje Electronics (that you referred to earlier in this case) has a YouTube video at the link below that demonstrates how to run TensorFlow Lite on a Raspberry Pi for object detection.

https://www.youtube.com/watch?v=aimSGOAUI8Y

In regard to installing TensorFlow Lite runtime on Windows, it seems to be a complex process. The best guide that I found is in the link below.

https://heartbeat.comet.ml/working-through-a-tensorflow-lite-tutorial-on-windows-10-e27ee0e8b8cc

[didi1150]

Ok, I think I haven't explained my problem properly:

I am trying to run EdgeElectronics scripts on my pc so I can develop faster, because I can use the Pycharm IDE here. The reason I want to do this is because I want to experiment with his code because it already worked for me on the raspberry pi. I personally don't like Intel's official guide/tutorial to the tensorflow wrapper because it doesn't mention tflite, which runs a lot quicker. With the official guide I only reached 1 fps at max on my pc, while on the other hand, edgeelectronics scripts ran with 4 fps on the pi. The only problem is that I cannot make pycharm import tflite_runtime without errors. I tried to import tflite_runtime with pip, and it didn't work. I found out that I already had tflite_runtime in the site-packages in my python folder installed, so I just copied them over to my workspaces site-packages. But now it says this:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\Didi1150\Desktop\TensorFlow\TFLite_detection_webcam.py", line 144, in interpreter = Interpreter(model_path=PATH_TO_CKPT) File "C:\Users\Didi1150\Desktop\TensorFlow\venv3\lib\site-packages\tflite_runtime\interpreter.py", line 348, in init _interpreter_wrapper.CreateWrapperFromFile( SystemError: <built-in method CreateWrapperFromFile of PyCapsule object at 0x0000022AFAEA88A0> returned a result with an error set

Ive had this problem before, but I don't know how to fix it. It has something to do with tflite_runtime importing its own numpy I think. I debugged the code and if I commented the importing tflite_runtime part it imported tensorflow and everything worked, just - slower. I really want to get this to work

[didi1150]

ok I think I combined the code successfully, however I didn't manage to get the distance:

#

This code is based off the TensorFlow Lite image classification example at:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/examples/python/label_image.py

#

Import packages

import os import argparse from turtle import color

import cv2 import numpy as np import sys import time from threading import Thread import importlib.util import pyrealsense2 as rs

class VideoStream: """Camera object that controls video streaming from the Picamera"""

def __init__(self, resolution=(640, 480), framerate=30):
    self.stream = cv2.VideoCapture(0)
    ret = self.stream.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG'))
    ret = self.stream.set(3, resolution[0])
    ret = self.stream.set(4, resolution[1])

    (self.grabbed, self.frame) = self.stream.read()

    self.stopped = False

def start(self):
    Thread(target=self.update, args=()).start()
    return self

def update(self):
    # Keep looping indefinitely until the thread is stopped
    while True:
        if self.stopped:
            self.stream.release()
            return

        (self.grabbed, self.frame) = self.stream.read()

def read(self):
    return self.frame

def stop(self):
    self.stopped = True

parser = argparse.ArgumentParser() parser.add_argument('--modeldir', help='Folder the .tflite file is located in', required=True) parser.add_argument('--graph', help='Name of the .tflite file, if different than detect.tflite', default='detect.tflite') parser.add_argument('--labels', help='Name of the labelmap file, if different than labelmap.txt', default='labelmap.txt') parser.add_argument('--threshold', help='Minimum confidence threshold for displaying detected objects', default=0.5) parser.add_argument('--resolution', help='Desired webcam resolution in WxH. If the webcam does not support the resolution entered, errors may occur.', default='1280x720') parser.add_argument('--edgetpu', help='Use Coral Edge TPU Accelerator to speed up detection', action='store_true')

args = parser.parse_args()

MODEL_NAME = args.modeldir GRAPH_NAME = args.graph LABELMAP_NAME = args.labels min_conf_threshold = float(args.threshold) resW, resH = args.resolution.split('x') imW, imH = int(resW), int(resH) use_TPU = args.edgetpu

pkg = importlib.util.find_spec('tflite_runtime') if pkg: from tflite_runtime.interpreter import Interpreter

if use_TPU:
    from tflite_runtime.interpreter import load_delegate

else: from tensorflow.lite.python.interpreter import Interpreter

if use_TPU:
    from tensorflow.lite.python.interpreter import load_delegate

if use_TPU: if (GRAPH_NAME == 'detect.tflite'): GRAPH_NAME = 'edgetpu.tflite'

CWD_PATH = os.getcwd()

PATH_TO_CKPT = os.path.join(CWD_PATH, MODEL_NAME, GRAPH_NAME)

PATH_TO_LABELS = os.path.join(CWD_PATH, MODEL_NAME, LABELMAP_NAME)

with open(PATH_TO_LABELS, 'r') as f: labels = [line.strip() for line in f.readlines()]

if labels[0] == '???': del (labels[0])

if use_TPU: interpreter = Interpreter(model_path=PATH_TO_CKPT, experimental_delegates=[load_delegate('libedgetpu.so.1.0')]) print(PATH_TO_CKPT) else: interpreter = Interpreter(model_path=PATH_TO_CKPT)

interpreter.allocate_tensors()

input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() height = input_details[0]['shape'][1] width = input_details[0]['shape'][2]

floating_model = (input_details[0]['dtype'] == np.float32)

input_mean = 127.5 input_std = 127.5

frame_rate_calc = 1 freq = cv2.getTickFrequency()

pipeline = rs.pipeline() config = rs.config() config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30) config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)

print("[INFO] Starting streaming...") pipeline.start(config) print("[INFO] Camera ready.")

time.sleep(1)

while True:

t1 = cv2.getTickCount()

frames = pipeline.wait_for_frames()
color_frame = frames.get_color_frame()
depth_frame = frames.get_depth_frame()

color_image = np.asanyarray(color_frame.get_data())
depth_image = np.asanyarray(depth_frame.get_data())

frame_rgb = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
frame_resized = cv2.resize(frame_rgb, (width, height))
input_data = np.expand_dims(frame_resized, axis=0)

if floating_model:
    input_data = (np.float32(input_data) - input_mean) / input_std

interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()

boxes = interpreter.get_tensor(output_details[0]['index'])[0]  # Bounding box coordinates of detected objects
classes = interpreter.get_tensor(output_details[1]['index'])[0]  # Class index of detected objects
scores = interpreter.get_tensor(output_details[2]['index'])[0]  # Confidence of detected objects

for i in range(len(scores)):
    if ((scores[i] > min_conf_threshold) and (scores[i] <= 1.0)):
        ymin = int(max(1, (boxes[i][0] * imH)))
        xmin = int(max(1, (boxes[i][1] * imW)))
        ymax = int(min(imH, (boxes[i][2] * imH)))
        xmax = int(min(imW, (boxes[i][3] * imW)))

        cv2.rectangle(color_image, (xmin, ymin), (xmax, ymax), (10, 255, 0), 2)

        object_name = labels[int(classes[i])]  
        label = '%s: %d%%' % (object_name, int(scores[i] * 100))
        labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.7, 2) 
        label_ymin = max(ymin, labelSize[1] + 10)
        cv2.rectangle(color_image, (xmin, label_ymin - labelSize[1] - 10),
                      (xmin + labelSize[0], label_ymin + baseLine - 10), (255, 255, 255),
                      cv2.FILLED)
        cv2.putText(color_image, label, (xmin, label_ymin - 7), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0),
                    2) 

cv2.putText(color_image, 'FPS: {0:.2f}'.format(frame_rate_calc), (30, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 0), 2,
            cv2.LINE_AA)

cv2.imshow('Object detector', color_image)

t2 = cv2.getTickCount()
time1 = (t2 - t1) / freq
frame_rate_calc = 1 / time1

if cv2.waitKey(1) == ord('q'):
    break

cv2.destroyAllWindows() pipeline.stop()

I don't know how to get the distance in this code.

[MartyG-RealSense]

It's great to hear that you made progress!

As your script does not seem to be generating a point cloud, the coordinates stored in the numpy array are likely to be 2D pixel coordinates rather than 3D world coordinates.

The second example in the RealSense TensorFlow wrapper suggests generating a vertices-based point cloud with point_cloud.calculate(depth_frame) and then storing its vertices in a numpy array and retrieving the XYZ coordinates individually.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/tensorflow#part-2---augmenting-output-using-depth-data

[didi1150]

Thank you very much! It was a pleasure working with your tips. This issue can be closed. No further help is needed. :D

[MartyG-RealSense]

It's excellent to hear that you were successful, @didi1150 - thanks very much for the update!