Searching with "track" command

[jp3spinelli]

• rpi-deep-pantilt version: 1.2.0
• Python version: 3.7.3
• TensorFlow version: 2.2.0
• Operating System: Raspbian

I am trying to integrate my own custom model into this repository. I have successfully gotten my model to work with the "detect" and "track" commands. However, I am trying to create a script that allows for the pan/tilt HAT to pan from the -90deg position to the +90deg position, in increments of 60deg, while running the track command (tilt stays at 25deg position). Additionally, once the camera successfully detects an object and tracking begins, I want the panning command to stop. Any suggestions? Now, my custom model is set to detect a Raspberry Pi camera (it only detects well at about 1ft away). What changes do I need to make to the PID scripts? I was thinking of creating a new Python script to combine the panning commands with the tracking command, but it doesn't work unless there is a way to differentiate when the tracking command is issued vs when the tracking begins. I figure the best way to do it is going into the repo files but I don't really know where to start. I noticed when the track command is issued it starts in a high tilted position and slowly moves down. Perhaps changing that script would work.

I am new to coding and software so please be in-depth.

[TCIII]

@jp3spinelli,

This Paul McWhorter Jetson Xavier NX tutorial may provide you with some insights into scanning for an object that goes out of the field of view so that the program can reinstate object tracking. His hardware setup uses a Jetson Xavier NX and two Pi V2 cameras mounted on pan/tilt gimbals. Be aware that McWhorter's Python code uses jetson-inference and OpenCV unlike Leigh's Python code. I have a setup similar to McWhorter's and it works as advertised. Regards, TCIII

[jp3spinelli]

@TCIII,

Thanks for the advice! Unfortunately, using his work for my purposes would take quite a bit of redoing work. I'm simply looking to tweak her repository files. I primarily need help identifying the files in interest, like when I utilize the "rpi-deep-pantilt detect" command, what file is that referencing?

Thanks, JP

[TCIII]

@jp3spinelli ,

The manage.py program in rpi-deep-pantilt/control/ is used to control the gimbal servos during "detect" so I would start there.

Regards, TCII

[jp3spinelli]

Thank you. I've been looking at it, and I'm starting to understand the process a bit more. I attached an LED to my system to show when the tracking is taking place. Now, I made the following changes to this function (marked w/**):

def set_servos(pan, tilt):
    # signal trap to handle keyboard interrupt
    signal.signal(signal.SIGINT, signal_handler)

    while True:
        pan_angle = -1 * pan.value
        tilt_angle = tilt.value

        # if the pan angle is within the range, pan
        if in_range(pan_angle, SERVO_MIN, SERVO_MAX):
            pth.pan(pan_angle)
            **GPIO.output(8,GPIO.HIGH)**
        else:
            logging.info(f'pan_angle not in range {pan_angle}')

        if in_range(tilt_angle, SERVO_MIN, SERVO_MAX):
            pth.tilt(tilt_angle)
            **GPIO.output(8,GPIO.HIGH)**
        else:
            logging.info(f'tilt_angle not in range {tilt_angle}')

But for some reason, the LED remains on from the time I issue the command until I kill it (I added the command to turn off the LED when ctrl+c is issued), regardless of whether or not an object is being tracked.

[TCIII]

@jp3spinelli, As long as either the pan or tilt angle stay within SERVO_MIN and SERVO_MAX your led is never going to turn off besides the fact that you have no code to turn it off if the pan and tilt angles are out of range. You might insert "GPIO.output(8, GPIO.LOW) after the "else" which will turn off the led because the pan or tilt angle is out of range and tracking has stopped until either the pan or tilt angle is within range.

Regards, TCIII

[jp3spinelli]

For some reason that wasn't working, so I went into visualization.py and added the command to line 339 under the visualize_boxes_and_labels_on_image_array function as follows:

def visualize_boxes_and_labels_on_image_array(
        image,
        boxes,
        classes,
        scores,
        category_index,
        instance_masks=None,
        instance_boundaries=None,
        keypoints=None,
        track_ids=None,
        use_normalized_coordinates=False,
        max_boxes_to_draw=20,
        min_score_thresh=.5,
        agnostic_mode=False,
        line_thickness=4,
        groundtruth_box_visualization_color='black',
        skip_scores=False,
        skip_labels=False,
        skip_track_ids=False):

**GPIO.output(8,GPIO.HIGH)**

    # Create a display string (and color) for every box location, group any boxes
    # that correspond to the same location.
    box_to_display_str_map = collections.defaultdict(list)
    box_to_color_map = collections.defaultdict(str)

Now, it lights once tracking begins, which I wanted. But once it is lit, it does not go out until the command is killed, even if the object leaves the FOV. Any ideas on a fix?

[jp3spinelli]

@TCIII It seems I was wrong about my previous statement, for some reason it isn't working now. Back to square one. Do you have any advice on the visualization.py route though? I want the LED to be lit as long as the bounding box is present, as that means tracking is also commencing.

[jp3spinelli]

@TCIII After messing with the visualization.py script I got the LED to fire once a bounding box is made. It doesn't turn off when the object is out of the FOV but I'm not worried about that for now. Do you have any suggestions on getting this scan to work? Your manager.py route seems to be the best. I'll continue looking into it tomorrow but if you have any advice let me know.

[jp3spinelli]

@leigh-johnson within the manager.py script, would editing the set_servos function be the best place to accomplish the scanning? I'm trying to program it to scan until the object is detected. I'm thinking of adding a statement to make 60deg rotations there and back three full cycles unless the LED is on (symbolizing tracking).

[jp3spinelli]

@TCIII @leigh-johnson I added the following into manager.py:

import pantilthat as pth
import signal
import sys
**import time**
**import RPi.GPIO as GPIO**

**from rpi_deep_pantilt.detect.util.visualization import draw_bounding_box_on_image**
from rpi_deep_pantilt.detect.camera import run_pantilt_detect
from rpi_deep_pantilt.control.pid import PIDController

#Specifying GPIO Pin name
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(8,GPIO.OUT)

scan_on=True

And the secret sauce:

def set_servos(pan, tilt):
    # signal trap to handle keyboard interrupt
    signal.signal(signal.SIGINT, signal_handler)

    while True:
        pan_angle = -1 * pan.value
        tilt_angle = tilt.value

        # if the pan angle is within the range, pan
        if in_range(pan_angle, SERVO_MIN, SERVO_MAX):
            pth.pan(pan_angle)
        else:
            logging.info(f'pan_angle not in range {pan_angle}')

        if in_range(tilt_angle, SERVO_MIN, SERVO_MAX):
            pth.tilt(tilt_angle)
        else:
            logging.info(f'tilt_angle not in range {tilt_angle}')

        while scan_on == True:
            pth.servo_one(90)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(30)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(-30)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(-90)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(-30)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(30)
            pth.servo_two(25)
            time.sleep(10)

            pth.servo_one(90)
            pth.servo_two(25)
            time.sleep(10)

            continue

While my visualization.py looks like:

.
.
def draw_bounding_box_on_image(image,
                               ymin,
                               xmin,
                               ymax,
                               xmax,
                               color='red',
                               thickness=4,
                               display_str_list=(),
                               use_normalized_coordinates=True):

    GPIO.output(8,GPIO.HIGH)
    scan_on = False
    return scan_on

    draw = ImageDraw.Draw(image)
.
.

So, the scanning works and it's constantly detecting, but when an object is detected, it doesn't break out of the while loop like I'd hope. It's weird because the LED turns on once an object is detected and a bounding box has been drawn, but the 'scan_on = False' doesn't do anything, and they're in the same function.

[TCIII]

@jp3spinelli, Even though Leigh Johnson has done an excellent job integrating available hardware and software to provide very professional detect and track capabilities using the Rpi 4B, CORAL, and the Pimoroni Gimbal assembly, if you want to get serious about detect, track, and scan (search), you need to move to the Jetson Xavier NX and dual Pi V2.1 Cameras on independent gimbals as seen here.

Regards, TCIII

[sn3kyJ3di]

Can you elaborate on why the nano is better?

Thank you,

Aaron Fish

On Aug 13, 2020, at 9:36 AM, TCIII notifications@github.com wrote:

 @jp3spinelli, Even though Leigh Johnson has done an excellent job integrating available hardware and software to provide very professional detect and track capabilities using the Rpi 4B, CORAL, and the Pimoroni Gimbal assembly, if you want to get serious about detect, track, and scan (search), you need to move to the Jetson Xavier NX and dual Pi V2.1 Cameras on independent gimbals as seen here.

Regards, TCIII

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[TCIII]

@sn3kyJ3di, First off, I didn't mention the Nano, I posted about the Xavier NX. However, the Nano is equal to or better than the Rpi 4B with the CORAL. The Nano can run two Pi Cameras at 10 fps, not just one, or one camera at 30 fps, and does not require an outboard GPU like the CORAL or NCS. The Xavier NX can run two Pi Cameras at 20 Fps and perform search (scan), detect, and track as shown in the Paul McWhorter video tutorial mentioned in my post above. Regards, TCIII

[sn3kyJ3di]

TCII

Thanks, I’m sorry I had nano on the brain for some reason. I have been eyeing up the Xavier now for a while, price point is a barrier to entry for me, but maybe I need to look again. That thing is powerful, I wonder how well it would do with cameras connected via IP instead of the CSI ports?

On Aug 13, 2020, at 11:20 AM, TCIII notifications@github.com wrote:

@sn3kyJ3di https://github.com/sn3kyJ3di, First off, I didn't mention the Nano, I posted about the Xavier NX. However, the Nano is equal to or better than the Rpi 4B with the CORAL. The Nano can run two Pi Cameras at 10 fps, not just one, or one camera at 30 fps, and does not require an outboard GPU like the CORAL or NCS. The Xavier NX can run two Pi Cameras at 20 Fps and perform search (scan), detect, and track as shown in the Paul McWhorter video tutorial mentioned in my post above. Regards, TCIII

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/leigh-johnson/rpi-deep-pantilt/issues/42#issuecomment-673541297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM742VTPQWL6DAHR65GI4LLSAQAENANCNFSM4PTQ4Y5Q.

[leigh-johnson]

Hey y'all! @TCIII is correct in his assessment that this repo is intended to be a reference implementation for hobbyists.

Performance tuning will vary based on your problem domain (and also your skillset!). In this repo, I provide a couple of off-the-shelf models from TensorFlow's object detection model zoo . These are an ok starting point for folks without much experience with computer vision, but they're not going to be suitable for most industry applications.

If I were looking to optimize for real-time inference speed, maximize accuracy AND keep cost-per-unit down ... here's what I would do:

1. Train a model with a quantization-aware configuration. The example models in this repo use post-training quantization (basically coercing floating-point model weights to uint8), which tends to degrade accuracy more severely than a quantization-aware training routine.
2. Sample frame buffer using a lower-level C/C++ MMAL API instead of Python PiCamera lib
3. Run TensorFlow Lite inference in C++, following this document to incorporate edgetpu.h

[jp3spinelli]

I completely understand. Thanks for this great piece of programming. I just wanted to fine tune it to scan, which is out of my domain but that's how you learn. I've been working on a solution for quite some time now, and I'm getting pretty close if you want to look at my in-depth question on Stack Overflow, it'd mean a lot :)

(https://stackoverflow.com/questions/63607576/scope-issue-between-scripts-due-to-multiprocessing)