Support OpenMV Camera - Githubissues

A few notes:

When connected to Dexter, it shows up as a /dev/ttyACM# where number starts a 0, then 1, etc... depending on the number of devices connected. SSH into Dexter, do an ls /dev/tty* with the device disconnected, then connect, do the command again, and notice the difference.

Connect at 115200 N 8 1 This works to test connectivity:

stty -echo -F /dev/ttyACM0 ospeed 115200 ispeed 115200 raw
cat -v < /dev/ttyACM0 & cat > /dev/ttyACM0

enter Ctrl+C to stop, then fg 1 and Ctrl+C again. For more on serial interface, see: https://github.com/HaddingtonDynamics/Dexter/wiki/Dexter-Serial-Peripherals

When connected, press Ctrl+C to stop any running program ( works nicely from PuTTY, but you can't do this from the cat trick shown above need to find a work around, probably echo $\cc>/dev/ttyACM0), Ctrl+D to reload / sync the files from the file folder, and import main to run the main.py file from the folder.

To find and mount the flash drive to re-program the OpenMV cam, you can use blkid to show available block devices without the device connected, then connect it and use the command again. The new device is the camera. It will probably appear as /dev/sda1: SEC_TYPE="msdos" UUID="4621-0000" TYPE="vfat". To access the contents, make a folder: mkdir /mnt/openmvcam then mount the device: sudo mount /dev/sda1 /mnt/openmvcam/ and the files will appear at /mnt/openmvcam. Edit main.py to change the cameras code. When finished, be sure to umount /mnt/openmvcam

Again, Ctrl+D causes the camera to reset and reload the file from the drive if you are connected via the serial terminal. Then you can type import main to run it. Ctrl+C to break it.

The following program allows you to ask the Camera to look for a number of things. For a Job Engine program to interface with this see: https://github.com/HaddingtonDynamics/Dexter/issues/60#issuecomment-605376411

import sensor, image, time, math, ubinascii
from pyb import USB_VCP

orange_threshold =( 18, 69, 40, 86, 0, 72)
orange_blob_pixel_thres = 50

q_tip_threshold =( 192, 255)
q_tip_width_min = 12 #decrease if more than ~8" away
q_tip_height_min = 30 #decrease if more than ~8" away or tilted > 45'
q_tip_width_max = 21 #increase if closer than ~6" or tilted > 45'
q_tip_height_max = 40 #increase if closer than ~6"
q_tip_area_min = q_tip_width_min * q_tip_height_min
q_tip_pixels_min = int(q_tip_area_min * 0.7)
q_tip_pixel_max = 0.9 #some pixels must be black, no rectangles
setup_delay = 200

sensor.reset() # Initialize the camera
mode = b'?' #default starting mode. Provide correct setup for this mode here:
#sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # QVGA (320x240) for more accurate length detection
sensor.set_auto_whitebal(True)
sensor.skip_frames(time = setup_delay)
clock = time.clock() # Tracking frame rate
#take a starting picture to define img
img = sensor.snapshot().compress()

f_x = (2.8 / 3.984) * 160 # find_apriltags defaults to this if not set
f_y = (2.8 / 2.952) * 120 # find_apriltags defaults to this if not set
c_x = 160 * 0.5 # find_apriltags defaults to this if not set (the image.w * 0.5)
c_y = 120 * 0.5 # find_apriltags defaults to this if not set (the image.h * 0.5)

# barcode type lookup table
barcode_type = {
    image.EAN2: "EAN2",
    image.EAN5: "EAN5",
    image.EAN8: "EAN8",
    image.UPCE: "UPCE",
    image.ISBN10: "ISBN10",
    image.EAN13: "EAN13",
    image.ISBN13: "ISBN13",
    image.I25: "I25",
    image.DATABAR: "DATABAR",
    image.DATABAR_EXP: "DATABAR_EXP",
    image.CODABAR: "CODABAR",
    image.CODE39: "CODE39",
    image.PDF417: "PDF417",
    image.CODE93: "CODE93",
    image.CODE128: "CODE128"
}

def barcode_name(code):
    # if the code type is in the dictionary, return the value string
    if code.type() in barcode_type.keys():
        return barcode_type[code.type()]
    # otherwise return a "not defined" string
    return "NOT DEFINED"

idno = "0"
try:
    idno = open('/id.txt').read()
except:
    print("[{\"error\":\"id not set in id.txt file\"}]")

usb = USB_VCP()

while(True):
    clock.tick() # Track elapsed milliseconds between snapshots().

    #ID
    cmd = usb.recv(1, timeout=10)
    if (cmd != mode):
        if (cmd == b'!'): #setup AND take a picture
            #sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
            #sensor.set_framesize(sensor.QVGA)   # Set frame size to QVGA (320x240)
            #sensor.skip_frames(time = 2000)     # Wait for settings take effect.
            #img = sensor.snapshot().compress()
            shrink = 50
            if (mode == b'b'): #barcode image is too big
                shrink = 20 #need to shrink it more
            img_data = str(ubinascii.b2a_base64(img.compress(shrink)))[1:]
            print("[{\"image_length\":"+str(img.size())+", \"data\":"+img_data+"}]")
            sensor.skip_frames(time = 2000)     # Wait for data to xmit
            cmd = mode #continue on in the same mode

        if (cmd == b'a'): #setup to look for April Tags
            #presetting for april tags does NOT work. Always errors over memory allocation.
            sensor.set_pixformat(sensor.GRAYSCALE)
            sensor.set_framesize(sensor.QQVGA) # QQVGA 160x120 faster
            sensor.skip_frames(time = setup_delay)
            #print("[{\"mode\":\"april tag\"}]")

        if (cmd == b'b'): #setup for Bar Codes
            sensor.set_pixformat(sensor.GRAYSCALE)
            sensor.set_framesize(sensor.VGA) #VGA 640x480
            #sensor.set_windowing((640, 240))
            sensor.set_auto_gain(False)
            sensor.set_auto_whitebal(False)
            sensor.skip_frames(time = setup_delay)

        if (cmd == b'o'): #setup for orange object
            sensor.set_pixformat(sensor.RGB565) # Format is RGB565.
            sensor.set_framesize(sensor.QQVGA) # QQVGA 160x120 faster
            sensor.set_auto_gain(False) # Turn off automatic auto gain.
            #By default, in color recognition, be sure to turn off white balance.
            sensor.set_auto_whitebal(False)
            sensor.skip_frames(time = setup_delay)

        if (cmd == b'q'): #setup for q-tips
            sensor.set_pixformat(sensor.GRAYSCALE)
            sensor.set_framesize(sensor.QVGA) # QVGA (320x240) for length detection
            #sensor.set_auto_whitebal(False) #This helps with light level variation
            sensor.skip_frames(time = setup_delay)

        if (cmd > b' '): # generally, just set to the mode to any valid command.
            mode = cmd

    sensor.skip_frames(time = 100) # Wait a titch so we don't jam up the queue

    if (mode == b'?'):
        print("[{\"camname\": \""+idno+"\"}]")
        mode = b' '
        continue

    if (mode == b' '): #be quite mode
        sensor.skip_frames(time = 100) # don't overheat
        continue

    if (mode == b'a'): #look for April Tags
        img = sensor.snapshot().lens_corr(1.8)
        april_tags = 0
        april_tags = img.find_apriltags(fx=f_x, fy=f_y, cx=c_x, cy=c_y)
        if len(april_tags) > 0:
            #print("April Tag Found")
            print(april_tags)
        continue

    if (mode == b'b'): #look for Bar Code
        img = sensor.snapshot()
        barcodes = 0
        barcodes = img.find_barcodes()
        if len(barcodes) > 0:
            #print("Bar code found")
            print(barcodes)
            #usb.send(barcodes)
            img.draw_rectangle(barcodes[0].rect(), 127, 4)
            img.draw_rectangle(0,0, 640, 50, 0, 1, True)
            img.draw_string(10, 10, barcodes[0].payload(), 127, 4, 0, 0, False)
        continue

    if (mode == b'o'): #look for Orange Blob
        img = sensor.snapshot()
        orange_blobs = 0
        orange_blobs = img.find_blobs([orange_threshold])
        if len(orange_blobs) > 0:
            #print("orange_blobs found")
            #print(len(orange_blobs))
            size = orange_blobs[0][4]
            #print(size)
            if (size > orange_blob_pixel_thres):
                print(orange_blobs)
                img.draw_rectangle((orange_blobs[0].rect())) # rect
            else:
                img.draw_cross(orange_blobs[0].cx(),orange_blobs[0].cy(), 0, 10, 2) # cx, cy
        continue

    if (mode == b'q'): #look for a q-tip
        img = sensor.snapshot()
        blobs = 0
        blobs = img.find_blobs([q_tip_threshold]
            , area_threshold=q_tip_area_min
            , pixels_threshold=q_tip_pixels_min
            , x_stride = 4
            , y_stride = 2
            ) #look for the stick and tip
        if len(blobs) > 0:
            x = blobs[0][0]
            y = blobs[0][1]
            w = blobs[0][2]
            h = blobs[0][3]
            #a = (math.atan2(h,w)/(2*math.pi))*360
            blobs = img.find_blobs([q_tip_threshold]
                , roi=(x,y,w,q_tip_width_min)
                , area_threshold=q_tip_width_min*q_tip_width_min
                , pixels_threshold=int(q_tip_width_min*q_tip_width_min*0.7)
                ) #now just look for the tip
            if len(blobs) > 0:
                img.draw_rectangle(x,y,w,h,color=128,thickness=2) # rect
                #img.draw_rectangle(0,0, 320, 20, 0, 1, True)
                #img.draw_string(10, 10, str(a), 255, 2, 0, 0, False)
                x = blobs[0][0]
                y = blobs[0][1]
                w = blobs[0][2]
                h = blobs[0][3]
                pixels = blobs[0][4]
                if (pixels < int(w * h * q_tip_pixel_max) and
                    w <= q_tip_width_max and
                    w >= q_tip_width_min #and
                    #h <= q_tip_height_max and
                    #h >= q_tip_height_min
                    ):
                    print(blobs)
                    img.draw_rectangle((blobs[0].rect())) # rect
                else:
                    img.draw_cross(blobs[0].cx(), blobs[0].cy(), 0, 10, 2) # cx, cy
            else:
                img.draw_line(x,y,x+w,y+h,0,2)
        continue

    #If nothing else
    print("[\"mode\":\"" + str(mode) + "\"]")
    sensor.skip_frames(time = 1000) # Wait so we don't jam up the queue

Supporting a single OpenMV cam isn't too hard, given the above. However, supporting multiple cameras at the same time is a bit more difficult. The port on which the device appears is assigned by the OS and we have no control over it. There is no serial number for the device in the port information.

A great way to know which device is sending information is to assign each device an ID number. You can do this by programming it with a custom program, or by simply adding this to your program:

idno = "0"
try:
    idno = open('/id.txt').read()
except:
    print("[{\"error\":\"id not set in id.txt file\"}]")

And place a file named id.txt in the cameras flash drive with whatever id you like as the contents of the file. Then the ID numbers can easily be changed in the field.

Then the device can be asked to identify itself, or start off by sending it's ID over and over on power up until it's told to send something else.

But what if you don't want to send commands to the device, and just want it to return the device ID along with the data coming back? e.g. if you do blob detection or read a barcode, and you are already printing that object out as JSON to Dexter, wouldn't it be great to just add the ID number of the device into that string? Sadly, these are built in objects (blob, barcode, april_tag, etc...) and it is totally impossible to:

Add an attribute to a built in object. e.g. "AttributeError: 'blob' object has no attribute 'id'"
Extend or change the class from which the built in object is made e.g. "NameError: name 'blob' isn't defined"
Edit in ANY way the data contained in the object. e.g. blobs[0][8] = ido "TypeError: 'blob' object doesn't support item assignment"

A way to "get 'er done" is to just unpack the data and print your own JSON:

                x = blobs[0].x()
                y = blobs[0].y()
                w = blobs[0].w()
                h = blobs[0].h()
                pixels = blobs[0].pixels()

                print("[{\"id\":"+str(idno)
                    +", \"x\":"+str(x)
                    +", \"y\":"+str(y)
                    +", \"w\":"+str(w)
                    +", \"h\":"+str(h)
                    +"}]")

It is possible to make your own object, and you can extract from the main object the items you want and place them in your object. However, this will print as e.g. <mutobj object at 30006920> You must provide your own method for printing your object.

class mutobj(object):
    #def __init__(self, ):
    #    setattr(self, a, v)
    def __str__(self):
        o = []
        for a,v in self.__dict__.items():
            o.append( "\""+a+"\":"+str(v) )
        return "[{" + ", ".join(o) + "}]"
    pass
....
                x = blobs[0].x()
                y = blobs[0].y()
                w = blobs[0].w()
                h = blobs[0].h()
                b = mutobj()
                b.idno = idno; b.x = x; b.y = y; b.w=w; b.h=h
                print(b)

Wouldn't it be nice if the constructor for mutobj could take an object to copy, and iterate it's attributes, copying them into itself? Sadly,

the built in objects to not support the .__dict__ object which appears to be how object attribute iteration is done in Python.

Perhaps we could pass in an array of the attributes we want to the mutobj constructor? Nope.

The built in objects don't really have attributes at all. They have array indexes, and methods which return specific indexes in the array.

So to get an attribute by NAME you would have to send an object with names and indexes. E.g. send me the [0] item and call it "X" e.g.


class mutobj(object):
    def __init__(self, obj, attr):
        for n, i in attr.items():
            setattr(self,n,obj[i])
    def __str__(self):
        o = []
        for a,v in self.__dict__.items():
            o.append( "\""+a+"\":"+str(v) )
        return "[{" + ", ".join(o) + "}]"
    pass

blobattr = {"x":0,"y":1,"w":2,"h":3} # etc.. pick the things you want and what to call them.

...

                    b = mutobj(blobs[0], blobattr)
                    b.idno = idno 
                    print(b)

which, frankly, is a stupid amount of work to jump through just to get an id back with the data. It probably takes less ram and fewer cycles to just edit the damn string. e.g.

                    print("[{\"idno\":"+str(idno)+", "+str(blobs)[2:])

and that will work with any of the built in objects. It only adds the id to the first returned blob, but the other methods had the same limitation.

HaddingtonDynamics / Dexter

Support OpenMV Camera #107