TensorFlow Lite Implementation and TPU offloading

[torinriley]

My thought is to begin integrating a Raspberry Pi equipped with Tensor Processing Units (TPUs) (e.g., Google Coral USB Accelerator) into our robot’s system. This addition would offload computationally intensive tasks, such as running custom TensorFlow Lite models for vision processing or machine learning (ML), allowing the REV Control Hub to focus on core robot control and communication with the Field Management System (FMS).

Connect the Raspberry Pi to the REV Control Hub via USB or UART.

• Attach TPUs (e.g., Google Coral USB Accelerator) to the Raspberry Pi for model inference.
• Power the Pi and TPU using the robot’s power distribution system.

  Software Integration:

• Raspberry Pi:
• Set up TFLite runtime with TPU support.
• Load and run ML models, sending inference results back to the Control Hub via USB or UART.
• FTC Control Hub:
• Implement a communication handler in the FTC app to request model inference and process results.

  Data Flow:

• The FTC app sends raw data (e.g., camera frames or commands) to the Raspberry Pi.
• The Raspberry Pi runs the ML model on the TPU and returns processed results (e.g., detected object coordinates).

You could integrate a method to handle request to the raspberry pi,

import java.io.InputStream;
import java.io.OutputStream;
import android.hardware.usb.UsbManager;
import android.hardware.usb.UsbDeviceConnection;
import android.hardware.usb.UsbDevice;

public class RaspberryPiHandler {
    private UsbDeviceConnection connection;
    private OutputStream outStream;
    private InputStream inStream;

    public RaspberryPiHandler(UsbManager usbManager, UsbDevice device) {
        connection = usbManager.openDevice(device);
        if (connection != null) {
            outStream = connection.getFileDescriptor();
            inStream = connection.getFileDescriptor();
        }
    }

    // Send inference request to the Pi
    public void sendInferenceRequest() throws IOException {
        if (outStream != null) {
            outStream.write("INFER\n".getBytes());
        }
    }

    // Read inference result from the Pi
    public String getInferenceResult() throws IOException {
        byte[] buffer = new byte[1024];
        int bytesRead = inStream.read(buffer);
        return new String(buffer, 0, bytesRead).trim();
    }
}

And integration with teleop (or likely auto) like this,

import com.qualcomm.robotcore.eventloop.opmode.OpMode;

@TeleOp(name = "PiIntegrationExample", group = "TeleOp")
public class PiIntegrationExample extends OpMode {
    private RaspberryPiHandler piHandler;

    @Override
    public void init() {
        UsbManager usbManager = (UsbManager) hardwareMap.appContext.getSystemService(Context.USB_SERVICE);
        UsbDevice device = findRaspberryPiDevice(usbManager); // Implement device discovery logic
        if (device != null) {
            piHandler = new RaspberryPiHandler(usbManager, device);
            telemetry.addData("Status", "Raspberry Pi connected");
        } else {
            telemetry.addData("Status", "Raspberry Pi not found");
        }
    }

    @Override
    public void loop() {
        try {
            if (piHandler != null) {
                // Request inference
                piHandler.sendInferenceRequest();

                // Get and process the inference result
                String result = piHandler.getInferenceResult();
                telemetry.addData("Inference Result", result);

                // Example: Use result for robot decision-making
                String[] parts = result.split(" ");
                double x = Double.parseDouble(parts[0]);
                double y = Double.parseDouble(parts[1]);
                if (x < 100) {
                    // Move left
                } else if (x > 200) {
                    // Move right
                }
            }
        } catch (Exception e) {
            telemetry.addData("Error", e.getMessage());
        }

        telemetry.update();
    }

    private UsbDevice findRaspberryPiDevice(UsbManager usbManager) {
        for (UsbDevice device : usbManager.getDeviceList().values()) {
            // Add logic to match the Pi's USB Vendor and Product IDs
            return device;
        }
        return null;
    }
}

[cheesebroccoli]

We are not allowed to use rasberry pi but we can use the camera that is allowed for tensorflow lite