Create a rosbag anonymizer tool

xmfcx commented 3 months ago

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I've agreed with the maintainers that I can plan this task.

Description

The Autoware Foundation seeks to develop a tool that anonymizes camera data within rosbags, specifically targeting the blurring of faces and license plates to maintain privacy. This initiative aims to enable the secure sharing of rosbags containing camera data amongst member companies and the wider community.

Purpose

The primary goal is to ensure the privacy of individuals captured in camera data shared within the Autoware ecosystem. By creating a tool that can anonymize sensitive information in rosbags, we facilitate a safer, privacy-compliant exchange of data that can be used for research, development, and testing of autonomous vehicle technologies.

Possible approaches

Fork the autodistill project to the Autoware Foundation's repository, as it is under the Apache 2.0 license and could serve as a starting point.
Manually test the SAM model for its effectiveness in anonymizing faces and license plates within camera data.
Based on the results, develop a tool (either as a standalone application or integrated within the autodistill project) that inputs a rosbag and outputs an anonymized version of the same data, with faces and license plates blurred.

Definition of done

[X] The autodistill project is successfully forked to the Autoware Foundation repository.
[ ] Some test data from people with consent are uploaded for testing.
[ ] The SAM model's effectiveness in blurring faces and license plates is manually tested and confirmed.
[ ] A decision is made on whether the anonymization tool will be standalone or integrated into autodistill.
[ ] The tool is developed and tested to ensure it accurately anonymizes faces and license plates in rosbag camera data.
[ ] Documentation is created to guide users on how to use the tool.

xmfcx commented 2 months ago

https://github.com/knzo25/rosbag2_language_sam

StepTurtle commented 2 months ago

In last situation we decide won't use autodistill anymore because it don't have any additional things from original DINO and SAM for us.

Instead of this, we used Grounding DINO and SAM from original repositories and we added a image classification method OpenCLIP to validate Grounding DINO results. The working scheme is as follows:

Project Link: https://github.com/leo-drive/rosbag2_anonymizer

system

The tool reads the images from a ROS2 bag file and feeds them to the Grounding DINO with original images. Grounding DINO takes an image and a list of prompts as input and finds the objects described by the prompts in the image.
Grounding DINO cannot find all objects properly, so we validate objects with the image classification model OpenCLIP. OpenCLIP takes objects and a list of prompts as input and returns a matching score for the object and prompts.
After validating the bounding box results, SAM segments all objects and finds masks. Then, the segmentation results are blurred and written to a new bag file.

Additional of these things, we want to add one more validation part. The new validation method will check whether some objects should be inside other objects or not. For example, a license plate should be inside of the car but should not be inside of human.

Also I will add detailed outputs, for now you can check this bag file which anonymized with our tool:

https://drive.google.com/drive/folders/1jUZw7Um3b-I_ivYrOWdBSnHTqWnZd7cL?usp=sharing

Definition of done

[ ] Add new validation part which checks the object positions
[ ] Add detailed outputs

xmfcx commented 2 months ago

@StepTurtle could you test https://github.com/knzo25/rosbag2_language_sam with the same data and compare them?

cc. @knzo25

I'm expecting the comparison to be a playback of the anonymized rosbag camera image which as a video shared here.

StepTurtle commented 2 months ago

@StepTurtle could you test https://github.com/knzo25/rosbag2_language_sam with the same data and compare them?

cc. @knzo25

I'm expecting the comparison to be a playback of the anonymized rosbag camera image which as a video shared here.

Here is the results:

https://youtu.be/7YeGVJ9ulL8
https://youtu.be/dQJeWQoZzec (one of the tool has low hz, maybe you can ignore this video)

In the video:

Left rqt window shows this tool: https://github.com/leo-drive/rosbag2_anonymizer

🔴 red boxes represents license plate
🟢 green boxes represents human faces

Right rqt window shows this tool: https://github.com/knzo25/rosbag2_language_sam

🟣 purple boxes represents license plate
🟢 green boxes represents cars

StepTurtle commented 2 months ago

Additionally, a validation component has been added to https://github.com/leo-drive/rosbag2_anonymizer to verify the object positions. You can view the results here:

https://youtu.be/xs3m2SlBnBg

Do you have any ideas or suggestions on what we can do in the upcoming stages?

xmfcx commented 2 months ago

I can read the text, blur is not enough.

There are so many places where the plates are not blurred well enough.

What happens if you look for license plates with low score threshold and if the plate is inside the vehicle for validation?

Blurring classes

Classes for the license plate detection

Parent classes

car
bus
truck
minibus
motorcycle
trailer
utility vehicle
tractor
golf cart
semi-truck
moped
scooter

Child class

license plate

Classes for the pedestrian face detection

Parent classes

person
child

Child class

human face

StepTurtle commented 2 months ago

@xmfcx

I can read the text, blur is not enough.

There are so many places where the plates are not blurred well enough.

I changed the blur parameters, I guess it is okey now.

https://youtu.be/w6lR1x4jtVg (with your child - parent labels)

For this question following schema could be helpful

system The first step of validation involves running OpenClip. OpenClip will return results similar to the following: - Assuming you have input prompts such as: ["license plate", "car", "face"] - The output will look like this: [0.95, 0.4, 0.1] If the score for the corresponding label is greater than 0.9, it will be selected as valid. In the second validation step, we verify whether the label is inside of the parent. If it resides within one of the parent categories, it must satisfy one of the following conditions: - Is the score for the corresponding label the highest among the scores? - Is the score greater than 0.3? > What happens if you look for license plates with low score threshold and if the plate is inside the vehicle for validation? For your example, license plate must have score greater than 0.3 or the highest score for the corresponding label the highest among the scores.

xmfcx commented 2 months ago

binary_classification

photo attribution from unsplash

My problem is with the false negatives, also known as, missed detections.

Does your proposal reduce FNs?

StepTurtle commented 2 months ago

When we implemented this proposal, it didn't have a direct impact on FNs, but it allowed us to lower DINO threshold.

By reducing DINO threshold, we're able to detect more objects, including some that were previously classified as FNs. Also reducing DINO threshold will return a lot of FP and we aim to determine these FPs with proposal

mitsudome-r commented 2 months ago

@StepTurtle We can put the repository under AWF GitHub organization. Please make sure that you are not violating the license term of all the codes/models that you used.

StepTurtle commented 1 month ago

@StepTurtle We can put the repository under AWF GitHub organization. Please make sure that you are not violating the license term of all the codes/models that you used.

@mitsudome-r @xmfcx we forked repository couple time ago.

But currently, I don't have write access. Could you give me a access to this repository? I believe I can create PRs, but I would prefer to push directly to the main branch as there might not be anyone to review for now. If this isn't acceptable, I'll create a PR whenever I need to update the code.

StepTurtle commented 1 month ago

https://github.com/autowarefoundation/autoware_rosbag2_anonymizer

StepTurtle commented 1 month ago

I am sharing the videos which shows the current results:

After labeling and training YOLOv8, we combined YOLOv8 and DINO to find bounding boxes and results improved.

StepTurtle commented 1 month ago

Hi @xmfcx,

The tool have usage instructions in the project README. Should we also add a user guideline for the tools in the Autoware documentation. And instruction for how to publish new public dataset with Autoware community.

xmfcx commented 1 month ago

@StepTurtle under here: https://autowarefoundation.github.io/autoware-documentation/main/datasets/

it would be nice to have a separate page, dedicated to data anonymization.

mitsudome-r commented 1 week ago

@mitsudome-r will find someone to test this tool.

autowarefoundation / autoware