How to reproduce experiments as in the paper

[marcoCalipari]

Hello!

I successfully retrieved the test dataset from UCLA Box and ran the system. I executed the evaluation script, which logs the scores for various experiments.

However, I encountered a couple of issues that I hope you can help clarify:

1. Lack of Visual Confirmation:

• The evaluation script currently only logs the scores for the experiments, but there is no clear visual feedback on the success or failure of the attack. It would be helpful to have a way to visually confirm the success of the attack instead of relying solely on the log files.

2) Visualizing the Results:

• I attempted to use the Visualize.py script for this purpose, but it seems it is intended for the full OPV2V dataset. However, the test dataset doesn't seem to have the required "train/validate/test" folder structure.

• Additionally, since the provided code uses pre-trained models, I'm unsure if the full OPV2V dataset is necessary.

Given the current codebase, could you clarify: Using the test dataset, What is the correct pipeline for visualizing the before and after attack images or LiDAR point clouds?

• Is it possible to visualize the attack's effects on the test dataset without downloading the full OPV2V dataset? If so, could you give some guidance on how to modify or use the Visualize.py script or any other method?

Any help would be greatly appreciated!

Thank you!

[zqzqz]

Hi @marcoCalipari. The evaluate.py should draw some figures about attacks and defenses under the result folder, by default /result. You can also look at the functions in /mvp/visualize. scripts/visualize.py is kind of deperated, sorry for that.

Basically, in scripts/evaluate.py, it does

from mvp.visualize.attack import draw_attack
from mvp.visualize.defense import visualize_defense, draw_roc