Problem with evaluation only working for atom_datasedt.json

[miguelriemoliveira]

FYI @Kazadhum

[Kazadhum]

We found an issue with the datasets saved after calibration. The optimized transformations were overwritten by their initial (i.e. pre-calibration) versions. We suspect this led to mountainous errors when running the evaluation. Working on a fix.

[Kazadhum]

Looks like this wasn't the problem after all

[miguelriemoliveira]

Hi @Kazadhum ,

found the problem. The transforms related to the patterns are not being copied from the train dataset to the mixed dataset. That's the difference.

I never thought about doing that because in most cases the estimations of the pattern poses cannot be used from the train to the test dataset. For example when the pattern is not fixed, collection 000 in the train and collection 000 in the test may not have the same transformation for a pattern pose.

But I guess I could copy from the train to the test if the transform is fixed ...

[miguelriemoliveira]

In fact, the only case I can think of where we actually want to copy the pattern poses from train to test when producing the mixed dataset is in the single_rgb_evaluation script.

Thus, I will add code separately to the single_rgb_evaluation script and leave the getMixedDataset function unchanged since this one is used by many other evaluations.

[miguelriemoliveira]

Implemented. Not sure if this copy should be done all the time. As I said before, the majority of cases will not require this.

Because of this, implemented a command line argument in the single_rgb_evaluation script that runs this or not. The argument is called copy_pattern_transform or -cpt. By default its false so if the riwmpbot_real we should use it.

[miguelriemoliveira]

Now we have always the same error, regardless of using atom_calibration.json or dataset.json.

[miguelriemoliveira]

Ran calibration for all collection in the merged dataset with this command:

clear && rosrun atom_calibration calibrate  -json ~/datasets/riwmpbot_real/merged/dataset.json -uic  \
-v -jsf "lambda x: True"  \                 
-csf "lambda x: int(x) not in [11,27,32, 34,45,57]"

Got great results either with always using atom_calibration.json or using the dataset.json as test.

[miguelriemoliveira]

So I guess the problem i really solved.

Thanks @Kazadhum for the GREAT help. A whole day of debugging. At least it paid off.