search

KavrakiLab / vamp

SIMD-Accelerated Sampling-based Motion Planning
Other
164 stars 22 forks source link

All path costs are the same over multiple trials? #10

Closed a-nooj closed 5 months ago

a-nooj commented 5 months ago

I'm running evaluate_mbm.py with these args:

def main(
    robot: str = "panda",
    planner: str = "rrtc",
    dataset: str = "problems.pkl",
    problem: Union[str, List[str]] = ['bookshelf_thin', 'cage', 'table_under_pick', 'table_pick'],
    trials: int = 10,
    **kwargs,
    ):

Here are a few elements of df["initial_path_cost"].tolist()

[7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018, 7.398220539093018,
6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887, 6.815661430358887,
8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461, 8.15865707397461,
7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418, 7.185542106628418,
6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637, 6.976244926452637,
11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, 11.428613662719727, ...

Seems suspicious that RRTConnect gives the exact same solution cost over multiple trials of the same planning problem?

zkingston commented 5 months ago

VAMP by default uses the Halton sequence (implemented in https://github.com/KavrakiLab/vamp/blob/main/src/impl/vamp/random/halton.hh) to generate "random" numbers - but this sequence is deterministic and produces the same sequence of numbers every time. We found empirically that this provides better performance on every benchmark we tested (which has also been observed in the literature, e.g., [1]). We also have an implementation of an xorshift random number generator (https://github.com/KavrakiLab/vamp/blob/main/src/impl/vamp/random/xorshift.hh), but you will need to go into the RRT-Connect code (https://github.com/KavrakiLab/vamp/blob/main/src/impl/vamp/planning/rrtc.hh) and replace the RNG used. Multiple trials are used to reduce hardware noise.

[1] Janson, Lucas, Brian Ichter, and Marco Pavone. "Deterministic sampling-based motion planning: Optimality, complexity, and performance." The International Journal of Robotics Research 37.1 (2018): 46-61. https://journals.sagepub.com/doi/pdf/10.1177/0278364917714338