search

carla-recourse / CARLA

CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms
MIT License
274 stars 61 forks source link

Update benchmarking example #193

Open jscanass opened 1 year ago

jscanass commented 1 year ago

I installed the package as recommend using pip install carla-recourse The version installed is carla-recourse==0.0.5 but I am not able to reproduce the benchmarking example. The error happen when importing evaluation module: Screen Shot 2022-10-04 at 11 27 42

xueyagaga commented 1 year ago

I installed the package as recommend using pip install carla-recourse The version installed is carla-recourse==0.0.5 but I am not able to reproduce the benchmarking example. The error happen when importing evaluation module: Screen Shot 2022-10-04 at 11 27 42

I also meet this BUG. The solution is easy. First, delete these codes (if you don't need the specific results):

import carla.evaluation.catalog as evaluation_catalog
evaluation_measures = [
    evaluation_catalog.YNN(benchmark.mlmodel, {"y": 5, "cf_label": 1}),
    evaluation_catalog.Distance(benchmark.mlmodel),
    evaluation_catalog.SuccessRate(),
    evaluation_catalog.Redundancy(benchmark.mlmodel, {"cf_label": 1}),
    evaluation_catalog.ConstraintViolation(benchmark.mlmodel),
    evaluation_catalog.AvgTime({"time": benchmark.timer}),
]

Second, change results = benchmark.run_benchmark(evaluation_measures) to: results = benchmark.run_benchmark() Then you can get all the metrics, such as: image