The rcp checker needs to know which benchmark it should be checking the RCPs for. It finds the benchmark name in a surprising and undocumented way: by parsing it out of the FOLDER command line argument. But this means that if you run the RCP checker to check some logs in the current working directory (or in a directory that isn't named exactly correctly), it will fail with the confusing log message:
python3 -m mlperf_logging.rcp_checker --rcp_usage=<usage> --rcp_version=<version> .
...
ERROR - Cannot find any RCPs, RCP test FAILED`
My recommendation would be to find the benchmark name from the submission_benchmark log lines (:::MLLOG ... "key": "submission_benchmark", "value": "<benchmark_name>).
Another possible solution would be to add a new command line option (--benchmark <benchmark_name>, or something like that).
In any case, if the benchmark_name can't be deduced, the error message should be that the benchmark_name can't be deduced, rather than Cannot find any RCPs.
The rcp checker needs to know which benchmark it should be checking the RCPs for. It finds the benchmark name in a surprising and undocumented way: by parsing it out of the FOLDER command line argument. But this means that if you run the RCP checker to check some logs in the current working directory (or in a directory that isn't named exactly correctly), it will fail with the confusing log message:
My recommendation would be to find the benchmark name from the
submission_benchmarklog lines (:::MLLOG ... "key": "submission_benchmark", "value": "<benchmark_name>).Another possible solution would be to add a new command line option (
--benchmark <benchmark_name>, or something like that).In any case, if the benchmark_name can't be deduced, the error message should be that the benchmark_name can't be deduced, rather than
Cannot find any RCPs.