search

mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) to make it easier to build and benchmark AI systems across diverse models, data sets, software and hardware
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
13 stars 20 forks source link

CM error when running MLPerf Submission Generation Script #538

Closed ChloeCrozier closed 1 week ago

ChloeCrozier commented 1 week ago

I am currently trying to generate a submission tree for my mlperf sdxl based on this documentation. This is the command I ran within the docker environment after running scc-base variant.

cm run script --tags=generate,inference,submission \ --clean \ --run-checker \ --tar=yes \ --env.CM_TAR_OUTFILE=submission.tar.gz \ --division=open \ --category=datacenter \ --env.CM_DETERMINE_MEMORY_CONFIGURATION=yes \ --run_style=test \ --adr.submission-checker.tags=_short-run \ --quiet \ --submitter=<...>

I received this error indicating that our results were invalid despite the benchmark completing without any errors/error logs.

image

Any suggestions on how to fix this?

arjunsuresh commented 1 week ago

Hi @ChloeCrozier are you using the latest CM4MLOps code? Ideally the "preprocessor" script should not be called for SCC24 code. May be can you try adding --env.CM_RUN_MLPERF_SUBMISSION_PREPROCESSOR=no to the run command?

ChloeCrozier commented 1 week ago

Thank you! I ended up being able to fix it by altering the user.conf parameters to be within the specified ranges.