search

pytorch / xla

Enabling PyTorch on XLA Devices (e.g. Google TPU)
https://pytorch.org/xla
Other
2.49k stars 480 forks source link

Run the verifier on Torchbench in the Nigthly CI. #8070

Open ysiraichi opened 1 month ago

ysiraichi commented 1 month ago

As far as I understand it, we have a nightly CI that runs Torchbench daily. I believe it would be beneficial to also run the verifier (as documented here), so as to catch, as early as, possible regressions.

cc @miladm @zpcore @JackCaoG

zpcore commented 1 month ago

Currently the CI script runs

python experiment_runner.py --suite-name=torchbench --accelerator=cuda --progress-bar --xla=PJRT --xla=None --dynamo=None --dynamo=openxla --dynamo=inductor

Hi @ysiraichi , correct me if I am wrong. Looks like we need to run the same command again with --verify flag to get the verification code for each configufation. Apart from that, we also need to add the field verification_code and verification_mean_rel_error etc in result_analyzer.py.

We also need to update our looker studio UI to display the verification result. I can take those action items later.

ysiraichi commented 1 month ago

That's about it. I don't think there's a verification_mean_rel_error anymore, though.