xyhuang
commented
3 years ago Infra WG:
- we might want to double check if the changes can work for different frameworks
- potential alternative solutions:
- log start/stop of weight initialization
- add a metadata specifying which layer is initialized, and the checker should verify if all required layers are present
mwawrzos
For each weights tensor, this entry should be logged right after the weights initialization is done. Here is an example in RNN-T reference: https://github.com/mlcommons/training/pull/430/commits/66d6c2dc784ff5b15641807a65036ffedd68af83#diff-eb3462d93ad4cb9033e2a2884ef241e7a866019ca804a0bc4a3453bcd96bf05cR105-R107
The compliance checker will validate if the number of entries in the log matches the reference. This way submitter can ensure if he reports all initializations. Here is an example for RNN-T checks: https://github.com/mlcommons/logging/blob/master/mlperf_logging/compliance_checker/1.0.0/closed_rnnt.yaml#L1-L16
The purpose of that change is to simplify the review process. Thanks to these log entries, a reviewer can quickly identify, which part of the code is responsible for determining initial tensor weight values.