Open bitfort opened 5 years ago
The rules actually mention <system_desc_id>_<implementation_id>_<scenario>.json
in the title, as <system_desc_id>_<implementation_id>.json
in the text of that section. It seems that _<scenario>
is superfluous here?
Furthermore, no examples are provided ([ TODO David Kanter to add ]
). But surely "Starting weights filename?" is benchmark-specific? Should <implementation_id
be actually benchmark_id
? Or are we missing a whole level here?
fixed with this PR: https://github.com/mlperf/inference/pull/496
We now allow what the doc wants <system_desc_id>_<implementation_id>_<scenario>.json
but we also allow <system_desc_id>_<implementation_id>.json
Thanks @guschmue. I'm still not clear what implementation_id
is. For the code/
subdir, I had to set it to task+'_'+scenario
, where task
is image-classification
or object-detection
, to make the checker happy, but I'm not sure if it's right.
this says
code/benchmark/implementation_id
and in my case I used reference
as implementation_id. It is undefined what the structure below that. I did not want to have the full inference tree there so I created a README that says git clone https://github.com/mlperf/inference
Thanks @guschmue, I think I get it now.
Do you have structure like this then:
code/
mobilenet/
reference/
README.md
resnet/
reference/
README.md
...
where each README.md
is identical? It's even worse in my case where for one implementation (image-classification-tflite
) I have dozens of models (mobilenet*
) and hence duplicate leaf README.md
s.
It seems it would be better to swap the levels:
code/
reference/
mobilenet/
README.md
resnet/
README.md
(although this doesn't solve the problem of duplicate README.md
s...)
Yes, I agree. That also would enable us to use code/implementation_id for everything as default that does not explicitly list a model. v0.7 I guess @petermattson .
The rules look for a file of: "__.json" then submission checker looks for a file of "_.json":
impl = system_file[len(system_desc) + 1:-5]