search

ctuning / ck-mlperf

This repository is outdated! Join the open MLPerf workgroup to participate in the development of the next generation of automation workflows for MLPerf benchmarks:
https://bit.ly/mlperf-edu-wg
BSD 3-Clause "New" or "Revised" License
32 stars 23 forks source link

Efficientnet-lite related changes (properly synced with master) #54

Closed slahiruk closed 3 years ago

ens-lg4 commented 3 years ago

@slahiruk , thank you for your contribution!

Let's try to keep to the existing conventions for signalling the quantized/non-quantized property:

1) we add a "quantized" or "non-quantized" tag to models - this allows us to fish out the desired one using tags 2) we also add two variables with specific names to the environment: one for the model's input data type ( ML_MODEL_INPUT_DATA_TYPE=fp32 ) and one for the model's internal data type ( ML_MODEL_DATA_TYPE=uint8 ).

At the first glance this looks like redundancy (the second implies the first), but the current version of CK kernel does not allow us to fish out things based on non-tag attributes, so for the time being we record this information twice.

slahiruk commented 3 years ago

Cool! I will do as mentioned :+1:

ens-lg4 commented 3 years ago

@slahiruk , A few more things that I noticed when trying to install one of the variations: (1) my system cannot find a dependent script: 4f99bd653a4cf94b mentioned in meta.json :

"use_scripts_from_another_entry": { "data_uoa": "4f99bd653a4cf94b", "module_uoa": "script" },

If it's a new entry, maybe it also has to be added?

(2) there is currently no "on_by_default": "yes" kind. It would be good to pick one, for example, lite0-int8 as the default one. This way if someone doesn't care about the specific kind, he can get at least something by default.

(3) I've noticed that the kind-datatype variations currently supported cover the whole cartesian product - all kinds (from lite0 to lite4) times {"fp32", "int8"}. You could make that section of the meta.json shorter if you independently describe 5 kind variations (from "kind.lite0" to "kind.lite4" - of which the latter could be "on_by_default": "yes") and also independently 2 datatype variations ("fp32" and "int8" - of which the latter could also be the default).

(4) expected square window sizes for different model kinds are different: lite0 expects 224x224 lite1 expects 240x240 lite2 expects 260x260 lite3 expects 280x280 lite4 expects 300x300 We have an established tag "side.X" for this (assuming the window is square). Please add tag "side.224" to extra_tags of "kind.lite0" variation, "side.240" to extra_tags of "kind.lite1" variation and so on.

Thanks for your help!

slahiruk commented 3 years ago

https://github.com/ctuning/ck-tensorflow/pull/141 @ens-lg4 corresponding pull request to address issue (1) mentioned in https://github.com/ctuning/ck-mlperf/pull/54#issuecomment-665731314