"Methodology" Directory is missing

[mehrdadh]

Hi,

I think the "methodology" subdirectory is missing from toyadmos_autoencoder_main.py.

In addition, I was wondering if the README file here is up to date to replicate your accuracy results for anomaly detection model.

Thanks!

[petertorelli]

Hi @mehrdadh,

You are correct, the missing Methdology folder has been added, it was missing.

To the best of my knowledge the accuracy results are up to date. What are you observing vs. what are you expecting?

Peter

[mehrdadh]

@petertorelli thanks for quick response! I use the setup from mlcommons/tiny and their base model and I get something aroung 0.85 AUC with float32 model. However, when I run the same setup but with the model that is provided in this repository, I get a very bad result around 0.66 (AUC).

Then I decided to run directly from this repository to make sure that I'm replicating the results correctly, but I think some stuff are missing based on README. For example what is /eembc/ToyADMOS_FC_AE directory that is mentioned in this README?

Also to clarify, for the submissions are you using the models that are provided in this repository or the models from MLCommon/tiny?

Thanks!

[petertorelli]

@mehrdadh Oh I see the problem, the models were originally developed for EEMBC by Silicon Labs. When I moved the models from their repo to EEMBC, I changed the names to match what we are using for ULPMark-ML & tinyMLPerf (e.g. Toy_ADMOS vs ad01).

This is the original README which points to their folder naming scheme, which is why they don't match in the new repository (this one).

Try cloning the SiLabs repo and running from there and let me know if that works. I will update the EEMBC readmes next week.

Peter

[petertorelli]

I updated the README files to use the new directory naming.

[petertorelli]

Commit https://github.com/eembc/benchmark-runner-ml/commit/79bdcacecdaef9cb192ff5dbb7e671fcd90b7ae6

[mehrdadh]

thanks @petertorelli for fixing this!