Machine learning model template

[Nolski]

Summary

Teams are hesitant to release their machine learning code open source as they feel it's not as simple as other application code. It would be nice if we had a succinct template to

Background

This stems from my talks with MacroEyes and Har Zindagi. They both are wary about releasing their machine learning models as they might be used improperly or not understood correctly. I think we could really provide a useful resource that hasn't been used beyond more than one project yet.

I was reading a white paper a while back which researched this quite a bit. Perhaps we could also adapt the publiccode.yml document created by the public code network?

Outcome

I think a great outcome for this would be a README template or at very least a how to use document you can embed in the repository/documentation. Something easily accessible like a markdown file

[Nolski]

Here's an initial pass, pulling from the above paper.

Model Title

Model Details.

Basic information about the model.

• Person or organization developing model
• Model date
• Model version
• Model type
• Information about training algorithms, parameters, fairness constraints or other applied approaches, and features
• Paper or other resource for more information
• Citation details
• License
• Where to send questions or comments about the model

  Intended Use

  Use cases that were envisioned during development.

• Primary intended uses
• Primary intended users
• Out-of-scope use cases

  Factors

  Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others.

• Relevant factors
• Evaluation factors

  Metrics

  Metrics should be chosen to reflect potential real-world impacts of the model.

• Model performance measures
• Decision thresholds
• Variation approaches

  Evaluation Data

  Details on the dataset(s) used for the quantitative analyses in the card.

• Datasets
• Motivation
• Preprocessing

  Training Data.

  May not be possible to provide in practice. When possible, this section should mirror Evaluation Data. If such detail is not possible, minimal allowable information should be provided here, such as details of the distribution over various factors in the training datasets.

  Quantitative Analyses

• Unitary results
• Intersectional results

  Ethical Considerations

  Caveats and Recommendations

[jwflory]

Included in 2020-03-24 issue triage and cleanup.

---

This is published on our resources site: https://librecorps.github.io/resources/ai/model-card/

Closing as complete. :clapper: