krmaffey
commented
3 months ago MLTE Evaluation Protocol
23 May 2024
Step 1: Project Familiarization
- Participants need time to become acquainted with their projects.
- This will vary by project and team. We recommend that participants have access to the code base, familiarization with the project goals, and an understanding of data sources before MLTE is introduced (generally one week of familiarization).
Step 2: Initial Survey
- Once participants have onboarded and familiarized themselves with their projects, the MLTE team will ask participants to fill out the survey.
Step 3: MLTE Introduction
- The MLTE team gives participants an introduction to MLTE, focusing on the process.
- This will likely require a ~1 hour meeting including the following content:
- MLTE overview
- Negotiation card demonstration
- Time for questions/interaction with participants
- Prior to this meeting, the team will set up a central location where negotiation cards can be saved and viewed by all participants and the stakeholders.
Step 4: Project Progression
- Participants need time to work on their projects at their own pace. Likely ~2-3 weeks, with weekly check-in meetings between the MLTE team and developers, and between the MLTE team and product/project owners, to understand how MLTE is being used and provide coaching as needed.
(OPTIONAL) Step 5: Testing with MLTE
- If a project progresses to the point where it has a functioning model that can be tested, the MLTE team will schedule a meeting with the participant for a demonstration of MLTE testing capabilities.
- This will involve another ~1 hour meeting in which the team conducts an overview and demonstration of MLTE test capabilities, with a hands-on portion with time to answer questions and help participants as needed.
Step 6: Final Survey
- At the end of the project or after ~1 month of work on it (whichever comes first), the MLTE team will ask participants to take the survey again.
The goal of this epic is to measure the impact of introducing MLTE into AI2C model development practices. To do so, model development teams (developers and product owners) will take a survey related to model development practices. We will then coach team members on the use of MLTE for Model T&E. AT the end, team members will take the same survey and ideally we will be able to see improvements in their answers.
Tasks:
376
395
396
398
397
443