Spike: Review Existing Cockpit Flake Analysis Tool

[MichaelClifford]

As a data scientists, I need to understand the current implementation of Flak Analysis with the cockpit tool for 2 reasons. 1) The existing implementation in AI Library is heavily based of the cockpit implementation, 2) Cockpit's most recent iteration may serve as a better solution for QE to adapt.

Success Criteria:

• [x] Review cockpit flake code with short write up on findings
• [ ] Taking existing RHV data set (that is incompatible with AI Library Flask) and run it with cockpit tooling.
• [ ] Connect with cockpit team about state of flake analysis.

[Shreyanand]

@MichaelClifford Where can I find the RHV data set? :thinking:

[MichaelClifford]

@Shreyanand you should be able to find the data here: s3://DH-PLAYPEN/ccit/

I'll add this to the documentation today. :)

[MichaelClifford]

@Shreyanand the main Cockpit repo for the ML work is cockpitupus, and not the cockpit repo in the cockpit project.

[Shreyanand]

The code in the AI library seems to be taken from the initial commit for cockpit (see this commit)

The subsequent major commits (see commits and cluster.py diff) seem to optimize code (parallelization and reduction in memory usage) and generate plots based on the clustering. I believe it should not change the model results.

@MichaelClifford Would we still want to try out cockpit tooling?

[MichaelClifford]

@Shreyanand Thanks for looking into this.

I think, given the above, we can go ahead and close this "Spike" issue as the above provides a clear enough explanation of the differences between the two flake analysis tools: They are identical, except cockpit has some additional optimization updates.

That said, moving forward we should update AI-library (or however we end up implementing this) to match the more recent optimized version in cockpit, there is not point in sticking with the older version of the code.