codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

[Test Analysis] Error message parsing #1487

Open rohan-at-sentry opened 6 months ago

rohan-at-sentry commented 6 months ago

Our normalization system depends on being able to parse failure messages. Investigate and implement improvements for messages output by each of the test runners.

Improvements should help

### Optimize Test Failure Message Normalization
- [ ] For pytest
- [ ] For vitest
- [ ] For jest
- [ ] For ruby (we have customers using test analytics)
### Optimize how we store failure messages to improve efficiency of error message comparison to determine flakes
- [ ] Investigate ways to detect and improve efficiency
- [ ] Investigate and implement ways to remove stop words
- [ ] Recommend other techniques
- [ ] Implement them
- [ ] Talk to Sentry ML team - to determine how to get feedback for model performance
- [ ] Investigate if we need our own test runner plugins - helpful in getting us the right error message details from source
rohan-at-sentry commented 4 weeks ago

We can descope the work to apply the framework detection to our flake normalization system for this quarter