search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

Problem validation: errors in JS exploration #1870

Closed codecovdesign closed 2 weeks ago

codecovdesign commented 4 months ago

Context: at a high level, we are exploring the problem space related JS related errors, related code quality workflows, how how market opprotunity for Codecov to serve as a error prevention tool (specific to said problem space). More here in this doc >

Problem to Solve

In the issue, we're seeking to understand the common problems and perceptions of users around type errors in JavaScript. Type errors are a significant source of issues, example from latest data roughly 40% of errors on Sentry (across orgs). Our goal is to identify/verify the data around the most common JS errors and validate the need for a prevention mechanism.

Data exploration, analysis, and verification

Let's identify the most common JS errors detected across organizations in Sentry. This data will help us explore whether type errors are a primary concern for developers and how they impact production environments. Then reach out to customers that may be facing these issues and learn how they perceive they and handle it today.

Questions to explore

  1. Are type errors significant for JS developers? Even if the most common error on Sentry (quantitively), do they recognize it's a problem (qualitatively)? (consider: is the the most because they are widely ignored OR that it's frequently + challenging to resolve)
  2. Is there value in having a prevention mechanism for these errors?
  3. How often are these errors introduced and perceived as leading production issues?
  4. What are the current methods developers use to prevent or fix these errors?

Outcome

Our ideal outcome is to better understand the problem space and data areas, with later goal to explore potential solutions that address the needs and concerns of JS developers.

Internal questions

Validation (what does the data tell us?)

Discovery ToDos / next steps

rohan-at-sentry commented 4 months ago

Re

What would a signal of the problem be?|

A qualitative signal from those we interview that these errors do both of the following -

  1. Have high customer / user impact. in other words, these errors cause bugs that disrupt a critical UX flow or business flow with material impact (lost revenue or angry users)
  2. These happen frequently enough (no quantitative measure of this today, but we can listen for qualitative signals for this)

What would validate that this is a problem worthy of focusing on solution?

If these errors cause visible and impactful bugs frequently, then it follows that people would be interested to prevent these from hitting production. We'd want to understand if

codecovdesign commented 3 months ago

Draft on customer interview script

view draft # Customer Interview Script / Outline ### Introduction: - Introduce ourselves and explain the purpose of the interview - Briefly describe Codecov and its current role in code coverage - Explain we are exploring opportunities to enhance our offering, particularly in preventing JavaScript errors (or just say frontend generally?) ### Purpose: - "We're exploring the problem space related to JS errors (or frontend generally?) and how developers handle them" ## Broad Questions ##### General Development Workflow: - "Can you describe your current development workflow for JavaScript applications?" - "What tools do you currently use for error detection and prevention in your code?" ##### Error Handling: - "How do you typically handle errors in your JavaScript code?" - "Can you walk me through your process from detecting an error to resolving it?" ## Focused Questions on Type Errors ##### Current Prevention Methods: - "What methods or tools do you use to prevent errors in your code?" - "How effective do you find these methods or tools?" ##### Experience with Type Errors: - "How often do you encounter type errors in your JavaScript projects?" - "Can you share an example of a recent type error you had to deal with?" ##### Impact of Type Errors: - "What impact do type errors have on your development process and production environment?" - "Do you find that type errors lead to significant issues in production?" ##### Perception of the Problem: - "Do you consider type errors a significant problem in your development workflow?" - "Why do you think type errors are the most common errors on platforms like Sentry? Are they often ignored, frequently occurring, or challenging to resolve?" ## Exploring Potential Solutions ##### Value of Prevention Mechanism: - "Would you see value in a tool specifically designed to prevent type errors?" - "What features or functionalities would you expect from such a tool?" ##### Frequency and Severity: - "How often are type errors introduced in your codebase?" - "Do you perceive these errors as leading to major production issues?" ##### Current Pain Points: - "What are the biggest pain points you experience with type errors?" - "What production errors keep you up at night?" ## Outcome and Validation ##### Feedback on Codecov’s Potential Role: - "How do you think Sentry could help in preventing JavaScript errors, particularly type errors?" - "What would make you adopt a new tool for error prevention from Codecov?" ##### Customer Profile: - "Can you describe your team or company’s profile? (size, industry, type of projects)" - "What characteristics make a tool appealing for your team?" ## Closing ##### Final Thoughts: - "Is there anything else you’d like to share about your experience with JavaScript errors?" - "Would you be interested in providing feedback on potential solutions in the future?" ## Internal Questions for Analysis ### Signal of the Problem: - What recurring themes or pain points did you identify related to type errors? - Did customers express a significant impact from type errors on their development and production environments? ##### Validation of the Problem: - Are type errors perceived as a problem worth solving based on customer feedback? - What data points support the need for a prevention mechanism? ##### Prevention Methods: - What are the current methods used by developers to prevent type errors? - How effective are these methods, and where do they fall short? ##### Production Errors: - What are the most common production errors reported by developers? - Do type errors align with these common production errors? ### Customer Profile: - What types of customers (e.g., company size, industry) are most affected by type errors? - What profiles should be targeted for further insights and solution validation?
codecovdesign commented 2 months ago

Sync with Sentry PD (audit error/issue filtering and types ux) 7/10

sync notes - What does a common workflow look like addressing issue/errors? - Is it primarily reactive or proactive of both? - Do user identify themes of errors/issues? or... - a: we haven't had users aggregate errors by type - it's more sorted by error events (this could be a way to show most common) - could be vasuda that is looking at showing insights. thre is a dashboard that could be used/created around issues type - rachel w. has brought up having insight on top of issues - How can user see breakdown of issue/error types? - How/Does Sentry influence prevention mechanisms? - it depends a lot on the platform. a lot of JS errors could be prevented by using TS and that happens in the IDE already. contract management could be useful too (not sure if JS could do this) - project details: this is a dashboard which shows some apdex is a performance metric and see a list of issues related to the project. i would say this could be the ideal place to show a preventative measure - and project details will live inside the dashboard bucket. - problem to solve: what is the IDE and linting tools not using, console logging and use type mismatches
codecovdesign commented 2 months ago

Sync with Sentry PM (data analytics around type errors) 7/12

sync notes - customer painpoints with certain error types (namely JS) - m: today, no, we dont have grouping or types of errors. we are talking about it a bit, but doesn't exist. - q: what has the hesitation been - a: one take is that ppl aren't coming in to browse around, the idea of logging in to find work todo is less common. consider alternative of when a reported problem occurs, the debugging from that (reactive) - having too many errors can be a result of bad job grouping - though sometimes there arn't errors instead logs they don't care about. the missing piece could be around re-occuring issues (proactive improvement). - r: knowing we have X type errors, doesn't mean user want/need inbox zero - so then the question is does it matter - nevertheless it's something we are thinking about. the opposite of this is no organization where we are at - just not sure at the moment error type is the winning strategy, but maybe worth considering. ("impact" => something not working, then what is the severity, consider how many users exposed to issue) - risky changes: - r: feels like gh comments we have, drumil is leading this already - m: auto-resolve also exists, though barely in use at the moment - IF static analysis existed and matched with errors in existing already to help filter / drive the action - consider linting rules compared with errors with sentry. idea is to leverage existing production data to help match with SA - questions: - how could we pull the data to allow ability to do these things - m: at first thinking, not easily, lots of ways to define impact - as lot that we have. likely would need ML to accomplish some of this where we are a bit far - r: consider reaching out to tillman (to learn more about feasiblity), sense here is that it passes smell tests with being compelling, but it's hard to prove the counterfactual that you actually solved a problem in the first place - m: show me the ones that are actually worth the fix - consider the fix is sometimes more expensive than that cost of error in production - data around common error types - customers that may be willing to talk with us
codecovdesign commented 2 months ago
Participant 1 interview 7/10/24 📓 interview summary ### Role Software developer with a focus on maintenance. ### Primary Language / Code Stack - **PHP** - **Frontend:** JavaScript (JS) ### General Development Workflow #### Usage of Sentry - Sentry is used to monitor new issues during releases. - Recently focused on the crash rate for JS, revealing a high number of errors. - Previously, the frontend (FE) was overlooked, but now it's actively monitored. - Daily issues range from 200–500, with many repeat issues having the same error descriptions and stack traces, making it challenging to pinpoint recurring errors. - Triage involves identifying responsible parties, often complicated by third-party dependencies. - Null errors and animation-related issues are common. - New performance reports from Sentry have been helpful. #### Detection of Error Themes - Performance tracking used to show the most improved and most regressed metrics, though currently showing no results. ### Current Development Workflow for JavaScript - Emphasis on PHP with unit tests achieving 28% coverage. - Use of the TypeScript compiler (TSC). - Handling exceptions and enabling error differentiation between PHP and JS. ### Error Handling in JavaScript - Process involves detecting, triaging, and resolving errors, with challenges in isolating third-party issues. ## Post-Interview Internal Notes ### Key Takeaways #### Static Analysis to Eliminate Noise - Potential to reduce errors and Sentry events by addressing issues in the app now. #### Third-Party Dependency Issues - Difficulty in diagnosing third-party issues is time-consuming and offers little control. - Significant market opportunity similar to Sonatype and Snyk, focusing on third-party open source security issues. ### Internal Feedback - Ideation, Thoughts, Etc: #### Interest in Follow-Up: - Exploring static analysis to reduce noise from Sentry reports. - Understanding issues from third-party dependencies vs. first-party code. #### Challenges and Solutions - Diagnosing third-party issues is a notable pain point. ### Market Insights + Ideation - Sonatype and Snyk have capitalized on third-party open source security issues. - Replicating this model for bug issues could be worth looking into - 💡 Potential to build a business around third-party open source bug issues. - 💡 Possible integration of machine learning to analyze errors from Sentry SDK.

📽️ source recording 1

participant 2 notes📓 interview summary General Development Workflow: "Can you describe your current development workflow for JavaScript applications?" - Current focus is a library, so not into product workflow too much. I have an app that uses that library so we are basically using honey badger in that. "What tools do you currently use for error detection and prevention in your code?" Mainly user testing / usage - Have an extensive suite of tests and when everything is passing we are in pretty good shape. Once heard a podcast why pay for manual testing when you could pay for ppl to automate tasks. The better examples of that… - A lot of your prevention methods sounds like automated tests - Have you found a workflow using static analysis tools? Or linting tools? - We have some static analysis in terms of linting and coding standards. Most of these tools are for enterprise. - I find most of these tools cause a lot of noise
Participant 3 interview 8/6/24📓 interview summary What do you do? - Ovia health and it’s a women's health company – the app outlines when the best time - Manages 3 backend developers now Do you use sentry? How do you manage your workflow? - We use the on premise version of sentry - We are mostly reactive toward Sentry based on a local error that is happening, every so often to see what is in sentry Q: do you notice any themes with the errors you have? - Right now yes, bc our codebase is old and written in PHP, a lot of the modern features in how they are built. Alot of errors have been bugs we are squashing, such as array key exception errors. Often we squash one and then it’s part of the one area and appears in another. - As we start moving up different levels of the codebase we start seeing more errors and we see them on sentry as well. All of our environments are test apis we build on demand for QA. Everything is reported into some sentry instance somewhere. Q: how many of these PHP flags end up in production and cause user issues. - We are pretty good about not getting these into product bc using https://phpstan.org/ - If there were errors occurring I’d like to know in the pull request. The reason why is it’s the cheapest time to fix it. Also seeing it in the pr allows us to visualize it and if we had a way to say this error is part of a theme, then the PR is a helpful gatekeeper since approvers and developers will see this – we can always override it and move on, but it could. We only use PHP codesniffer and "Can you describe your current development workflow for JavaScript applications?" "What tools do you currently use for error detection and prevention in your code?"

📽️ source recording 3

codecovdesign commented 2 months ago

Data analytics / redash

pending: follow up to revisit/update redash

codecovdesign commented 2 months ago

Internal sync @rohan-at-sentry @eliatcodecov @aj-codecov 7/12

sync notes - Rohan: signal we want is IF someone is using existing static analysis tool, what are the problems with this. consider, ppl use sentry and errors have frequently enough and invest in product (such as sonar) - q: is there a causality of sentry errors driving users to seek preventative tools - when there are a lot of errors of the same type - it's not just about what is most common - to further compound the challenge there could be different proxies of pain contextual. the assumption is that we don't start with static analysis until you are at the point that they want - q: do we think it's customer gotten big enough or that there are too many issues to start caring about - it's good segmented on size and revenue, but there is another to look at: what kind of product are they building. consider top quality is their priority. - customers considering static analysis / or what they are using - opportunity to flesh out: if customers are considering sa, it's a strong indication they care about prevention - which means it's a target profile. if already, what is the problems around setting these tools up and configuring them. - going a layer deeper: hypothesis you care about prevention when there are errors that happen frequently when it railroads you business and impacts customer - consider unknown upload error: - Eli: consider sonarcube product offering, it's a validated tool in the market. what is the question of the subset of SA prevention offering that we could do to prevent bugs ahead of merge - what I'd like to hear is how customers are considering thinking about triaging bugs, such as one of the biggest issues with SA is that it feels loud and - signal: what would drive real user action of potential error that will happen - ... - criteria for signal, - what drives error prevention action prior to merge - the differentiation is to build a link between static analysis and production errors ### review with adalene - let's review the actual error type how it looks in sentry - the looks why they don't care
codecovdesign commented 2 weeks ago

sync with @rohan-at-sentry and @eliatcodecov on sept 19th