Open codecovdesign opened 7 months ago
sync with @nicholas-codecov
TLDR; The participant's attempt to configure the bundle analysis tool was unsuccessful due to reliance on micro bundling, which wasn't compatible. Additionally, they encountered delays and confusion in the GitHub app recognizing the installation, impacting their ability to view their organization and repository promptly.
How do you currently manage your bundle configurations?
Participant mentions using plain JS and micro bundling for Project Wallace, with limited use of Rollup. They express uncertainty about whether to bundle modules and the effectiveness of their CSS tree implementation.
What challenges have you faced in this area?
The participant discusses inconsistencies between tests run on source files and the bundled JS file, leading to "green" tests that fail in the bundled state. They also mention not using Codecov for test validation.
Is there anything in particular you wish you had to make this easier?
The participant did not specify any tools or solutions.
TLDR: The developer successfully configured the bundle analysis tool but encountered slow loading times for the PR comment report. They appreciated the in-app summary data but expressed a desire for visibility into changes across commits.
TLDR: The user successfully configured the repository, beginning with the documentation before moving to the app for the repo token. Their focus was on repo-level data, contributing many valuable suggestions for utilizing this data (see insights below).
TLDR The user successfully configured the bundle analysis tool, navigating through documentation and the app to obtain the necessary upload token. They experienced some confusion and errors within the UI but ultimately accessed the report. Insights on desired features and improvements are provided below.
Bundle Configuration Management:
Uses a TS config parser option. Managing the config file is challenging due to poor segmentation and large size, making it difficult to locate specific variables.
Challenges:
The main issues are the config file's complexity and finding the right variable due to its size and unstructured additions by developers.
Desired Improvements:
Interested in continuous integration for easier management of releases and PRs, with an emphasis on the importance of bundle size analysis.
TLDR: The participant discussed their current bundle configuration process, highlighting dependency and optimization challenges. They encountered issues and confusion during onboarding and configuration, particularly with the sequence of steps and understanding bundle names. Despite this they successfully configured the featured. Key insights include the desire for a tool that estimates optimization impacts and offers deeper dependency analysis.
Bundle Configuration Management:
Uses Next.js bundle analyzer for personal projects, rspack.dev for Sentry, noting inefficiencies in frontend code delivery and compilation by Babel.
Challenges:
INSIGHT: encountered issues with duplicate packages and difficulty determining unnecessary plugin removal.
Desired Improvements:
INSIGHT: expressed a need for tools to estimate the impact of optimizations on dependencies and insights on removal impacts.
TLDR: This participant successfully configured the bundle analysis tool, with some points highlighting a need for clearer documentation and improved UI clarity, especially regarding bundle names and step sequences. They expressed a desire for advanced features such as detailed dependency analysis and change tracking.
Bundle Configuration Management:
"We use webpack with an integrated configuration that heavily utilizes code splitting to identify how specific elements are utilized across various parts of the project."
Challenges:
INSIGHT: Noted challenges in tracking the usage of certain elements in multiple places.
Desired Improvements:
The lack of detail on individual chunks in previous tools was limiting.
INSIGHTS:
This participant's feedback illuminates the critical areas for tool enhancement, from onboarding to advanced analytical features, driving home the need for a user-centric approach in tool development and documentation.
📽️ See condensed video related to GitHub issues below (full)
Participants encountered confusion regarding the order of operation steps and the purpose of the "bundle name" in the configuration input.
UI elements, notably the persistent "unknown error" banner, and the default file/structure sorting mechanism presented usability issues, with a noted lack of options to sort/filter effectively.
There is a strong interest in features that enable users to see changes and differences among components/assets, not just in PR comments but also from one release to another and trends over time.
Participants expressed a desire for a more detailed breakdown of dependencies and their impact on bundle size. This reflects a broader theme of seeking ways the tool could facilitate actionable insights and optimizations; examples i) duplicate dependencies, ii) impact of removal, iii) proportion of assets to bundle, iv) system suggestions to improve.
Review incoming issues, issues from our v2 discovery, and the ones listed below from the audit. Then, identify which ones we'd like to groom and explore in upcoming design/engineering sprints.
Problem to solve
As we release the MVP of bundle analysis, we'd like to better understand the usability of configuration, user perception, and identify areas of improvements.
Configuration Experience:
How intuitive was the configuration process?
What challenges, if any, did user face during the configuration?
Data Interpretation:
Repo-Level Insights:
Participant Criteria
test scipt
# User Test Script for Bundle Configuration Tool ## Introduction ### Greet the Participant: - "Hello [Participant's Name], thank you for joining us today. We're excited to get your input on our new bundle analysis tool." ### Consent to Record: - "Before we start, we'd like to record this session for analysis and improvement purposes. Do you consent to us recording this session?" ### Explain the Test: - "We'll be asking you to complete a task using the tool, and we'd like you to share your screen and walk us through your process. Your insights are invaluable to us." ### Intro Questions: 1. "Before we begin, could you tell us a bit about how you currently manage your bundle configurations?" 2. "What challenges have you faced in this area?" 3. "Is there anything in particular you wish you had to make this easier?" ## Task Instructions ### Task Introduction: - "Imagine you've just heard that Sentry/Codecov has launched a new tool for bundle analysis. You're interested in configuring it for your organization's repository to see how it can improve your workflow." ### Specific Task: - "Your organization is 'turing-corp', and you need to configure the tool for your repository. Let's pretend the repository name is '[Insert Repo Name]'. How would you go about setting this up?" ## During the Task: - Encourage the participant to think aloud as they navigate the configuration process. - Take notes on any difficulties they encounter, any feedback they offer, and their overall user experience. ## Wrap-Up: - Thank them for their participation and valuable feedback. - Ask if they have any final thoughts or questions about the tool. - Inform them about the next steps and how their feedback will be used to improve the tool.Next steps
vuluongj20
)billyvg
)anonrig
)evanpurkhiser
)