search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

Discovery: configuration without a yaml #2234

Open codecovdesign opened 1 month ago

codecovdesign commented 1 month ago

Problem to Solve:

One of the core challenges with Codecov is the reliance on YAML configuration files. Customers often face issues with outdated or incorrect YAML configurations, leading to confusion and potential misconfigurations. As new features are introduced, the YAML files may not get updated accordingly, causing further complications.

Concept Overview:

Discovery to explore the feasibility of configuring Codecov directly through the UI instead of relying on YAML files. This approach could streamline the setup process, ensure configurations are always up-to-date, and reduce the potential for user error. Key Questions to Explore:

giovanni-guidini commented 1 month ago

My 2 🪙

The one interesting aspect of the configuration being YAML based and living in the repo is that it's "testable" in a sense. The changes you make to a commit take effect in that commit, so you can see the result (usually) before merging to main. Having this ability is interesting. Effectively we piggy back on git to make our config have version control in a way.

YAML is well known option for configuration. Is the problem truly the YAML file or the mess of options that we have? There are too many and they are confusing / redundant.

If we decide to shift to some UI and an internal representation, how is the deprecation of YAML going to happen? Is there enough of a need to justify that?

I do think we need an interface to make it easier to work with our messy yaml config options, but under the hood if it can spit a nice and working yaml, would that be enough?

Is it possible to automatically generate or update YAML files based on UI configurations, and push these changes as pull requests to the repo?

Looking at GitHub (which has the most complicated permissions) we need Repository Contents Write access. I don't think the app currently has that permission (only read). It would probably be better to have a dedicated app to write the PRs. Only users interested give that app the extra permissions it needs, so we don't have to make changes to the permissions of the regular CodecovApp

For the other providers (and GitHub with no apps) as long as we have a token that can create a commit and a pull request in a repo we should be fine.

I'd think about other options like having a copy-paste YAML so users can make the PR themselves if they don't want to give us more permissions.

drazisil-codecov commented 1 month ago

YAML is a well known option for configuration. Is the problem truly the YAML file or the mess of options that we have?

It might be more how python parses it. @eliatcodecov may remember more (also, in my opinion, errors are very hard to understand if it's a mapping /list/indenting issue)

eliatcodecov commented 1 month ago

Some aspects of YAML make it annoying to parse, yes, but it's not a game breaking issue. @drazisil-codecov is also correct in that YAML errors can be kind of ambiguous and hard to interpret at times, especially if you don't have a lot of experience with the language.

Also, we can never truly fully sunset our yaml configuration because we have thousands of customers who are actively using it. As we've seen with our uploaders, getting long time customers to change their behavior is really difficult, time consuming, and you never reach 100% success -- there will always be holdouts.

With this in mind, we should consider configuration instead of a YAML, but I'm highly doubtful we could ever sunset it.

The YAML itself is converted and stored as JSON in our backend, anyway. So we should really just consider YAML an interface for configuration. If we want to layer on more interfaces (such as a web UI, etc), we should consider these interfaces for communication that ultimately get stored in the same manner in our backend. This keeps our configuration compatible regardless of what the customer is using -- it is more work to support multiple configuration formats, of course. However, the minute we release anything that isn't YAML we're accepting that burden, anyway.