Proposal: `## devspec` for machine readable development/maintenance metadata

[roberth]

Is your feature request related to a problem? Please describe.

• OfBorg doesn't always know what to test, especially when the file isn't a package definition.
• Users often don't know how to test changes in a given file (if it's not a package, where we have the tests attribute).
• A template for adding this info to a file would be helpful even if not machine-readable:
  • Fewer thoughts required for each comment.
  • Users and contributors can recognize it and create a habit.
    • Suitable for quick "find in file" navigation.
  • Consistency is good.
• Highly reused modules currently have a maintenance problem because testing all use cases is somewhat difficult. Nixpkgs maintainers do not want to reverse engineer a test protocol, and shouldn't have to.

Describe the solution you'd like

• A standard format
• A machine readable format
• If it's a package, just keep using the tests attribute and such. A meta attribute for docs could be helpful, but out of scope for this discussion. I'm specifically trying to solve a problem for non-package files.

Draft

Example:

## devspec
# documentation:
#   - ../doc/foo-bar.md
# tests:
#   - /pkgs/tests/foo/bar.nix
# test-commands:
#   - nix-build -A tests.foo.bar
#   - nix-build -A nixosTests.foo-bar
# maintainers:
#   - alice
#   - bob
# ---

• ## devspec marks this as a standard header that must be parseable
  • #-style comment because it's about the implementation; not the interface, like documentation comments (/** */)
  • ## is a bit like the ** in /**, marking that it's more significant somehow.
  • devspec is "grep-able", e.g. 6 or 7 keypresses while already editing the file
  • devspec is searchable in issue tracker, documentation, etc
• The lines after are s/^# / / and parsed as yaml, until # --- is encountered or a line that doesn't start with #

Then any tooling can use the info. Standard fields:

• documentation: list of paths and/or URLs that must be maintained
• tests: list of paths that have tests that check the current file
• test-commands: how to run the tests. This attribute can perhaps also be specified in the tests files instead. These are meant to be Nix CLI compatible, but tooling is allowed to parse the commands. Test files should contain collections of tests that are fast enough, or taking the test-commands from the tests files would be a problem. Factoring large suites into multiple files seems like a good idea anyway.
• maintainers: people to ping, somewhat like CODEOWNERS, but controlled by us and without the mass ping problem

All paths are "relative URLs" (which include paths) that are resolved relative to the base directory, but / refers to the root of the project (ie Nixpkgs) instead of the system root.

Implementation steps:

• [ ] Refine this spec while writing one or two parsers
• [ ] Add support in ofborg
  • Enumerate files that have changed
  • Parse the devspec in all files that have changed. Fail if invalid
  • Implement a heuristic that avoids running all tests on tree-wide changes. E.g. limit number of test-commands
  • Add jobs and commit status(es)
• [ ] Add spec and user docs in Nixpkgs manual

---

Add a :+1: reaction to issues you find important.

[philiptaron]

Robert, I think I'd like to use the currently unused shebang at the top of these files for testing purposes, so that when you ran the file it'd run the tests for the file.

[roberth]

That's an interesting idea, but it is a bit less capable.

• not applicable to files that are already executable
• limits the usefulness of the executable bit when looking for scripts that actually do something
• not as easy to have multiple test commands; not impossible but harder to shoehorn that into a single shebang

More subjectively, I also expect that bundling it with the other attributes makes it a nice, coherent experience that's similar to the package tests and meta attributes.

[SuperSandro2000]

• The lines after are s/^# / / and parsed as yaml, until # --- is encountered or a line that doesn't start with #

we should limit that to a very limited subset of yaml, like only lists of strings. Otherwise we might run into subtle differences between yaml 1.1 and 1.2 or some other less fun parsing things.