search

JuliaRegistries / RegistryCI.jl

Continuous integration (CI) tools for Julia package registries, including registry consistency testing, automatic merging (automerge) of pull requests, and automatic TagBot triggers
https://juliaregistries.github.io/RegistryCI.jl/stable
Other
31 stars 30 forks source link

CI should require presence of test/runtest.jl for new packages #313

Open oxinabox opened 3 years ago

oxinabox commented 3 years ago

A much weaker version of https://github.com/JuliaRegistries/RegistryCI.jl/issues/120 and similar to #261 (though less crucial)

Even if we are not going to run it, or check that it contains something meaningful, I think it is worth checking the bare minimum effort has been put into making sure the package is good. At least be aware enough of what is going on to know that packages have these.

We used to have this requirement at one point for registring packages back in 0.4 days.

DilumAluthge commented 3 years ago

Makes sense to me

KristofferC commented 3 years ago

It seems unnecessary to add friction to registering a package that can just be circumvented with a @test 1==1 (which is what people that didn't add any tests in the first place likely will do).

ericphanson commented 3 years ago

The friction is only in the case where there are no tests, so if we want to encourage tests, that seems like a good place to add friction, no?

Sure, it can be easily circumvented, but it’s at least a nudge in the right direction. I think it would also help set the expectation that tests are needed, eg a new developer might not know they’re something they should have, and learn about it from an automerge failure.

In general I think it’s fine for automerge to be opinionated in order to try to help nudge the package ecosystem in a good direction (since RegistryCI’s biggest client, the General registry, is a shared resource we want to keep healthy, not just an easy way to access random pieces of code).

ericphanson commented 3 years ago

With the technology of ~tokei~ (https://github.com/XAMPPRocky/tokei, packaged as Tokei_jll), we could count lines of Julia code in /test (ref #351 for /src). That could be used to create a more stringent requirement than just that tests/runtests.jl exists (but still pretty cirumventable, just write @test true a bunch of times).

My opinion is that we probably should not do this (at least at this point), and just start with checking that test/runtests.jl exists. IMO that covers most of the bases regarding getting new devs up to speed regarding the package expectations and where to put tests, without being too burdensome.