Check is setup.cfg is present.

[SimonGreenhill]

pytest will not run lexibank tests if there is no setup.cfg file (I noticed some lexibank datasets had tests that weren't being run). Do we want to implement this as a lexibank check or cldfbench check? (i.e. if 'test.py'.exists() and not 'setup.cfg'.exists())

[SimonGreenhill]

..and the inverse of this -- where setup.cfg is present (expecting a test.py), when test.py is missing.

[xrotwang]

The inverse does not need to be checked since it will result in a build error anyway.

[xrotwang]

Specifying args for pytest in a setup.cfg is just a convention, though. I think we want to also allow datasets to simply have a test_dataset.py module, which will be discovered by pytest without a need for setup.cfg. (test.py does not work, because the default pattern pytest looks for is test_*.py - note the extra underscore)

[SimonGreenhill]

Ok, we have the lexibank_XX.py convention, we could move all the test.py to test_XX.py, and remove setup.cfg (annoying to change lots of datasets now, but it makes one less point of failure?). And one less boilerplate file ("setup.cfg")

[xrotwang]

I kinda like setup.cfg. Why not keep it as is - i.e. by default there's setup.cfg and test.py, but datasets may also remove setup.cfg and have tests in test_xx.py? Or maybe change the dataset in cldfbench so that in the future the layout is different. But I don't think there's enough reason to retro-fit all existing datasets.

[SimonGreenhill]

Yeah, and we could just work around this on buildbot by changing the pytest config to something like:

[pytest]
python_files = test_*.py test.py

[xrotwang]

Moved to cldf-buildbot issue.