enso-org / enso

Hybrid visual and textual functional programming.
https://enso.org
Apache License 2.0
7.34k stars 323 forks source link

Make tests part of Enso projects #9039

Open JaroslavTulach opened 7 months ago

JaroslavTulach commented 7 months ago

The current structure of distribution and test and std-bits is confusing for new comers, a bit hacky and certainly not something we'd like to encourage our customers to do.

Let's improve the situation for us as well as regular users by making tests part of Enso project structure.

### Tasks
- [ ] Write [future press release](http://wiki.apidesign.org/wiki/Working_Backwards) or manual to describe how the testing shall work
- [ ] Redefine [layout of Enso project](https://github.com/enso-org/enso/blob/04161b33e4cb1ecc74ec1d8d89b1da7ded82262f/docs/distribution/packaging.md) to include `test` directory
- [ ] Design a `--test` argument to discover/enumerate/execute all/some tests
- [ ] Allow tests access `private` modules of the same project without any special option (see https://github.com/enso-org/enso/issues/9039#issuecomment-2284552083)
- [ ] Make sure code cannot access tests
- [ ] Demonstrate [sharing of code between Table projects](https://github.com/enso-org/enso/issues/9039#issuecomment-1941852329) is possible
JaroslavTulach commented 7 months ago

The most painful open question:

we need to have some way to keep sharing test code between libraries. Like currently Database is tested inside of Table_Tests - even if we split it - we still need to share the common specification of 'table' behavior that is supposed to be satisfied both by the in-memory Table and the Database one.

The "Maven way" is to create module TestingKit.Table and put all the shared infrastructure into it. Then depend on it from Standard.Table tests and Standard.Database tests.

Maven doesn't treat tests as project artifacts (unlike all the fancy successor build systems like sbt). As such one cannot depend on them. If one wants to share tests, one needs to create a special project for that.

Akirathan commented 1 month ago

Task: Allow tests access private modules of the same project without any special option

This is not a good idea in general. Vast majority of the tests for a project Proj do not access private stuff from Proj. Such tests are meant to test the public API of the Proj. In these tests, we want the access to any project-private stuff to throw an error.

Only a small subset of tests need access to private stuff. Whenever private access is desired, a special method with_private_checks_disable ~action can be used. The code in action will be allowed to access private stuff from the same project.

radeusgd commented 1 month ago

Task: Allow tests access private modules of the same project without any special option

This is not a good idea in general. Vast majority of the tests for a project Proj do not access private stuff from Proj. Such tests are meant to test the public API of the Proj. In these tests, we want the access to any project-private stuff to throw an error.

Only a small subset of tests need access to private stuff. Whenever private access is desired, a special method with_private_checks_disable ~action can be used. The code in action will be allowed to access private stuff from the same project.

But ideally it would only be possible to use with_private_checks_disable within tests. Using this method in user projects should not be allowed, at least not by default (setting some special flag could be required to allow it).