Open davidanthoff opened 1 year ago
Another idea would be to add something like a @testfolder
macro. Things might look like this:
@testfolder "Foo" begin
@testfolder "Bar" begin
@testitem "Something begin
end
end
end
The rule would presumably be that a @testfolder
can only contain other @testfolder
s and @testitem
s.
Could also be named @testcollection
...
First let me say thank you for developing this package. It's one of those packages where you immediately think: I didn't know I needed this but I really do!
Reusing the file system does have a certain elegance and robustness.
You may have already considered this, but as an alternative could the existing @testset
be repurposed for this? It is very natural to refer to a set of items, and the hierarchy in this system would reflect the default test runner.
New idea: we conceptually change things so that each test item has a name/id and a label. The name/id can be hierarchical. If no label is specified, the name automatically becomes the label. The name/id can be specified either as a String
or as a normal Julia identifier.
Implications would be this:
@testitem "Foo" begin
end
Here "Foo"
is the id/name, and because no label is specified it also is the label.
@testitem Foo label="Something else" begin
end
Name is Foo
, but label is "Something else"
@testitem Bar.Foo begin
end
The name is Foo
, and the hierarchy will now show the file name, then a folder Bar
, and Foo
will appear inside the folder `Bar.
@testitem Bar.Foo label="Something else" begin
end
Would be the obvious thing.
And we would of course allow arbitrarily deep nesting:
@testitem Foobar.Bar.Foo begin
end
Currently I have not felt the need of another level of nesting besides that of the files, and actually having the files as the top labels for the test makes it very easy to visualize where the tests are.
However, I do know that other packages have large files, and the list of tests can become complicated, thus I like the idea of allowing nesting by labeling.
My doubt would be: If the @testitem
is not nested, will it be at the top level, or as subitem of the "file"? Having to necessarily nest all testitems with a redundant filename information maybe a downgrade of usability.
ps: Another syntax alternative could be:
@testgroup "Tests1" # (actually `@testset` sounds nicer, but may cause conflicts).
@testitem "Foo" <: Tests1 begin
end
That would not allow arbitrary nesting, but neither does our type system :-D.
Ah, I should clarify: my idea is that the file/folder type nesting we have already stays in place, this would just be an optional way to add more nesting below the file system nesting we already have. I completely agree, I really like having the file system structure in place.
So this would also mean that if one has two files, fileA.jl
and fileB.jl
, and they have the following respective @testitem
s in them:
In fileA.jl
@testitem Foo.Testitem1 begin
end
In fileB.jl`
@testitem Foo.Testitem2 begin
end
Then the hierarchy would look like this:
fileA.jl
Foo
Testitem1
fileB.jl
Foo
Testitem2
So the two Foo
"folders" wouldn't be the same because they have different parents.
Maybe unrelated, but I think it would be great if the Pkg.test
also showed even the current hierarchy. It currently lumps all @testitem
s as a single testset named Package
.
+1 for doing the nesting by directory/file structure. this maps to what they do in golang, and that seems to work well enough for them! :)
I'm ambivalent on the sub-file nesting, but it does seem like it could be nice, yeah! Either by having yet another grouping macro, or using nested names as you suggested.
my idea is that the file/folder type nesting we have already stays in place, this would just be an optional way to add more nesting below the file system nesting we already have. I completely agree, I really like having the file system structure in place.
Note that we don't seem to be doing actual nested folder -> nested tests yet; for example, if I have 2 subdirectories in my test directory, each w/ their own @testitem
s, it shows up like:
i.e. each individual file w/ @testitem
s is listed as a top-level "set" of tests
Whereas in go (which @NHDaly was referring to), has:
so here we have the std
directory, in which we have the bytes
directory, and finally the boundary_test.go
file has a nested list of individual tests.
As far as I can tell, we need to restructure how we're providing testitems to the vscode request by utilizing the children
field to have the appropriate nesting show up.
If I can figure out how to get a dev env setup to hack on the julia extension, I'll see if I can take a crack at making this change.
Note that we don't seem to be doing actual nested folder -> nested tests yet
Yep, I was more describing an aspiration ;) Not yet implemented, and definitely the go
style is better as one can then run all tests in a given folder.
I think this folder hierarchy thing could be implemented purely in the VS Code extension, essentially https://github.com/julia-vscode/julia-vscode/blob/d0d8c6b43fc7cf3f96cd41493e7a943829c421e1/src/testing/testFeature.ts#L295-L328 would just need to parse params.uri
into the individual folder parts and then construct the correct hierarchy of TestItem
s. The implementation might be a bit annoying because the publishTestitemsHandler
notification is per file, so one then needs to go into the tree of TestItem
s and only modify parts of it. I'm also not entirely sure whether we handle deletion of files properly (even in the current implementation).
And we might then also need to check https://github.com/julia-vscode/julia-vscode/blob/d0d8c6b43fc7cf3f96cd41493e7a943829c421e1/src/testing/testFeature.ts#L414 to make sure it can deal with TestItem
s that are folders, but I think that should be fairly straightforward.
Any progress on this? I wanted to do
@testitem "Set of tests" begin
f(x) = x
@testitem "Single part" begin
import ..f
f(1)
end
end
but this didn't seem to work. I think according to https://en.wikipedia.org/wiki/Principle_of_least_astonishment it would be nice to have this since it's possible with testset
, and intuitively I think of the @testitem
as defining little modules.
So, I don't think we would ever allow nesting of @testitem
, they are really the atomic leaf unit that one can execute. If we were to enable some in-file nesting it would presumably use one of the solutions that we discussed above.
But for now I think the low hanging fruit is to just properly implement a tree view for the folder/file hierarchy in the extension, that seems the first step.
At the moment the hierarchy for test items is essentially the file system, and then a flat list of
@testitem
per file.Is that good enough? One really nice thing about this is that it is simple, plus we reuse an existing well established hierarchical structure, the file system, to order things. So right now I'm thinking that we probably don't want anything beyond that, but who knows...
One could also consider something like
@testitem "Level 1/Level 2/My test" begin end
, and then the hierarchy would be constructed out of these paths of names...