Adds more granular capability controls

[ConfusedSky]

I love this module, the only problem I've ever had with it is that there is it will fail if any of the non-core builtins aren't included in the runtime you are using.

Maybe this isn't the best way to do this, but I've modified stacktrace_report to have conditional imports for each of the capabilities and hide and show different features accordingly.

Tests all still run fine with wrenc

Note

The main testie module also uses "io" and "os" which isn't a huge issue since my runtime also implements them. That being said I don't implement Process.exit, but if there is an error it terminates beforehand anyway due to mirror not being implemented either.

[joshgoebel]

I'll have to noodle on this. Much of this interop was originally written before I forked Wren Console - which I still thought I could get things into CLI so that Exercism could use it... now that that has failed to happen until CLI adds tracebacks I'm not sure I see the value in interop - as all sorts of little things become a problem (Process.exit), etc... and it's only likely those things will increase in the future. It was on the list to rip out this interop (making this library Wren + Wren essentials only), but I just never got around to it.

I hadn't really considered other non-CLI runtimes at all...

Could you not use it "as is" if you just supplied your own custom test reporter? (which the API allows)

[joshgoebel]

non-core builtins aren't included in the runtime you are using.

What runtime are you using?

[ConfusedSky]

Could you not use it "as is" if you just supplied your own custom test reporter? (which the API allows)

I could not, since CuteReporter (which in turn imports StackTraceReport) is an unconditional import. Perhaps instead then, it can be a global constant DEFAULT_REPORTER and have CuteReporter conditionally imported as DEFAULT_REPORTER, or have StackTraceReport be a conditional import on cute.

Also, that being said if I did have a custom test reporter it would be basically 95% copied from CuteReporter

What runtime are you using?

I use wren as an embedded language within other applications for plugins and such, so I keep my runtimes pretty sandboxed (so far). So rather than directly accessing a File I'd access a Resource for example. But there are other places where I do use wrenc so I'd rather it work in both.

Even for CLI I have my own (incomplete) implementation that I'm still working on. Since I'd rather write rust than C.

Considering wren-testie is mostly core I think it would be valuable for others (considering that it's the only unit testing framework that I find pleasant to work with that hasn't been abandoned. Otherwise, I'm happy to just keep using my fork.

[ConfusedSky]

Even for CLI I have my own (incomplete) implementation that I'm still working on. Since I'd rather write rust than C.

Most of the work so far has been being able to create foreign methods that are written as decorated functions

[joshgoebel]

Your runtime supports traces?

[joshgoebel]

I could not, since CuteReporter (which in turn imports StackTraceReport) is an unconditional import.

Right, but which part of that import blows up again exactly? If it's File can't you just define a stub class that does nothing to get past the compiler?

[ConfusedSky]

Your runtime supports traces?

Not yet. Like I said I've been much more focused on the high-level wrapper over the C API to make writing foreigns easier. Hypothetically it would also work with wren-console's plugin api. Although that might make building projects more annoying.

Right, but which part of that import blows up again exactly? If its File can't you just define a stub class that does nothing to get past the compiler?

Yea it's File.

I could but that feels kinda hacky, and not universally applicable since an application could have an "io" module that doesn't implement File.

[joshgoebel]

Not yet.

How does traceSummary still work then? Or were you merely starting with a tiny amount of work to start a discussion and see how it would be received?

Give the other missing functionality (Process.exit, etc) I'd be curious to know how you're using it inside your runtime presently at all - or perhaps you're not - you've just used it with wrenc and would now like to ALSO use it inside your runtime?

[ConfusedSky]

Oh traceSummary doesn't. I initially made a change to make it compile, but the other changes I made were to make it more universal.

IE: it should work if embedding environment supports traces + files + parsing or traces + files or just traces.

I currently use it from within my runtime, process exit isn't a huge deal since it's called after the first_error block which submits an error anyway so it never hits that line.

[ConfusedSky]

My only point is that it's not (for the most part) strongly coupled to any system, there are other things like Stdout.flush() which I do implement. But, let's say I wanted to use DOME or TIC-80? Aside from a few touches here and there, it's compatible.

[joshgoebel]

process exit isn't a huge deal since it's called after the first_error block which submits an error anyway so it never hits that line.

Actually in reading this just now tests are not guaranteed to have the same result across runs (one of the preconditions could now be different, etc)... so first_error and "call it again" is an even worse ugly hack than I originally thought and it may be best to remove it entirely...

---

If we were to consider this further I think we:

• need a better solution than probing for imports one at a time
• also need an automated way to test ALL supported environments

I didn't get around to setting up CI, but it's easy to make sure everything is golden in Wren Console, just run the test suite... if we're going to officially support other environments then there needs to be an equally simple way to CI those environs. It seems the minimal environment would almost need to be "pure Wren"... but that's impossible since such a thing doesn't exist... even "Wren CLI" adds a TON of stuff above pure Wren...

But mostly I'm still worried that testie isn't testie when you take away 80% of the functionality... To me a large benefit of this framework (and why I wrote it) is the detailed stack traces and comprehensive inline code summaries. I'm not sure I want to start saying "well you don't get those KEY features if your env doesn't support them". Again the existing Capabilities code going that direction is more legacy than current intention.

If you have any more thoughts or other ideas here, I'm all ears.

[joshgoebel]

But, let's say I wanted to use DOME or TIC-80? Aside from a few touches here and there, it's compatible.

I think for MANY use cases one could test Wren model code in another scriptable runtime (like Wren Console)... if was pure Wren it ought to have the exact same behavior - for testing purposes.

[joshgoebel]

I feel like you want more of a pure Wren test runner and wish testie was that, but I'm not sure I'm interested in that direction. As mentioned I'm not even sure it's worth it to support Wren CLI at this point given the issues with the current hacked first_test solution.

Just running tests on the CLI (properly!) seems to require thinking about the problem entirely differently... you'd almost have to print the name of each test - in real time... so that then when you hard-crashed you'd have a workable stack trace on the screen... (right below the test name)... that greatly restricts the type of reporters, etc...

---

If you went and built something different (or even forked testie) and it turned out later there were ways to re-integrate smartly (in layers) I'd be open to discussing that... like having "core Testie" vs "CLI testie" but right now I'm not seeing a super clean way to do that (or test it).

[ConfusedSky]

Actually in reading this just now tests are not guaranteed to have the same result across runs (one of the preconditions could now be different, etc)... so first_error and "call it again" is an even worse ugly hack than I originally thought and it may be best to remove it entirely...

Great point. I didn't even consider that

But mostly I'm still worried that testie isn't testie when you take away 80% of the functionality... To me a large benefit of this framework (and why I wrote it) is the detailed stack traces and comprehensive inline code summaries. I'm not sure I want to start saying "well you don't get those KEY features if your env doesn't support them". Again the existing Capabilities code going that direction is more legacy than current intention.

IMO the big draw for me is the elegant API, and that I don't really have to think about testing when I start a new project. Between testie.wren, reporters/cute.wren and src/expect.wren I don't really need much else

All these points make sense though.

[ConfusedSky]

If you went and built something different (or even forked testie) and it turned out later there were ways to re-integrate smartly (in layers) I'd be open to discussing that... like having "core Testie" vs "CLI testie" but right now I'm not seeing a super clean way to do that (or test it).

I was actually thinking the same thing

[ConfusedSky]

Like configurable with dependency injection

[joshgoebel]

Like configurable with dependency injection

Curious to see what you come up with. :)

IMO the big draw for me is the elegant API

Thanks, though -mostly- the API is all cribbed from other great testing frameworks/patterns. :-) I did put some effort into making it as flexible as it is to accommodate multiple styles vs just one.

src/expect.wren

I wish the Wren ecosystem was larger... you could make a good argument to split this out (nothing about it depends on testie), but that just makes it harder to use testie. That's the reason I vendored colors.... I haven't seen anyone else trying to solve packaging/deps issues like I have with the wren-package stuff... For Wren Console you could just declare wren-expectations as a depency and get one command installs, but that doesn't help anyone else...

And of course packaging/deps is a very runtime dependent problem as well...

[ConfusedSky]

I wish the Wren ecosystem was larger...

Yea... these things don't exist till they do. But same here... same here. The irony is that people don't like to join an ecosystem until it's established. So if I were to guess (if/when) we do see improvements to the ecosystem it's going to go from slow and stagnant to explosive growth

I haven't seen anyone else trying to solve packaging/deps issues like I have with the wren-package stuff...

Haha that's honestly the reason I found wrenc in the first place, I think I found it after testie cuz I saw the package.wren. Personally not the biggest fan on the node_modules like wren_modules rather than storing it in a ~/.directory, simply cuz it causes a lot of bloat when you've got a lot of projects with the same dependencies. Although it does help with packaging and distribution...

I think I might support deno style imports eventually

import "https://raw.githubusercontent.com/joshgoebel/wren-testie/0.3.1/testie.wren" for Testie, Expect

[joshgoebel]

You can install Wren modules in your home folder, just not a . folder. (With wren console)

[joshgoebel]

Closing. Updated README to indicate that this is wren-console only until such a time that wren-cli has the necessary capabilities to make this decision worth reconsidering.

Again, feel free to fork if that would be helpful!