Command line runner, CI server support

[TobiaszCudnik]

I'd like to execute my tests from the command line. Didnt find any way to do this with wallaby, which is quite weird. Are you planning to support only IDEs/editors?

[ArtemGovorov]

Wallaby.js main idea is to integrate with editors, run tests for the code that you change and display the results in the editor. Providing a command line runner is relatively easy, but why would you want to use it instead of a grunt/gulp task for example?

[TobiaszCudnik]

Thanks for the quick response Artem.

Actually I care about executing tests in the terminal, so gulp/grunt would work for me as well. While I think that IDE/editors support is great, it creates a dependency on them (and different people use different editors). Also it can't be automated on a CI which expects a json output of the tests results for example.

I know I cant set it up manually (which I usually do), but wallaby seemed to offer all the features out of the box (mocha, babel and webpack in my case). I just miss the most basic one ;)

[ArtemGovorov]

I have to admit it's a bit unexpected feature request for a commercial tool like wallaby, as one of its strong (and competitive) parts is tight integration with editors and running tests incrementally. There are many free and OSS alternatives to just run all tests from CLI or on CI server. In any case, I'm open for all ideas that are useful for wallaby user, adding "enhancement" label.

[fvanwijk]

You are focusing so much on the IDE awesomeness that you sometimes forget the Wallaby is also a lot faster than i.e. Karma, thanks to web workers. So in a CI environment Wallaby in command line mode would be a good friend too ;).

[ArtemGovorov]

Fair enough, will add a command line runner.

[nickpresta]

Just so I am understanding the documentation, the only way to run Wallaby.js is through an IDE/editor of some sort?

If you're using emacs, Vim, or anything not support today, you are out of luck?

Thanks

[ArtemGovorov]

@nickpresta yes, at this stage wallaby.js is only supported in the listed editors.

[fvanwijk]

You could use SkippyJS. Still WIP, but runs on command line.

[ghost]

I also would like to run the tests with Wallaby on the command line for Continuous Integration. This would save me the configuration work for other tools like Karma and Jasmine CLI runner.

[samhatoum]

+1

[karl]

:+1:

I'm investigating Wallaby as a possible alternative test runner at work and was very surprised to see there is no way to run it on the command line! It was the first thing I looked for.

It seems that your have to use a different test runner for the command line, with its own version of the test configuration that must be maintained and kept in sync with the Wallaby config (with the risk that tests pass locally but not on CI due to the config getting out of sync or due to differences in how the test runners work).

Is this the official way we are expected to use Wallaby, or have a missed something!

[ArtemGovorov]

@karl Correct, for now wallaby.js is not a CI tool. Its focus is to help you to be more productive when writing your code (by running tests, but the emphasis is on the development time experience).

It can't guarantee that your tests will pass on CI server, in all browsers, across all devices that your web app intends to support, etc. We'll get there sooner or later, but for now if you need to run your tests on CI server, in different browsers or on different devices, I'd recommend to use another solutions. BTW, with the testRunerConfig module you may re-use a big part of your config for different test runners.

[aarondandy]

@ArtemGovorov I would just like to let you know that configuring wallaby worked great while setting up karma for our scenario involved a good deal of frustration. I would love to be able to use your tool on our CI server provided you support it from a technical and licensing point of view.

[jrista]

I have to throw in my support for a Wallaby CLI as well. I've been struggling more and more with Karma lately, as I write more and more ES6 and TS code, using more and more modern features, frameworks, etc. Karma is falling behind, not keeping up with the times, and has really just become much more of a hinderance than a compliment to my daily routines.

I am really impressed with what Wallaby offers, the IDE integration and real time, continuous instrumentation of my code is pretty nice. However, at the end of the day...it all has to run through CI. That, at the moment, means extra work to keep Karma working, which is a lot harder than keeping Wallaby working.

Having a Wallaby CLI would definitely reduce the daily overhead of keeping CI rolling.

[rollymaduk]

+1

[zedL]

+1

[IPWright83]

+1

For us we just find it a pig to get tests running consistently. We have tests working in Karma that don't in Wallaby, and vice versa. The more time we have to spend making 'good tests' run in another environment the more time we've wasted fixing actual issues or writing more tests.

[ArtemGovorov]

@IPWright83 Not related to the feature request, but if you create a sample of tests working in Karma that don't in Wallaby, I'm happy to have a look and find out why it's happening (perhaps different test runner is used, in which case you may try Electron or the same PhantomJs version as used in Karma).

[IPWright83]

At the moment it's worse the other way around. Tests failing in wallaby are due to es6 (Symbol not being defined) and I guess a bad configuration.

On 12 Sep 2016 14:08, "Artem Govorov" notifications@github.com wrote:

@IPWright83 https://github.com/IPWright83 Not related to the feature request, but if you create a sample of tests working in Karma that don't in Wallaby, I'm happy to have a look and find out why it's happening (perhaps different test runner is used, in which case you may try Electron https://wallabyjs.com/docs/integration/electron.html or the same PhantomJs version https://wallabyjs.com/docs/integration/phantomjs2.html as used in Karma).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wallabyjs/public/issues/72#issuecomment-246341492, or mute the thread https://github.com/notifications/unsubscribe-auth/ABT6N_9nQ2M1Sdbtku1uPAYuAcpLWslKks5qpU7GgaJpZM4EBv_x .

[ArtemGovorov]

@IPWright83

Symbol not being defined

It definitely sounds like the default runner (PhantomJs 1.9.8) limitation. Try using the Electron runner, it's similar as running your code in the latest Chrome. Another solution is to use some ES6 polyfill, for example npm i babel-polyfill plus add it to the files list.

[samhatoum]

While people are waiting for this, you might be interested in this setup I have going where I run Wallaby.js locally and Gulp + Karma + Mocha on CI.

https://github.com/xolvio/qualityfaster/tree/dev

Hope that helps someone

[bfricka]

I can understand the desire to maintain a narrow focus on your goals of seamless, high-quality editor support and speed. However, by ignoring CI, a huge portion of your market is going to be forced to write the same complex, convoluted build and test scripts they are desperately trying to avoid by switching to Wallaby, because many companies have CI requirements for tests and coverage as a part of their deployment pipeline. And frankly, these requirements make sense to me, since they are globally visible.

Additionally, being able to use Wallaby as a simple, headless runner for free would be a very useful feature. I see this being beneficial in two scenarios:

1. Companies who use contractors, who are required to write unit tests, but who cannot really be forced to cough up the money for all editor licenses.
2. Open source developers who would dramatically increase the uptake of this tool. Capturing more market share could really drive demand within the orgs that these developers belong to, and I don't think it would take away from the excellent benefits of having tight IDE integrations. Think of this like JetBrains Community Edition products.

Just a thought, on the last points. But with regard to lack of CI: For our company this makes the proposition of buying a bunch of licenses much harder to justify. I'll be evaluating this tool, and it's going to be hard to recommend without this feature.

[ArtemGovorov]

@bfricka Thanks for the feedback. We are planning to support the CI scenario, it's our next major goal right after we provide/improve test debugging experience.

Given that CI support is happening in wallaby sooner or later, it's worth mentioning that switching to wallaby for CI builds will obviously be much easier and straightforward for those companies that are already using wallaby and have it set up and running on dev machines.

[tugend]

+1

[jonaslewin]

+1.

We also want to be able to use wallabyjs as a command line runner in our CI environment

[xav4nte]

+1

[mbcooper]

+1. And option to generate lcov file. But ... It might be more problematic than a simple test runner

[benvineyard]

+1

[BLamy]

I bought wallabyjs about 6-8 months ago because I loved it. Work on the other hand turned me down because of this exact issue. This issue would sell licenses. Is there any way I can contribute? I would really like to see this.

[zpydee]

+1

[ChrisBellew]

I would also contribute to get this.

[noomorph]

+1

[ondrasak]

What is the status? :)

[benvineyard]

I can beta test with my company. Currently we use Visual Studio Team System as our git build/CLI server.

[vreddi]

Curious about the status as-well. Is there any documentation available for this if/when this is available?

[hiredgunhouse]

I'd love to have such runner cause I could use it in CI to have faster build and more consistent setup and also occasionally I could use it on a dev machine if I don't feel like (or cannot) run my IDE. I believe this could be a big bonus point for Wallaby that would increase your user base (and thus sales, it would make it easier for us devs to convince our managers to pay for Wallaby). Please, please implement a CLI runner, especially considering you said that it should be relatively easy. BTW. NCrunch (Wallaby for .NET) already has a CLI runner that you can use for CI.

[elevatebart]

I would really love a use of wallaby in Command line for two purposes :

• some people in my company use wallaby some don't take the time to install it. They still write unit tests and sometimes create mocks. Sometines those mocks are not included in wallaby.js but are in karma.conf.js. Once they are done, I would like to know on the CI if wallaby is still running. If they are broken I can then repair. It would probably drive the adoption rate.

• I have contributed to a couple templates around the webs such as vue-webpack-template and I would love to be able to propose integrations of wallaby on those. It is impossible to run tests on those integrations if wallaby does not run command line.

Thank you for making js testing so pleasant. Bart

[carlansley]

Not everyone uses wallaby all the time here either. Without command line, we have no automated/CI way of ensuring wallaby+WebStorm still works for new commits. It's an increasingly significant pain point for us, and definitely creates a barrier to wallaby use.

[BLamy]

We've given up and moved to just using jest.

https://github.com/jest-community/vscode-jest

[Alino]

+1

[IPWright83]

@ArtemGovorov is there any update on this? It's been over a year since the last update and it would be good to know if/how it's progressing.

[ArtemGovorov]

@IPWright83 No updates/ETA to share yet, but having said that, we are planning to grow the team this year and CI support (most likely as a separate product) is one of our top priorities.

[cecilemuller]

Depending on the test runner, in the meantime it's possible to use regular CLI tools to run tests on CI: Jasmine + Typescript example (but of course it would be nicer to use Wallaby on CI directly 😃)

[JacopKane]

For me, this feature is essential when developing the wallaby configuration itself.

Cause It requires extra configuration even for a simple scenario as CRA app (Thanks to you guys usually you can find a doc or GitHub issue for everything).

I probably spent two days while trying to make Wallaby seamlessly work in a complex monorepo structure. Most of the scenarios on the web didn't work as a drop-in solution and always led to another problem and needed to keep googling.

At least a pure open source CLI to debug as would make this process much more comfortable to debug instead of keeping restarting it inside Vscode. I'm sure you or maybe as a community we can achieve this somehow as your plugins already open-source.

If there's something similar already for debugging, ofc I would love to hear.

[IPWright83]

We've been using Wallaby for a few years now, and unfortunately this is becoming a significant pain point. We have to use different test runners on the build servers, and their characteristics are just different to Wallaby in how they run the tests which can cause problems.

The most recent for example, in the ideal world no test would have side-effects on another test. It looks like a recent change allows Wallaby to run multiple tests within a single file at the same time - which causes tests to fail in Wallaby but not on the build server. Due to the frequent frustration of tests failing on the build server, a number of people in the team are now resorting to using the command line runners instead of Wallaby - which leaves our wallaby tests in a broken state.

Unfortunately this is significantly de-valuing wallaby for us, not being able to have a gated check-in process that works exactly the same as the local dev environment hinders workflow & wastes hours of development time.

@ArtemGovorov do you still consider Providing a command line runner is relatively easy to be true? If so could you guys look into it - it's not moved in over 3 years and is becoming a real struggle to justify renewing our licences, yet I'm fairly sure everyone in the team would be happy if it "just worked".

[ArtemGovorov]

@IPWright83

A new team member is joining Wallaby in a couple of months and the current plan is that he'll take a look into this feature.

Could you please clarify a couple of things for us to better understand the issues that you're currently facing with Wallaby running tests differently?

It looks like a recent change allows Wallaby to run multiple tests within a single file at the same time

Wallaby never runs tests within a single file at the same time. We haven't added anything like that recently, and there's no plan to add anything like that. If you share a specific example/repo - we are happy to take a look, maybe there's an issue with configuration. Because despite some differences in how Wallaby runs tests by default comparing to other runners, there's almost always a setting to make it work the same/similar way to how the underlying test runner works.

which causes tests to fail in Wallaby but not on the build server. Due to the frequent frustration of tests failing on the build server

So tests are failing on the build server too, or in Wallaby only?

[IPWright83]

Hi @ArtemGovorov, I don't want to focus too much on my issue in this thread (but thanks for the clarification - I'll pick this up in a separate issue if I need further assistance) and more highlight the problems that we face having a different CI runner and how the impacts us as a team and effects potential uptake/renewal of Wallaby.

[ArtemGovorov]

@IPWright83 Sure, no worries at all, we appreciate your feedback! Speaking of your team and CI server - what CI server(s) are you guys using?

[IPWright83]

We currently have 8 active wallaby users (we peaked at 10).

We have 20 packages in a browser based mono-rep that can be split:

• 17 of these use jest on node on a dev machine.
• 3 of these use jasmine in chrome on a dev machine.
• 17 use jest on node on the build server.
• 3 use karma in chrome on the build server.

We use both Jenkins (main build server for JavaScript with test running) and TeamCity (currently builds the overall application but doesn't run JS tests).

We also have 4 node reps:

• all of these run mocha on a dev machine.
• all of these run mocha on a build server.

The node reps are build on TeamCity with tests running on them using mocha.

[ArtemGovorov]

Thanks! Appreciate your feedback and sharing the details.

We use both Jenkins (main build server for JavaScript with test running) and TeamCity (currently builds the overall application but doesn't run JS tests).

Great, thanks for sharing. The reason we've asked about it is because there's a plan to have a dedicated person (joining by the end of the year) working on the CI servers feature, including possibly nice plugins for various CI servers, as opposed to just releasing a CI tool, so we'd like to better understand where to start.

If anyone else subscribed to this issue could share what CI servers you and your team is using, as well your team size (optional), that'd be very helpful.