What is the official way to run jest programmatically?

IgorAufricht commented 6 years ago

My question is how to run jest programmatically. There are some examples on the internet: https://stackoverflow.com/a/30562836/1652273 https://stackoverflow.com/a/33669530/1652273 https://github.com/facebook/jest/issues/3848

All of them reference requiring jest or jest-cli and calling runCLI(), but it does not seem to be documented anywhere (I haven't found anything in the jest docs nor is there any documentation for the jest-cli package).

Is this the official supported way? Thanks.

IgorAufricht commented 6 years ago

I guess my main question is: is the runCli() a part of public API?

For example this commit https://github.com/facebook/jest/commit/200b032d170054dc3860f3cd869a94201c54dd19 changed the runCli() - it removed the onComplete callback and replaced it with a Promise, which is not reflected in the commit message and/or the changelog.

thymikee commented 6 years ago

I think there's no official way, that's why it's not referenced in the docs and changelog. Other way to do it would be jest-editor-support package by @orta

@cpojer what do you think about an "officially supported" way of invoking Jest as a node process? @IgorAufricht would you be interesting in writing about this?

IgorAufricht commented 6 years ago

To add a bit of context - I'm adding jest into our grunt-based build process. I basically have these options:

Run jest outside grunt (e.g. the build script could be something like npm run jest && grunt build)
Use the deprecated grunt-jest
Run jest from grunt as a process (e.g. using grunt-run)
Run jest programmatically from a grunt task From these options the nicest would be the last one, hence my question.

If there is currently not an official way to run jest from node, then adding it would involve more than just a documentation change (probably some more thinking about the public API, adding tests and documentation, etc.). While I personally think it might be useful to add an official way to run jest programmatically, it's up to the jest maintainers to decide whether to add support for this or not.

@thymikee I'm not entirely sure what you're asking (writing about what?). But yes, I'm happy to help.

cpojer commented 6 years ago

@thymikee yeah, I'm not opposed, Jest already has an API that can be used but it requires people to read the source to understand how it works, so it's not optimal. The other thing is that we may choose to break the API from major to major.

orta commented 6 years ago

FWIW, I'd love this - I think exposing a consistent JS API to run tests in jest-editor-support would mean that jest-x deps APIs can break between majors, but editor-support could provide a facade for whatever has to actually happen under the hood.

cpojer commented 6 years ago

I'm gonna close this because the issue is a question. If you'd like to use Jest's programmatic API, check out the main module of jest-cli. If you'd like it to do things that it doesn't currently do, please send as a well-tested PR that adds or changes behavior.

tquinlan1992 commented 6 years ago

const jest = require('jest');

jest.run(argv);

slavafomin commented 5 years ago

I'm also interested in running jest programmatically from node.js, sadly it's not documented anywhere and the API doesn't look very thoughtful for this use case.

SimenB commented 5 years ago

It's not. That's something we want to improve though (although a bit down the priority list at the moment)

slavafomin commented 5 years ago

@SimenB I've just found out that Jest is working pretty fine in this scenario actually, the only problem is the documentation and typings support.

I've posted a small example on how to use Jest in this scenario: — How to run Jest programmatically in node.js (Jest JavaScript API)

@cpojer Is there an issue where the progress on this feature could be tracked?

SimenB commented 5 years ago

No issue currently. We'll try to split up jest-cli for the next major and clean up the programmatic API at least a little bit (so that yargs and argv stuff is removed). After that, we'll probably create an issue trying to gather feedback for what use cases people have for using Jest programmatically in order to decide how to best move forward

slavafomin commented 5 years ago

@SimenB that sounds great. Thank you for your hard work!

We are calling Jest as part of our build framework inside of the existing node.js process. The current implementation works great for us, we get the console output as well as results object with more information regarding tests outcome.

My only suggestion would be to have a better naming for the API and a typings support, so the module could be used in the TypeScript environment directly.

SimenB commented 5 years ago

We're going to be migrating to TS (#7554), so typings will definitely be there 🙂

brendon-codes commented 5 years ago

Just in case anybody is wondering, I was able to successfully do this:

npm install --save-dev jest jest-cli

const jest = require("jest-cli/build/cli");
jest
  .runCLI(
    {json: false},
    [process.cwd()]
  )
  .catch((error) => {
    console.log("Error:");
    console.log(error);
  })
  .then(() => {
    console.log("Done");
  });

EDIT: Fixed typo.

SimenB commented 5 years ago

Teeny, tiny start here: #7696. Not ready for consumption (don't rely on us not breaking its API without bumping major), but it will happen at some point 🙂

JSONRice commented 5 years ago

Just in case anybody is wondering, I was able to successfully do this:

npm install --save-dev jest jest-cli

const jest = require("jest-cli/build/cli");
Jest
  .runCLI(
    {json: false},
    [process.cwd()]
  )
  .catch((error) => {
    console.log("Error:");
    console.log(error);
  })
  .then(() => {
    console.log("Done");
  });

Small typo Jest should be jest

SimenB commented 5 years ago

Please don't reach into the build directory - that's bound to break.

We've created a @jest/core package in master (the PR linked above - will be part of the next release, whenever that happens) that will evolve into the programmatic API. It's API should be considered experimental (aka, don't come complaining if it breaks outside of a major), but it's moving forward at least 🙂

JSONRice commented 5 years ago

Please don't reach into the build directory - that's bound to break.

We've created a @jest/core package in master (the PR linked above - will be part of the next release, whenever that happens) that will evolve into the programmatic API. It's API should be considered experimental (aka, don't come complaining if it breaks outside of a major), but it's moving forward at least 🙂

@SimenB hey that sounds really good. Along with the new code a tutorial would be very nice. Thank you.

SimenB commented 5 years ago

Whenever we've landed on an API we're happy with, we'll document it properly as well 🙂

JSONRice commented 5 years ago

Whenever we've landed on an API we're happy with, we'll document it properly as well 🙂

@SimenB also I'd be happy to be your use case tester as I'm sure others here would be. 💯

ghost commented 5 years ago

@SimenB I would also love some documentation on this area. Looking forward to it! :+1:

SimenB commented 5 years ago

The core team will be meeting up in London later this month, and I've written this issue down as one of the things I'd like to discuss at that time. So hopefully we'll have a plan for programmatic API after that 🙂

another-guy commented 5 years ago

@SimenB ❤️ it's very nice to hear. Is there any way to affect the priority on this issue?

SimenB commented 5 years ago

Until we gave a plan (which we'll hopefully have in 3 weeks or so), no.

In the meantime, maybe you can provide som API suggestions? What do you want from a programmatic API?

The first thing that comes to mind as difficult (the way things currently are) is to trigger events in watch mode - the only interface is internal FS watchers and watch plugins, nothing that can be interacted with from the outside. But is that something you need, or is just spinning up Jest and awaiting the promise it returns enough?

So if we can focus on "run jest" and "run tests multiple times (and interact with a running version of Jest)" as separate things, that'd probably be good.

For one-time runs, do you need an argv parser? Should you be able to provide partial config, or mix in defaults yourself?

SimenB commented 5 years ago

Can open this for tracking purposes

another-guy commented 5 years ago

@SimenB Thank you for being open to proposals! I'd definitely consider offering input on the API shape but I don't feel qualified at this point.

My interest is coming from a specific use case, however. You can read about it on Medium: End-to-End Testing VS Code Extensions via Jest: Setting Things Up.

My only suggestion for now would be to provide first-class typings please. The existing typings felt outdated to me. When I'm calling runCLI() in my code, I'm passing the first argument as any. And I had to dig through the jest-cli and jest-config code base to see what I need to pass in. Even if documentation is in a perfect shape, many developers are editor's code-completion driven. 🤷‍♂️

If I was a lib contributor, I'd try to make jest-config responsible for providing API for

reading config objects from files,
creating config objects from scratch (by hand),
merging config objects hierarchically.

While jest-cli to be responsible for exposing APIs that allow triggering Jest executions:

via raw CLI parameters,
as well as by passing config objects.

Hopefully, it makes sense. I'm probably missing other important functions, but this seems to be minimal set I may need. 🤔

P.S. For Watch functionality you may want to get more input from someone else. I'm unsure here.

SimenB commented 5 years ago

Thanks for the feedback!

My only suggestion for now would be to provide first-class typings please.

Jest 24.3.0 has been rewritten in TypeScript, so all modules provide full typings

another-guy commented 5 years ago

@SimenB then I'm missing something because as I said

I had to dig through the jest-cli and jest-config code base to see what I need to pass in.

...as in typings were misleading in my case and I had to fall back to runCli(config as any, ...) in my code

SimenB commented 5 years ago

Oh sorry, Jest 24.3 was released earlier today so I assumed you'd tested with an older version.

As mentioned, we don't really have a public API meant for consumption yet, but typings should be in place for all modules. You'll probably have to jump through a few hoops to be able to use it properly though (we only really have a CLI API and yargs handles creating the object passed to runCLI)

another-guy commented 5 years ago

@SimenB the type info showed here is missing a config field... Either I am doing something very wrong then, or the typings are not quite right. 🤷‍♂️ Sorry, I'm in the office now and can't offer more details than below.

  "devDependencies": {
    "@types/jest": "^24.0.9",
    "@types/jest-cli": "^23.6.0",
    "@types/node": "^11.10.4",
    "jest": "^24.1.0",
    "jest-cli": "^24.1.0",
    "coveralls": "^3.0.3",
    "prettier": "^1.16.4",
    "rimraf": "^2.6.3",
    "ts-jest": "^24.0.0",
    "ts-node": "^8.0.2",
    "tslint": "^5.13.1",
    "tslint-config-prettier": "^1.18.0",
    "typescript": "^3.3.3333",
    "vscode": "^1.1.30"
  }

The dependencies were up to date with latest (at the time of coding -- about a week ago).

SimenB commented 5 years ago

As mentioned, we released a version of jest with bundled typings today. You're not looking at Jest's official typings, I don't know where your typings are from

ghost commented 5 years ago

I actually managed to get Jest running programatically using the API by piecing together bits and pieces of code from various places on the internet.

As it stands now, it's working perfectly for me. Can't really think of anything I'd need at the moment.

All I need/needed was a basic example on a wiki page, that could have spared me some time. But I guess that will arrive when the API goes public?

I like Jest more and more the deeper I dive, thank you for sharing a great piece of software! 👍

SimenB commented 5 years ago

Talked with @orta about this today, and we concluded that we don't really know how this should look. 😀 I'll just dump my notes from our conversation here so they're not lost (no concrete API suggestions here really, just some rambling about use cases and how they might work).

We figured that the thing that makes sense is:

Invoke jest with a partial of the configuration, and it returns a stream of events (once the stream ends, the jest run is complete)
- not sure if this should be rx, node core event emitters or something else entirely
Expose some way of loading Jest's configuration (a la babel's loadOptions, but not as fancy) that can be modified, then passed to jest
- should mix in our own stuff with the partially supplied config as well
jest needs to expose its reporters in a way that allows you to print the results as they come back in a way that pretty much looks like jest

Things out of scope for a first iteration:

watch mode (not sure how to allow you to send signals to the watch mode to make use of it)
run jest as a server, and ask it to execute tests to avoid the initial startup cost
CRA currently wraps Jest, we should support their use case without them needing to spawn the Jest CLI

wadethestealth commented 5 years ago

Moving forward, which is the module that is supposed to support programtic calls jest-cli or jest-core?

SimenB commented 5 years ago

it'll be @jest/core. Still no news on this btw, I haven't had the time to design or discuss anything more here

lh0x00 commented 5 years ago

in my toolkit have a test runner based on Jest, currently I using like below:

import jest from 'jest';

const config = {}; // something
args.push('--config', JSON.stringify(config));

await jest.run(args);

I suggest syntax to use like this, same webpack:

import * as jest from 'jest/core';

const runner = jest(config);

// we can add some event listeners before run
runner.on('event', callbackOfEvent);

runner.run(callback); // run once
runner.watch(callback); // run watch

what do you think about this?

SimenB commented 5 years ago

instantiating jest then calling run is an idea I can get behind! We probably won't add events or watch for a first go, but it will leave the door open for it to be added without breaking the API. I like it!

kiprasmel commented 4 years ago

Up - any updates on this? The lack of documentation sucks..

I don't know how to reproduce the same behaviour as running jest --config=./path/to/config and just ended up using execSync (from child_process) to call the npm script for running tests lul

https://stackoverflow.com/a/32872753 https://nodejs.org/api/child_process.html#child_process_child_process_execsync_command_options

joscha commented 4 years ago

we have code like this:

async function getListOfTests(partition: Partition): Promise<string[]> {
  // unfortunately the jest node interface is quite rudimentary at the moment,
  // there is no way to redirect the output to a different stream whilst using
  // a public API, so we need to capture it
  // See https://github.com/facebook/jest/issues/5048 and
  // https://github.com/facebook/jest/pull/7696
  // so for the time being we hijack stdout
  // TODO(joscha): remove this workaround once https://github.com/facebook/jest/issues/5048
  //  is closed
  const output = await capture(async () => await runCLI({
    _: [],
    $0: '',
    listTests: true,
    json: true,
    testPathPattern: partition.testPathPattern,
    testPathIgnorePatterns: partition.testPathIgnorePatterns,
  }, [rootDir]));

  return (JSON.parse(output) as string[])
  // we sort here to keep the list as stable as we can
      .sort((a: string, b: string) => a.localeCompare(b));
}

with

/**
 * Captures writes to a given node process for the time of the execution of a given function
 *
 * Caveats: monkey-patches process.stdout, output that is bigger than the node heap will cause an
 * overflow because it will be kept in memory instead of piping it through
 */
async function capture(
    fn: () => Promise<any>,
    p: WriteStream = process.stdout,
): Promise<string> {
  const originalWrite = p.write;
  let output = '';
  try {
    p.write = (chunk: Uint8Array | string) => {
      if (typeof chunk === 'string') {
        output += chunk;
      }
      return true;
    };
    await fn();
  } catch (e) {
    throw e;
  } finally {
    p.write = originalWrite;
  }
  return output;
}

if it helps someone as a workaround. We use it to shard our tests on CI automatically using https://github.com/joscha/ShardyMcShardFace to achieve something like in https://github.com/facebook/jest/issues/6270

pauldraper commented 3 years ago

instantiating jest then calling run is an idea I can get behind!

For comparison, see Jasmine

import * as Jasmine from 'jasmine';

const jasmine = new Jasmine();
jasmine.loadConfigFile('spec/support/jasmine.json'); // or jasmine.loadConfig
jasmine.configureDefaultReporter(...); // by default, console
jasmine.execute();

https://jasmine.github.io/setup/nodejs.html

nicojs commented 3 years ago

Whenever this gets implemented it would be awesome if we could leverage --bail without the process exiting on us. Right now this is not possible without monkey patching process.exit. It would be awesome if we could leverage --bail in Stryker.

OR13 commented 3 years ago

I'm not sure if this is needed to get what I want, but I landed here searching for it.

I love jest, and I want to write a web server which will run jest tests against user supplied input, instead of fixtures checked into version control.

consider a function magicOperation which takes an input and produces an output.

everyone implements magicOperation differently, and the input and out are complicated, so I want to describe magicOperation in terms of its input and output and call some helper functions in it and cover positive and negative tests, then I want to produce JSON associated with the test run and return it in the response to the web request.

now the web server will run the magicOperation test suite on any input and output supplied by a client and report failures over http.

I was not able to figure out how to do this with Jest, so I ended up implementing it with vanilla JS functions... and its feeling a bit awkward...

Here I apply the approach to demonstrate normative statements of a specification are testable with positive and negative tests... https://w3c.github.io/did-test-suite/#did-parameters

If anyone has a better solution to this problem that does not involve running jest programmatically, I would be very grateful... here is the repo with my current solution: https://github.com/w3c/did-test-suite

EDIT: I ended up pooling some of the code from the comments here, and building this: https://github.com/transmute-industries/vc.transmute.world/tree/master/packages/jest-test-server

evenstensberg commented 3 years ago

Any plans to make a programatic api of some sort?

JustinGrote commented 3 years ago

Came here wanted to test my vscode extension E2E UI with Jest, but finding the runCLI method rather limiting

socketopp commented 3 years ago

Any progress here? I want to run njsTrace. However, it requires Jest to be run similar to Jasmine

var Jasmine = require('jasmine');
var jasmine = new Jasmine();
jasmine.loadConfigFile(config);
jasmine.execute([], filters);

pauldraper commented 3 years ago

See https://github.com/facebook/jest/issues/9233 for a request for custom instrumentations.

socketopp commented 3 years ago

See #9233 for a request for custom instrumentations.

I'm not sure that's 100% related. So when using a test suite, like jest, you will have to run your own "test runner", say create a "test-runner.js" file, which will include njsTrace and call inject, then you will have to somehow "require" jest and run your tests. Unfortunately, there is no way of require jest, is there?

JustinGrote commented 3 years ago

@socketopp not really, you can use the RunCLI method to invoke tests, but it's not supported. Why on earth they haven't invested in a simple programmatic interface is beyond me.

manuman94 commented 3 years ago

Hi people! We are planning to migrate our Jasmine projects to Jest. @JustinGrote what is the problem of using runCli()?

I have done some tests and it seems to work without problems. I can build the configuration and pass it to the runCli function, useful for us.

We want to know if we are making a bad approach. Thanks!

JustinGrote commented 3 years ago

@manuman94 it's not supported and could break at any time, that said it's the "best" approach if you need to run tests using jest.

jestjs / jest

What is the official way to run jest programmatically? #5048