APIv2 discussion

[inikulin]

This is a general place for the APIv2 discussion. I'll move proposals from wiki to here. Any suggestions/ideas/proposals are welcome. \cc @DevExpress/testcafe

[inikulin]

Assertion system

Use only async assertions

To increase stability I suggest to get rid of sync assertions. With new powerfull assertion system where will be no need for them. We will continiously execute assertion condition until it will not be met or timeout will not happen (3sec by default). This is the only possible consensus beetween stability and speed. E.g.:

test `Button color changes`
    .save('color', () => document.querySelect('#btn').style.color)
    .click('#change-color')

    .expect.element('#btn').style('color').not.eql.saved('color');

You can use custom timeout by using .within()

test `Button color changes`
    .save('color', () => document.querySelect('#btn').style.color)
    .click('#change-color')

    .within(3000).ms
    .expect.element('#btn').style('color').not.eql.saved('color');

Or if you want assertion to run once and immediately use now

test `Button color changes`
    .save('color', () => document.querySelect('#btn').style.color)
    .click('#change-color')

    .now.expect.element('#btn').style('color').not.eql.saved('color');

Actual argument

• expect(func) - no plain values, only funcs, since everything is async and should be calculated
• expect.stored(name) - stored value
• expect.element(selector) - element
• expect.element(selector).attribute(name) - element attribute
• expect.element(selector).width - element offset width
• expect.element(selector).height - element offset height
• expect.element(selector).count - number of elements matching the selector
• expect.element(selector).text - element inner text
• expect.element(selector).style(styleProp) - element computed style properties

Assertions

Common

• .eql(value), .eql(func), .eql.stored(), .eql.element(selector), ... - deep equality
• .to.be.true - boolean true equality
• .to.be.false - boolean false equality
• .to.be.null - null equality
• .to.be.undefined - undefined equality
• .to.be.NaN - NaN equality
• .to.be.ok - non-null, non-undefined, non-false
• .to.be.empty - empty array, object without fields, empty strings
• .more.than(value), .more.than(func), .more.than.stored(value) - more than comparison
• .less.than(value), .less.than(func), .less.than.stored(value) - less than comparison
• .contain(value), .contain(func), .contain.stored(value), .contain.element(selector), ... - string contain substring, array contain value, object contain key, element contain text, element contain element

String-only

• .match(regexp) - string matches regular expression

Element-only

• .visible - visibility check
• .exists - element present in DOM
• .hasClass(className) - element has one or more classname
• .disabled - element is disabled
• .readonly - input is readonly
• .checked - checkbox is checked

Negation

• .not.[assertion] - negotiate assertion

Timing prefixes

• within(ms).ms.expect, within(ms).expect - test assertion within given time range
• now.expect - test assertion once and immediately, sync assertion alternative

[inikulin]

Fixture before/after/beforeEach/afterEach

E.g.:

import deleteAllPosts from './mixins';

fixture `example.org forum functionality`
.beforeEach
   .go('http://example.org')
.after
   .do(deleteAllPosts);

[inikulin]

Roles

Use roles to observe results or perform actions from different user perspectives. Also, it's a solution for the forms authentication problem. Role initialization will be executed once per task on first demand and can be shared among tests and fixtures. Technically role saves created cookies and storages state. When we switch role in tests and if it's not initialized yet then initializations steps run and we return back to the page on which we stopped execution. If it's already initialized then page will be just reloaded with the new credentials.

helpers.js

import testcafe from 'testcafe';

export var registeredUser = testcafe.role()
    .go('http://example.org')
    .type('#login', 'TonyStark')
    .type('#password', 'swordFish');
    .click('#login');

export var anonymous = testcafe.role();

test.js

import { registeredUser, anonymous } from '../helpers';

fixture `example.org tests`;

test `Anonymous users can see newly created comments`
    .as(registeredUser)
    .go('http://example.org')
    .type('#comment', 'Hey ya!')
    .click('#submit')
    .as(anonymous)
    .expect.element('#comment-area').contain('Hey ya!');

[inikulin]

Waits

Autowaits are enabled by default.

Disable all:

.autowait(false)

Enable all:

.autowait(true)

Optional:

.autowait({ xhr: true, pageLoad: false, timers: true })

Forced (fails test if not occure), default timeout 3sec:

.waitPageLoad(timeout)
.waitXhr(timeout)

[kirovboris]

Conditional statements

I suggest to use next syntax:

.if(() => isBtnVisible())
    .click('#button')

    .if(() => isFormOpened('testForm'))
        .click('#submit')
    .endif

.elif(() => getCheckboxes().length === 1)
    .click('input[type="checkbox"]')

.else
    .click('#close-btn')

.endif; 

if(func), elif(func) parameter is function, which will used as condition of statement.

[churkin]

Looks harmoniously

[churkin]

How are we going to report about forgotten endif?

[inikulin]

@churkin Should we?

[churkin]

@inikulin I think this is a potential error when the number of if does not match the number of endif

[inikulin]

Well, we can raise error then.

[churkin]

Oh, we have to build a virtual logical tree and catch structure errors. Also with loops, if we will implement them.

[inikulin]

@churkin we will need virtual control flow tree anyway

[churkin]

@inikulin I was hoping we greatly simplify the analysis of the test code structure with the new API.

[inikulin]

Tree will be built in runtime via API calls, syntax analysis will not be involved

[VasilyStrelyaev]

Let's try to make assertions grammatically consistent.

If we stick to the expect smth to phrase, we would have to add a lot of wording:

.to.eql(value), .to.eql(func), .to.eql.stored(), .to.eql.element(selector), ...- deep equality .to.be.true - boolean true equality .to.be.false - boolean false equality .to.be.null - null equality .to.be.undefined - undefined equality .to.be.NaN - NaN equality .to.be.ok - non-null, non-undefined, non-false .to.be.empty - empty array, object without fields, empty strings .to.be.more.than(value), .to.be.more.than(func), .to.be.more.than.stored(value) - more than comparison .to.be.less.than(value), .to.be.less.than(func), .to.be.less.than.stored(value) - less than comparison .to.contain(value), .to.contain(func), .to.contain.stored(value), .to.contain.element(selector), ...- string contain substring, array contain value, object contain key, element contain text, element contain element

.to.match(regexp) - string matches regular expression

.to.be.visible - visibility check .to.exist - element present in DOM .to.haveClass(className) - element has one or more classname .to.be.disabled - element is disabled .to.be.readonly - input is readonly .to.be.checked - checkbox is checked

My suggestion is to use the phrase expect [that] smth

.eqls(value), .eqls(func), .eqls.stored(), .eqls.element(selector), ...- deep equality .is.true - boolean true equality .is.false - boolean false equality .is.null - null equality .is.undefined - undefined equality .is.NaN - NaN equality .is.ok - non-null, non-undefined, non-false .is.empty - empty array, object without fields, empty strings .is.more.than(value), .is.more.than(func), .is.more.than.stored(value) - more than comparison .is.less.than(value), .is.less.than(func), .is.less.than.stored(value) - less than comparison .contains(value), .contains(func), .contains.stored(value), .contains.element(selector), ...- string contain substring, array contain value, object contain key, element contain text, element contain element

.matches(regexp) - string matches regular expression

.is.visible - visibility check .exists - element present in DOM .hasClass(className) - element has one or more classname .is.disabled - element is disabled .is.readonly - input is readonly .is.checked - checkbox is checked

[AlexanderMoskovkin]

New APIv2.1

Fixture defenition

fixture `My fixture`;

or

fixture('My fixture');

Test, actions and assertions

test('My test', async t => {
    await t
        .click('#myelem')
        .expect.element('#myelem').visible;
});

Hybrid functions

var getElementById = Hybrid(id => document.querySelector('#' + id));

var elem = await getElementById('myid'); // -> returns ElementDescriptor (props from recorder)
                                                                 // Can return any serializable value as well

// Can be passed to action (in that case evaluated in one round trip to client):
await t.click(getElementById('myid'))

// Or you can pass ElementDescriptor 
await t.click(elem);

Roles

var registeredUser = Role(async t => {
    // initialization steps
});

// Usage
test('roleTest', async t => {
   await t.as(registeredUser);
});

Frame switching

await t
         .switchTo('#frame')
          ...
         .switchToMain();

Script injection

// Inject on all pages (and frames)
t.alwaysInject(() => {
   // code
});

t.alwaysInject('filename,js');

// Inject ones
t.inject(...);

Assertions

All assertions are async and chainable:

await t
    .expect.element('div').text.contains('Yo!')
    .expect.element('span').not.visible;

L-value

expect(value) - value, expect(hybridFunction) - hybrid function return value, expect.element(selector) - element, expect.element(selector).text - element text expect.element(selector).attr - element attr

R-value

.eql(value), .eql.element(selector). ...- deep equality .contains(value)- string contain substring, array contain value, object contain key, element contain text, element contain element .matches(regexp) - string matches regular expression

.is.true - boolean true equality .is.false - boolean false equality .is.null - null equality .is.undefined - undefined equality .is.NaN - NaN equality .is.ok - non-null, non-undefined, non-false .is.empty - empty array, object without fields, empty strings

Element-only:

.visible - visibility check .exists - element present in DOM .hasClass(className) - element has one or more classname .disabled - element is disabled .readonly - input is readonly .checked - checkbox is checked

Negation

.not + R-value .is.not for R-values that starts with .is

[inikulin]

I think we should restrict usage of the global test, fixture, beforeEach, etc. only to the test file and don't inject them to the globals on the project level, So, any module that will be require by test file will not be able to declare tests. Meanwhile, things like Hybrid and Role can be declared and used anywhere, but on the other hand they don't represent the minimal test harness. So, user will need to require them:

import { Hybrid, Role } from 'testcafe';

Let me know what do you think \cc @DevExpress/testcafe

[AlexanderMoskovkin]

I like this way

[miherlosev]

1)I prefer the following variant:

fixture `My fixture;`

2)If Hybrid function executing on client maybe we will name it as Client of ClientCode. 3)Frame switching is good. I think we should rename switchToMain -> switchToTop and add switchToParent. (simular as window.top and window.parent) 4)Script injection. I think that name is not clear. I should be injectClientScript, injectClientScriptForFrames, injectClientScriptForPages. The main goal - describe where we will inject script. 5)I think that Hybrid is a very usable feature. Therefore it should be avalible without additional import directive.

[inikulin]

@miherlosev

1)I prefer the following variant:

Both will be available

3)Frame switching is good. I think we should rename switchToMain -> switchToTop and add switchToParent. (simular as window.top and window.parent)

top frame may have different meaning depending in which frame you are currently. Meanwhile. we have only one main frame

4)Script injection. I think that name is not clear. I should be injectClientScript, injectClientScriptForFrames, injectClientScriptForPages.

Too long, however, I'm not quite happy with the original naming either

5)I think that Hybrid is a very usable feature. Therefore it should be avalible without additional import directive.

This will require globals pollution even in test dependencies. It's not good at all. Moreover, we will introduce conceptual inconsistency in test declaration interface and this utility API.

[AlexanderMoskovkin]

We need to add native dialogs handling api (alert, confirm, prompt, beforeUnload). Will it be look like an action?

await t.click(el1)    // causes alert
    .handleAlert()
    .someOtherActions...

[inikulin]

@AlexanderMoskovkin sure. But I'd prefer to not use predictive model like we had before:

await t
    .click(el) // <-- causes alert
    .handleAlertDialog();

[AlexanderMoskovkin]

Yeah, i mean the same

[inikulin]

Guys, just found fundamental issue with our current concept of Hybrid functions.

The problem: Currenlty we just declare Hybrid functions somewhere and execute them without passing them any test run context. So we don't know in which Hammerhead session given Hybrid function should be executed. It became even worse if we think about calling Hybrid function from fixture dependency.

The solutions: In general it's just the same solution with different control directions.

1. Hybrid function call should always accept TestApiHost as the first argument:

import { Hybrid } from 'testcafe';

var getElementById = Hybrid(id => document.querySelector('#' + id));

test('Yo', async t => {
    var btn = await getElementById(t, 'myId');
});

Pros: we have explicit way of hybrid function declaration, so there is no room for confusion. Cons: calling convention is a little bit confusing, it's not as elegant as we hoped.

2. Hybrid wrapper will be built-in into the TestApiHost:

var getElementById = id => document.querySelector('#' + id);

test('Yo', async t => {
    var btn = await t.hybridCall(getElementById , 'myId');
});

or

var getElementById = id => document.querySelector('#' + id);

test('Yo', async t => {
    var btn = await t.hybrid(getElementById)('myId');
});

Pros: calling convention looks a little bit cleaner, but still it's not quite elegant. Cons: it's hard to distinguish hybrid function declaration.

Please, let me know what do you think and if you have better ideas. \cc @DevExpress/testcafe

[churkin]

I vote for the first variant

[AlexanderMoskovkin]

I suppose it's critical to distinguish hybrid and common function declarations, so I choose the first.

[inikulin]

I vote for first as well, since it's also looks better in the action calls:

await t.click(getElementById(t, 'myId'))

vs

await t.click(t.hybrid(getElementById)('myId'))

However, it's still sucks =(( I was hoping we could create something better

[inikulin]

Well, actually there is a "black magic" way to keep old syntax, without passing t explicitly. We can wrap testrun-fn with the function with special name that will contain test run ID. When in Hybrid fn we can traverse up callstack using v8 StackTrace API to determine the test run. What's cool about it: we can keep the old syntax. What's not cool:

• it's may be fragile,
• we will need to introduce a global pool of test-runs as well, so Hybrid function will be able to peek appropriate test-run,
• behavior of the Hybrid function which was called out of the test run context is undetermined (we can just throw, actually).

[miherlosev]

I prefer the second variant:

var getElementById = id => document.querySelector('#' + id);

test('Yo', async t => {
    var btn = await t.hybridCall(getElementById , 'myId');
});

Because for this variant we directly declare that we call hibrid function in the specified test context. Also it is simular to http://api.qunitjs.com/QUnit.test.

I think for actions we need to create build-in selector instructions:

await t.click(t.findById('myId'))
await t.click(t.findByContainClass('myClass'))
await t.click(t.findByContainText('myText'))
etc.

or maybe

await t.findById('myId').click()

[churkin]

I am categorically against mixing hybrid and common functions

[churkin]

The hybrid function is not very obvious concept for a beginner, do not complicate the interface even more

[kirovboris]

I like the first variant too, but i think 'hybrid function' is still a complicated conception for novice. The term 'hybrid' doesn't associate with the call function on the client side.

[VasilyStrelyaev]

So, which is the final variant?

[inikulin]

@VasilyStrelyaev initial, without any context. But it's still doubtable and require further research

[VasilyStrelyaev]

Actual argument expect.stored(name) - stored value

We have the capability to pass a stored value as the expected argument, which does make sense. But does it make sense to use a stored value as the actual argument?

[inikulin]

@VasilyStrelyaev we don't have stored values in the APIv2. You can just use plain vars now

[VasilyStrelyaev]

Where can I see an updated version of APIv2?

[inikulin]

New API (25/2/2016)

Fixture defenition

fixture `My fixture`;

or

fixture('My fixture');

Fixture page definition (if not specified then defaults to about:blank)

fixture `My fixture`
    .page `https://example.org`;

or

fixture `My fixture`
    .page ('https://example.org');

Test, actions and assertions

test('My test', async t => {
    await t
        .click('#myelem')
        .expect.element('#myelem').visible;
});

Hybrid functions

var getElementById = Hybrid(id => document.querySelector('#' + id));

var elem = await getElementById('myid'); // -> returns ElementDescriptor (props from recorder)
                                                                 // Can return any serializable value as well

// Can be passed to action (in that case evaluated in one round trip to client):
await t.click(getElementById('myid'))

// Or you can pass ElementDescriptor 
await t.click(elem);

Roles

var registeredUser = Role(async t => {
    // initialization steps
});

// Usage
test('roleTest', async t => {
   await t.as(registeredUser);
});

Frame switching

await t
         .switchTo('#frame')
          ...
         .switchToMain();

Script injection

// Inject on all pages (and frames)
t.alwaysInject(() => {
   // code
});

t.alwaysInject('filename,js');

// Inject ones
t.inject(...);

Assertions

All assertions are async and chainable:

await t
    .expect.element('div').text.contains('Yo!')
    .expect.element('span').not.visible;

Actual-expr

expect(value) - value, expect.element(selector) - element, expect.element(selector).text - element text expect.element(selector).attr - element attr

Expected-expr

.eql(value), .eql.element(selector). ...- deep equality .contains(value)- string contain substring, array contain value, object contain key, element contain text, element contain element .matches(regexp) - string matches regular expression

.is.true - boolean true equality .is.false - boolean false equality .is.null - null equality .is.undefined - undefined equality .is.NaN - NaN equality .is.ok - non-null, non-undefined, non-false .is.empty - empty array, object without fields, empty strings

Element-only:

.visible - visibility check .exists - element present in DOM .hasClass(className) - element has one or more classname .disabled - element is disabled .readonly - input is readonly .checked - checkbox is checked

Negation

.not + expected-expr .is.not for expected-expr that starts with .is

[VasilyStrelyaev]

Cool! Notice no beforeEach, afterEach, page in fixture... No waits or autowaits... Actions won't change from what I gather...

[inikulin]

• Added page, but semantics has changed.
• beforeEach, afterEach - not sure if we'll be able to finish them for the first version.
• waits and autowaits - are in discussion and not sure if we'll be able to finish them as well. For now it will work as it worked before (autowaits always enabled).
• Actions won't change in general, but they now accept regular functions, Hybrid functions, element descriptors returned by Hybrid functions and strings as a selector. No arrays, no jquery objects

[VasilyStrelyaev]

.eql(value), .eql.element(selector). ...- deep equality

does "deep equality" relate to both value and element?

[inikulin]

@VasilyStrelyaev elements are equal if they are the same element.

[VasilyStrelyaev]

Thanks!

one more thing - no expect(hybridFunc) so far?

[inikulin]

expect(val) works with raw values. So it will examine hybrid functions as plain JS object. To convert hybrid function in the selector you will need to pass it to expect.element()

[VasilyStrelyaev]

Seems that t.alwaysInject is more like t.injectEverywhere, isn't it?

[inikulin]

@VasilyStrelyaev this one is still debatable. I'm starting to think that we should use this syntax instead:

fixture `Fixture`
    .useClientHelper('dx', '../dx.js'); // Will be built using browserify

test('Some test')
   .useClientHelper('$', '../vendor/jquery.js')
   .body(async t => {
       // Helpers are visible via provided alias in the scope of Hybrid function.
       var gridElem = Hybrid(() => dx.getGrid())();

       t.click(() => $('.btn'));
   });

\cc @DevExpress/testcafe What do you guys think?

[kirovboris]

I like this way. Also I think an alias should be the second argument, because it isn't always required.

[inikulin]

@kirovboris how you will access helper without alias?

[kirovboris]

E.g. if i want to prepare loaded site in a certain way - set start sizes or change css or someone else.

[VasilyStrelyaev]

test('Some test').body(

Oh, changes arrive so fast... :(

useClientHelper

makes sense to me