Add automated tests

[kusti8]

As Proton Native grows, automated tests are needed for it to work smoothly. There are a lot of edge cases that should be tested.

Currently it seems like the only way is with Xvfb, which is pretty complicated to handle with. Ideally there would be a way to test only the logic, like making a fake libui-node without GUI or something like that, or possibly on the proton-native side. I need to research and brainstorm more.

[mischnic]

Currently it seems like the only way is with Xvfb, which is pretty complicated to handle with. Ideally there would be a way to test only the logic, like making a fake libui-node without GUI or something like that, or possibly on the proton-native side. I need to research and brainstorm more.

= Creating a wrapper around libui-node and recording all library calls. For libui-node we will soon be comparing a screenshot from Xvfb to reference screenshots (see here).

[kusti8]

Yeah that's sort of what I was thinking. Comparing screenshots wouldn't work as well for proton native cause most of it is state stuff and logic stuff, not display.

[mischnic]

Just for later reference, this is how Vuido does it (mocking libui). Maybe the mocked libui code can be automated

test/index.js:

const chai = require( 'chai' );
const sinon = require( 'sinon' );
const sinonChai = require( 'sinon-chai' );
const mock = require( 'mock-require' );

const libui = require( './mock/libui' );

chai.use( sinonChai );

afterEach( () => {
  sinon.restore();
} );

mock( 'libui-node', libui );
mock( 'libui-node-dom', '../packages/libui-node-dom' );

require( './spec/nodes' );
require( './spec/elements' );
require( './spec/widgets' );
require( './spec/window' );

test/mock/libui.js:

const libui = {};

libui.UiWindow = class {
  constructor( title, width, height, menu ) {
    this.title = title;
    this.width = width;
    this.height = height;
    this.menu = menu;
    this.margined = false;
    this.fullscreen = false;
    this.borderless = false;
  }

  setChild( child ) {
  }

  show() {
  }

  close() {
  }
};

libui.UiControl = class {
  constructor() {
    this.visible = true;
    this.enabled = true;
  }
};

libui.UiBox = class extends libui.UiControl {
};

libui.UiText = class extends libui.UiControl {
};

libui.UiButton = class extends libui.UiControl {
  constructor() {
    super();
    this.text = '';
  }

  onClicked( handler ) {
  }
};

module.exports = libui;

[kusti8]

Yeah that's what I was looking at. I tried Jest and using its automatic mocking feature, but it crashes for some reason. Something to do with not liking C++ I think. So I was working on a mocked implementation. That implementation above is pretty barebones.

There's also the possibility of not mocking, but spying on selection functions. You would have to define it differently beforehand for each test depending on what you're testing, and also need Xvfb. I learning towards mocking everything, but it's going to get annoying.

[mischnic]

The classes of the libui-node NAPI port (probably going to be released as version 0.4.0 - should be 100% backwards-compatible) are declared in javascript as opposed to C++. That might work better with automatic mocking? Example: https://github.com/parro-it/libui-napi/blob/master/js/combobox.js "Port" of proton-native: https://github.com/mischnic/proton-native/tree/napi

[mimecorg]

I agree that mocking everything manually is not a great solution, but from my experience so far it's much less work than writing the actual unit tests, so it's not that annoying. I didn't spend too much time trying to automate this.

I'm only creating a constructor for each class that I need and some empty methods, just enough so that the tests can run without any errors. I'm also using a sinon spy when I want to test if a method is called correctly. The only exception is appending/removing children from a container, where the mock methods manipulate a real array of children which can be easily compared with the expected result.

[kusti8]

The NAPI port works fine. My main concern about manually mocking isn't about the initial work putting into it, but the continued maintenance, which I don't really want to deal with. This has less room for errors.

import React, { Component } from 'react';
import { render, App, Window } from '../src/';
import libui from 'libui-napi';
jest.mock('libui-napi')

describe('Window and loop', () => {
  afterEach(() => {
    jest.resetAllMocks();
  });
  test('All defaults', () => {
    class Test extends Component {
      render() {
        return (
          <App>
            <Window />
          </App>
        );
      }
    }
    render(<Test />);
    expect(libui.startLoop).toHaveBeenCalled();
    expect(libui.UiWindow).toHaveBeenCalledWith('', 500, 500, true);
  });
});

[maciek134]

You could use docker with X.org like Selenium does and not have to worry about Xvfb. Could you elaborate on "spying on selector functions"? Because my only idea was to use GTK Inspector and its logs to do assertions.

[mischnic]

You could use docker with X.org like Selenium does and not have to worry about Xvfb.

In my experience, the biggest problem with screenshot-based testing is maintaining the "reference images". If you change/add a test, every tested platform needs a new screenshot (and we also had issues with date pickers and varying date formats on different machines).

spying on selection functions

I'm guessing logging calls to libui-node but still executing them.

[maciek134]

GTK Inspector is not for screenshot/reftests, it attaches to the app and can read the widget structure, styles, etc.

[kusti8]

With jest, there is a spyOn feature that basically wraps functions with mocks, but allows the function to still run normally. Usage would be defining the functions you want to spy on before the fact, then running the test, and then checking them at the end. It's better than creating new mocked functions, but I still don't like the fact that I have to specify all of the functions to spy on at the beginning.

I don't really like screenshot based testing because it is inherently more unstable. I'm not testing libui or libui-node. I'm just testing my wrapper around that, so I just need to worry about my logic.

[maciek134]

Oh, you meant using spyOn, I was confused about the "selector functions". I'll repeat that GTK Inspector is not a screenshot tool, but you are right, there is no need to test GTK at all.

Is a switch to libui-napi a thing you want?

Is this something you want to do or can I do it?

[parro-it]

Is a switch to libui-napi a thing you want?

libui-napi is a reimplementation of libui-node that uses the new N-API node libraries. It's code will completely replace libui-node when we'll publish next release. No need to switch.

[kusti8]

I decided to just mock all of libui, so I published tests and added some tests for Window, which seem to work.