Proposal: Type Checking

[johnspackman]

Type Safety

Next year we (the dev team at Zenesis) would like to start thinking about adding some type safety features into the Qooxdoo compiler - the key principal is that JSDoc in existing Qooxdoo apps provide a lot of core information about types which can be read by the compiler in order to inject validation code into function prologue and epilog to catch bugs earlier (during "source" builds at least).  If we took this a little bit further, there will be a number of times when the compiler would able to detect bugs at compile time, and not just at runtime.

In the State of JS survey 2022, the top missing feature was static type checking: https://2022.stateofjs.com/en-US/opinions/#top_currently_missing_from_js

Lately, I've taken to documenting private variables with the @type jsdoc, which would allow the compiler to do the same for privates as well as methods; and a similar approach could be used for variables inside a function.

For example given this code:

construct() {
  super();
  this.__map = {};
},

/** @type{Object<String,mypkg.SomeClass>} my map of things */
__map: null,

/**
 * Sets a value, returns the number of keys that have been set
 *
 * @param {String} key the name of the value to set
 * @param {mypkg.SomeClass} value the value to set
 * @return {Integer}
 */
setSomething(key, value) {
  /** @type{Object<String,mypkg.SomeClass>} */
  var map = this.__map;
  map.set(key, value);
  return Object.keys(map).length;
}

In that example, when the compiler outputs transpiled code it could include something like this:

/** @type{Object<String,mypkg.SomeClass>} my map of things */
__map: null,

/**
 * Sets a value, returns the number of keys that have been set
 *
 * @param {String} key the name of the value to set
 * @param {mypkg.SomeClass} value the value to set
 * @return {Integer}
 */
setSomething(key, value) {
  qx.core.Assert.assertTrue(typeof key == "string");
  qx.core.Assert.assertTrue(value instanceof mypkg.SomeClass);

  /** @type{Object<String,mypkg.SomeClass>} */
  qx.core.TypeCheck.assertMapValues(this.__map, mypkg.SomeClass);
  var map = this.__map;
  map.set(key, value);

  var tmp = Object.keys(map).length;
  qx.core.Assert.assertTrue(qx.lang.Type.isInteger(tmp));
  return tmp;
}

Obviously that's a bit of a contrived and simplistic example 🙂  but the principal is that if we write code that declares what a member variable, local variable, or function parameter/return should be, then the compiler can inject code to check (or enforce) it.

Challenges to overcome

Inconsistency of usage

There are a few things that need to be addresses before this is possible - firstly, the use of JSDoc is inconsistent, both in that Qooxdoo has historically used a version which is not quite JSDoc compatible, and there are inconsistencies within the framework itself.

The most common example is how @param are declares, that Qooxdoo JSDoc uses @param variableName {TypeName} description whereas JSDoc uses @param {TypeName} variableName description

Different / outdated type names

There is also some variance in type names, for example Map is sometimes used in Qooxdoo to represent a map of key/value pairs stored in an ordinary POJO {} object, and not an ES6 Map instance.

What the new type name should be is open to debate - TypeScript defines Record<K,V> for example, and states that Object as a type does not support parameterisation.

null, undefined, and default values

We often define variables in JSDoc as optional with `?`, or optional with an implied default value following - eg {String?} or {String?null} - while that expression allows the default, it does not allow null to be required as an explicit value, eg it would be stricter to write {String|null} which means that undefined is not allowed

Richer type expressions

Type expressions need to be a suitably rich expression in a string which can be parsed to provide lots of checking - for example, this would include parameterised types / generic types and this would require changes to classes like qx.data.Array

Hopefully this would be little more than just adding parameterised types (eg qx.data.Array<String>), but these can become more complex fairly quickly:

• {qx.data.Array<String|Integer>?}
• {qx.data.Array<String>|qx.data.Array<Integer>?}
• {(qx.data.Array<String>|qx.data.Array<Integer>)?}
• {qx.data.Array<String>|qx.data.Array<Integer>|null}

Excessive Warnings (at least initially)

There could be a lot of warnings to overcome, so being able to define what is a warning or an error (or just ignored) in the Manifest.json, on a per class basis would be really important.

Could we (should we) switch to TSDoc?

Tooling support (especially in IDEs) for TypeScript Documentation comments (ie TSDoc) is very good and it would seem that support for TSDoc will continue to be good.  JSDoc and TSDoc are quite similar, but have a few significant differences (eg how parameters are defined).

TSDoc is a subset of JSDoc, and has a narrower definition focused on types which is well-tested

TSDoc site url: https://tsdoc.org/

JSDoc cannot inform tooling (IDEs) of sufficient type information

JSDoc does not support the required information for IDEs because we cannot define inheritance - the compiler adds this in, but there are probably areas where it's not good enough

Pros and Cons

TSDoc will (presumably) have thought of all the variations needed for modern types, but it could also add expressions which we are not able to support.

Maximising IDE support

Consistency with Typescript will make the most of the existing IDE support; VSCode is very popular for Javascript development and it's Typescript support is first class, so there is a huge benefit in being as compatible as possible with it.

Popularity

State Of Js shows that Typescript is more popular that native Javascript: https://2022.stateofjs.com/en-US/usage/

Mapping JSDoc types to *.d.ts files

We currently output Typescript definition files (*.d.ts files) so that Typescript IDEs can understand our type system - but this is not complete, partly because we would need to map from some types or expressions (eg "*" or Object<>) to work properly.

Reduce Debate

Another nice benefit could be that settling on a strict type system like TSDoc could reduce debate and speed up the development cycle - similar to how PrettierJS is not always to our individual tastes, but the lack of configuration options remove debate.

Adding actual TypeScript syntax in code

It may make sense to experiment with providing first-class support for TypeScript syntax in the source files; this would allow type definitions to be included in the code rather than in JSDoc/TSDoc comments.

There is already a plugin for babel which apparently supports Typescript syntax.

Return types of async functions

A while ago there was some discussion about whether async functions should be documented as returning a Promise or not.  IMHO the async keyword is sufficient for the documentation to be generated and that is what was used most frequently in the code.  It was agreed that with a one-line change to the API Viewer to show that the function is async, then an explicit Promise return value is not necessary.

This is not consistent with other type systems and both Dart and TypeScript expect an async function to be documented as returning Promise<Type> or Future<Type>.

Automatic updates

A tool similar to qx es6ify could be built to update JSDoc comments in existing source code, minimising a lot of the repetitive work of correcting the docs.

[oetiker]

Going TS would be a very good move indeed. It also aligns with the type of developer I see as attracted to qx. So yes, all for it.

I think a very important part will be to make it very simple to upgrade existing projects. Most of us are maintaining huge code bases.

[tcsaddul]

TypeScript and QooxDoo are great together in Visual Studio. Implementing strict "types" really helps a lot especially in managing large applications. We have been using qx-typed: https://github.com/jbaron/qx-typed since 2014. @jbaron just updated it for the QooxDoo version 7.5.0 in January 2023 and still works today. Maybe his work can help.

[jbaron]

For those interested, there are actual 4 public hosted projects:

1. qx-parser - creates the qooxdoo.d.ts declaration file based on the jsdoc json API files
2. qx-generator - generates a single bundled JavaScript file
3. qx-typed - a sample project that demonstrates how to develop TypeScript qooxdoo applications using the JavaScript and TS declaration file from the above two projects

My findings are that qooxdoo is mostly compliant with TS (and OO principles in general), but there are some exceptions. For example subclasses sometimes override methods with a wrong signature. So moving to TS will require some incompatible changes (but in general for the better).

Overall I think that using TS as a basis would be a nice path to explore and possible some of this could be generated.

[cboulanger]

I am also definitively in favor of adopting a tried and tested standard even if this mean breaking compatibility. TS is awesome, and we we can outsource a whole lot of conceptual problems which are not central to what qooxdoo does.

[goldim]

To be honest I don't trust Microsoft bc of their "Embrace, extend and distinguish" strategy I am too cautious. I would like see type checking maybe with typescript but it has to be not forceable and optional.