Add a TypeScript declaration generator for a typed-function?

[gwhitney]

This is an idea that came up in the discussion for https://github.com/josdejong/mathjs/pull/2448. Since in some sense, typed-function is a library that adds runtime type-checking and overloading to JavaScript that has a similar flavor to TypeScript's compile-time type-checking and overloading, it would be nice if there were a way to generate a TypeScript declaration for a typed-function that would compile (as close as possible to) exactly when the actual JavaScript call generated will not throw a TypeError. Because the type conversion and matching rules are fairly complex, I think this could only be done as a facility provided by this module, since it sees/knows everything that is going on with an individual function.

One potentially significant stumbling block is to get the sort of type inference that's actually of value to TypeScript users, the declaration should have some information about what the return types of functions are. So I think this would really only be doable in a useful way if typed-function added return-type annotations. So I will add an issue for that possibility as well.

[josdejong]

Thanks, that would be a good idea. It would be nicest if the types could be defined "on the fly" using generics, but I expect that to be too complex. Generating the type definitions statically would be a good alternative solution.

Anyone interested in doing an experiment with this?

[cshaa]

Here's a quick proof of concept!

[gwhitney]

Here's a quick proof of concept!

Well, that's pretty awesome. Thanks! I see that the type BaseTypes is defined at the top. I presume, though, that TypeScript types are not extensible? In other words, there isn't a way to update the BaseTypes type as someone calls typed.addType()? If we knew a static collection of all the types one ever wanted math.js, say, to deal with, then I would see no obstruction to simply rewriting it entirely in TypeScript from the ground up. It seems to me precisely that the ability to add types on the fly is what makes math.js/typed-function and TypeScript hard to reconcile. But I would be happy to be proved wrong. Until that happens, I despair of this automatically-from-the-TypeScript-compiler generation of .d.ts, and fear that a manual string-manipulating .d.ts generator from a typed-function instance is all we can hope for... But yes, I hope I am wrong.

[cshaa]

TypeScript does not allow types to be dynamically modified: if you, for example, define a type as string, you cannot retroactively redefine it to string | number elsewhere in the code. Such code would be impossible to statically analyze and much harder to reason about, and I don't think there is any type checker that would let you do this.

What you can do, however, is to have a typedFactory, to which you pass all type definitions and it gives you a typed function that correctly recognizes all of them. In the context of math.js, this would mean that each bundle would have its separate typed function which only recognizes the types present in such bundle.

You are right that “the ability to add types on the fly is what makes math.js/typed-function and TypeScript hard to reconcile”, because mutating the type of something is generally just a terrible idea and I'm not aware of any system that would let you do this. However, with typedFactory, no type is mutated – instead a new function with a new type is created.

Example usage of typedFactory here!

The part with the (x): x is something => x instanceof Something might be confusing for a non-TS user, so I'll quickly explain it. The TS syntax for an arrow function return type is (x): returnType => implementation(x). If you have a function which returns a boolean, you may declare that it's a so-called type guard using the syntax x is foo as the return type. If you do this, you tell TS that you guarantee the function will only return true if its argument is of type foo.

Therefore, the argument of typedFactory is an object which maps type names to their corresponding type guards. This is useful both for TS, as it can extract the necessary type-name pairs to use for static analysis, but also at runtime, as you can use these type guards to check what type something is.

[gwhitney]

Well that is excellent, in the sense that it looks really promising for writing a typescript declaration generator for mathjs more or less as it stands so we can avoid the drain of hand-maintenance of the index.d.ts. We could just run it twice, once for the main bundle and once for the number-only version. It still hasn't quite dawned on me yet how something like a "math.ts" would work, though, so pursuing a full rewrite of mathjs in TypeScript isn't really a priority for me. (I'm not in principle against such a rewrite, though, and am willing to help out if someone cleverer comes up with an architecture for it.)

[gwhitney]

(In particular based on your lovely proof of concept, my thinking is that we could write a little typescript program that loads in a math.js bundle holding its nose about its types, and then for each typed-function object, it extracts the map from string signatures to all of the (direct and converted) implementations of that typed-function, and passes that map to the typed-factory, producing a bona-fide correctly-typed TypeScript function but more importantly generates a .d.ts with the correct declarations which can then just be installed as the index.d.ts for the library, so that math.js can then be used from TypeScript without going through that "translation" process.)

Edit: or rather, upon some reflection, a javascript program that loads a mathjs instance, and quite trivially writes out a typescript program which defines the proper object assembling all the types and conversions, calls the typedFactory, and generates a proper index.d.ts for the mathjs instance. A bit more roundabout but I actually think this would work and cut through the treadmill of updating a manually-written index.d.ts. Seems well worth putting effort into.

[josdejong]

@m93a 😎 this POC in https://github.com/josdejong/typed-function/issues/123#issuecomment-1207205021 looks really promising! I cannot think of a real show stopper right now. We will need to change the API though, and make it immutable again: have a typedFactory where you pass all types and conversions, instead of having methods addType and addConversion, maybe some difficulties will arise when working out this experiment.

so we can avoid the drain of hand-maintenance of the index.d.ts. We could just run it twice, once for the main bundle and once for the number-only version.

@gwhitney I'm not sure what you mean: I think this smart, generic TypeScript definition will replace the manual written out type definitions. I guess this solution will not have a spelled out type definition for each individual function anymore, that is derived by TypeScript based on the generic type definitions.

[gwhitney]

so we can avoid the drain of hand-maintenance of the index.d.ts. We could just run it twice, once for the main bundle and once for the number-only version.

@gwhitney I'm not sure what you mean: I think this smart, generic TypeScript definition will replace the manual written out type definitions. I guess this solution will not have a spelled out type definition for each individual function anymore, that is derived by TypeScript based on the generic type definitions.

Well, at the moment a typedFactory requires a fixed collection of types (in other words, all of the keys of the argument object to the typedFactory() function have to be known at compile time). So m93a's lovely proof of concept, as far as I can see, is not consistent with any version of typed-function in which you can add types at runtime. I think that's a feature that's definitely wanted. If so, then what you can do is in javascript, load all of mathjs with all the types you want (either the whole shebang as in the current main mathjs bundle, or just the number stuff as in the number bundle) and then have a utility program in javascript that writes out a little dummy TypeScript program that declares a static list all of the types actually used (and their conversions, which the proof of concept doesn't actually have yet) and then calls the resulting typedFactory for each of the mathjs operations. Then the build process compiles each of those dummy TypeScript programs created, producing .d.ts files which are then published with the actual javascript mathjs. Thereby, we do not have to undertake the daunting task of directly writing a typed-function -> TypeScript .d.ts translator, which seems like a lot more work than leveraging the TypeScript compiler.

Does this explanation clarify my comment?

[josdejong]

Hm, yes, I understand what you mean. Maybe we have to look a bit different to the problem: instead of dynamically importing new data types and functions into an existing mathjs instance (which is not possible to support by TypeScript if I understand it correctly), it may be possible to create a new mathjs instance from scratch with all the datatypes+functions that you want?

I've done some fiddling around with the second example of @m93a , trying to mold it into this model of the user importing the datatypes and functions that he/she wants (trying to grow it a bit towards the pocomath concept):

Dynamically construct an instance with data types and function signatures that you want

EDIT: updated the POC a bit, we're not there yet

[gwhitney]

Well, that is a lot more like Pocomath, that's true. Nice! Seems like a reasonably promising path. The main concern I see at the moment is that it seems to require (in much the same way as mathjs is organized now) that for each operator there is one place where every implementation is listed. One of the main ideas of Pocomath was trying to get away from that, as (pardon my saying so) it really breaks modularity and independence of different types.

The problem is this is getting beyond my experience/knowledge/comfort level with Typescript -- would there be a way to collect up the different implementations of 'add' from different imports, and then when one is done with all the imports, cook up the final "typed-function" version of 'add'? I don't really mind if you need to have a file for a given bundle that lists all of the types in that bundle -- since after all you have to import the modules for each type anyway, so they already need to be listed somewhere -- but having to reiterate all of the type-variants of each operator would be very verbose and redundant and would create otherwise unnecessary dependencies among different parts of the code. It seems like it would make custom bundling really prohibitively difficult, as you would have to track down for every single operator the exact list of implementations that you need/want.

Anyhow, I guess I will have to be mostly on the sidelines and cheer on the effort to find a TypeScript-y approach to typed-function.

[josdejong]

It's far from there, but I would like to see how far we can get. I'm not sure if it's possible to make it fully work with TypeScript. I'm not a Typescript expert either, but I'll do some more fiddling and see what I can learn 😁. @m93a more help would be welcome.

Anyhow, I guess I will have to be mostly on the sidelines and cheer on the effort to find a TypeScript-y approach to typed-function.

no problem at all👍

[cshaa]

Hey Jos! I've been a little overwhelmed by the length of typed-function's code (which was the main reason I never got around to completing #89), so I decided to give the TS direction a fresh start. In m93a/over.ts I've implemented the core functionality. My original plan was to have it published on NPM with a complete documentation by now, but then life got in the way. During the coming week I'll try to finish the documentation and add a merge function which would take multiple overloaded functions and combine them, in a similar way you described here: https://github.com/josdejong/typed-function/issues/123#issuecomment-1218156677

[gwhitney]

I love the brevity and focus of "over.ts". I am commenting simply to register my vote that to build a full-fledged math.ts (that will be as comfortable/powerful/usable for folks like me coming from the "math world" side of things) on top of something like over.ts,

• definable automatic type conversions will be a must, and
• I think the Pocomath prototype shows that parametrized implementations and types (like min<T> with signature T,T -> T or Complex<T> for complex numbers with component type T) are extremely valuable; I was getting set to do a PR to typed-function to add these things. Perhaps these things somehow come for free because of TypeScript? Or maybe the parser part would have to be extended to support them? Hard for me to tell at my level of familiarity with TypeScript.
• And in a similar vein, it would be very handy to be able to declare that one type is strictly a subtype of another, e.g. NumInt being the type of entities that are JavaScript built-in number instances and which happen to be integers. Same questions here, as to whether this somehow comes for free in the over.ts approach, or would need additional infrastructure. (To clarify, the point of the overloader knowing that NumInt is a subtype of number is that if a specific overloaded function has a number implementation but no NumInt implementation, it dispatches NumInt arguments to the number implementation, whereas if it has both it dispatches NumInts to the NumInt implementation and numbers that are not NumInts to the number implementation. This is like an automatic conversion with the identity conversion function, except that signature matches using only subtype matching are considered exact matches rather than conversion matches, which has consequences in terms of dispatch priority and referencing one implementation from another.)
• And that reminds me: I haven't looked at the dispatch algorithm in over.ts, but some of the length of typed-function.js comes from (a) properly prioritizing signatures that overlap in their applicability to a given actual argument list, and (b) allowing implementations to refer to each other without the cost of a full re-dispatch. So I suspect that to get a math.ts fully working and running efficiently, some of these aspects may need to be replicated in something like over.ts if they are not already there.

Anyhow, I have gotten the impression that at least the automatic type conversions are not of too high a priority for you @m93a, but I hope that once over.ts has reached a "releasable" level you will be open to PRs pursuing at least some of these directions? Thanks so much for illuminating a possible way for math.js to become math.ts!

[josdejong]

Thanks @m93a , take it easy :).

I see with over.ts you are working out the earlier POC you posted in this issue. I tried to bring this POC a couple of steps further myself, but I didn't get very far yet. m93a/over.ts looks promising, though I still see some bears on the road 🐻

Some thoughts:

1. Do you see a possibility to merge multiple objects with signatures or multiple typed-functions together, so we can compose functions?
2. like Glen points out: it would be great to support automatic conversions (i.e. if you have signatures add(number, number) and add(Complex, Complex), and a conversion from number to Complex, then you can support mixed use too: add(number, Complex) and add(Complex, number)). It could possibly be implemented differently in some two-step process maybe.

[gwhitney]

The Pocomath prototype referenced above now supprts return-type annotations as well, which at least conceptually brings it closer to TypeScript. Given my TypeScript skills, I remain unclear on whether there's a fully TypeScript approach to support the kinds of operations Jos highlights in his last message, but if there's a way to build a math.ts along the lines of how Pocomath is organized, that would be great; or if still uses javascript internally for some of the dynamic merging purposes but is organized in a more typescript-friendly fashion and typescript can be used to automatically generate the index.d.ts files for the package, I would say that would by itself be a big win. Looking forward to the further developments/suggestions by the TypeScript gurus here.

[gwhitney]

Oh, and I took a quick look at the dispatch among the different implementations in over.ts, and as far as I can tell, it just tries all of the implementations in the order they are supplied and executes the first one that matches. As suspected, that's very different from typed-function, which carefully orders the supplied implementations for such purposes as preferring exact type matches over "any" parameters, preventing a "rest" parameter from eating arguments that could match individual parameters of another implementation, etc. And I think that difference is a big part of the brevity of over.ts as compared to typed-function. But I also think that not relying on the order the implementations are listed is very important to the modularity desire (as discussed in https://github.com/josdejong/mathjs/issues/1975) -- if the implementations for a given operation are going to be split up into multiple source files, say to deal with all of the operations for one type in one place, and for another type in another, then it seems like it becomes difficult to rely on their being ordered a priori in any helpful/sensible fashion.

Note these are not criticisms of the over.ts approach -- as I've said if anyone more expert in TypeScript can find a way to support the key needs of a potential math.ts, I'm all for it. Just trying to help in delineating what those key needs are. I hope these comments are constructive.

[josdejong]

👍

[gwhitney]

Apologies for a long comment. This one has the details of what I did, and the next one will have my conclusions. I decided to take a break from Pocomath (benchmarking) since there seems to be a strong desire to pursue TypeScript conversion before a major overhaul to math.ts. So I got off the sidelines and tried to move that ball forward.

The biggest obstacle at the moment seemed to me to be that over.ts seems to require all of the implementations for a given operation in one place, but efforts like https://github.com/josdejong/mathjs/discussions/2741 (also very constructively initiated by @m93a) and Pocomath that grew out of that discussion push toward different implementations being in different source files, to make mathjs easier to extend with new types, etc. So it seemed to me that the top priority was resolving that tension.

The key need seemed to be a way to build up the argument to the generated typed factory incrementally so that it remains as narrowly typed as if the whole thing had been typed as a single literal. And indeed, there are articles about the "builder pattern" for creating objects one property at a time.

So I tried to emulate those examples to create a plain object whose keys are signature descriptions and whose values are implementing functions, and came up with the following (the part about checking the key is based on this other stackexchange answer):

import { useTypes } from 'over.ts/src/index.js';

const types = {
   number: (x: unknown): x is number => typeof x === 'number',
   bigint: (x: unknown): x is bigint => typeof x === 'bigint'
}

const overload = useTypes(types)

type TypeName = keyof typeof types
type CheckTuple<T> = T extends TypeName
  ? T
  : T extends `${TypeName}, ${infer Rest}`
  ? CheckTuple<Rest> extends string ? T : false
  : false
type CheckDecl<T> = T extends `${infer Args} -> ${TypeName}`
  ? CheckTuple<Args> extends string ? T : never
  : never

class ImpBuilder {
  addImp<T extends string, U>(decl: CheckDecl<T>, imp: U) {
    return Object.assign(this, {[decl]: imp} as Record<T, U>)
  }
  finalize() {
    delete this.addImp
    delete this.finalize
    return this
  }
}

const negateImps = new ImpBuilder()
  .addImp('number -> number', (a: number) => -a)
  .addImp('bigint -> bigint', (a: bigint) => -a)

const negate = overload(negateImps.finalize() as Omit<typeof negateImps, 'finalize'|'addImp'>)

console.log('Negation of 5 is', negate(5))
console.log('Negation of 5n is', negate(5n))

And lo and behold, this does work and produces just the overloaded negate I wanted. (And yes admittedly the step of removing the addImp and finalize keys from the result of the builder is a big blemish but one I imagine could be fixed without too much difficulty; I just wanted to get something working.)

But what we ultimately want is that add say is defined in two different files, say one for numbers and one for bigints. So I modeled this with:

const addImps = new ImpBuilder() // imagining this is in some "initialize a proto-bundle" file
addImps.addImp('number, number -> number', (a: number, b: number) => a+b) // imagining this line in the `number` file
addImps.addImp('bigint, bigint -> bigint', (a: bigint, b: bigint) => a+b) // and this is in the `bigint` file
const add = overload(addImps.finalize() as Omit<typeof addImps, 'finalize'|'addImp'>) // We wrap everything up elsewhere

console.log('Sum of 5 and 7 is', add(5,7))
console.log('Sum of 5n and 7n is', add(5n, 7n))

try {
    //@ts-expect-error
    console.log('Mixed sum is', add(5n, 7))
} catch {
    console.log('Mixed sum errored as expected.')
}

But this now errors on compilation:

src/steps/two.ts:72:34 - error TS2349: This expression is not callable.
  Type '{}' has no call signatures.

72 console.log('Sum of 5 and 7 is', add(5,7))
                                    ~~~

(The file name is because this sort of "divvying up" of specifying the implementations of an over.ts overload was step two of a plan I laid out for getting over.ts close enough to Pocomath to build math.ts on top of it.)

So huh, all that nice type information has been lost in dividing the specification of addImps into multiple statements (I did verify that if I put them all onto one line like with negate, it does work, so it's not that this pattern has some problem with the binary functions.)

So then I thought that maybe the difference is in the negate case my negateImps value was the return from the final invocation of .addImp(), so I tried changing the last bits of the construction to:

const addAll = addImps.addImp('bigint, bigint -> bigint', (a: bigint, b: bigint) => a+b) // and this is in the `bigint` file
const add = overload(addAll.finalize() as Omit<typeof addAll, 'finalize'|'addImp'>) // We wrap everything up elsewhere

leaving everything else the same, but all this did is change the error to:

src/steps/two.ts:72:38 - error TS2345: Argument of type 'number' is not assignable to parameter of type 'bigint'.

72 console.log('Sum of 5 and 7 is', add(5,7))
                                        ~

or in other words, the only implementation visible to the TypeScript type inference in the call to overload is the bigint one on the line where addAll is generated.

[gwhitney]

So based on the experiments in the previous comment, I conclude that either

(a) I have missed some key aspect of the "builder pattern" in TypeScript in my effort to adapt it to this situation, thereby ruining its ability to be split up into multiple statements; or (b) it really isn't possible to split up the generation of a "literal-ish" object with its detailed literal type into multiple files gathered by imports.

Since in all of the examples of the builder pattern I found in various articles, they always chain all of the uses of their analogues of the .addImp method rather than call them on different statements, I am forming a relatively strong hunch that (b) is the case (but of course would be happy to be proved wrong).

To me, given that we want to reorganize math.js/ts so that it's more approachable to add new types (or for clients to extend it with their types), (b) would be a real showstopper.

And then as I was working on this, another consideration occurred to me: these type specifications to allow TypeScript to generate the properly narrow type information for the operations of math.ts are very intricate and pose considerable difficulties both to write and to read. Therefore, they would seem to pose a real maintenance risk. Why would we want to put the project and its maintainers through that when there seems to be a readily available alternative without much drawback:

• Implement a solid, flexible runtime type specification (that can be as close to TypeScript as we want) in JavaScript (or in TypeScript where we just don't try to strongly type the "sausage-making" of the overloads, but just use it to keep cleanliness in strings and numbers and so on -- whatever is straightforward to type). I'm not wedded to its details, but I think Pocomath shows a roadmap for how this is entirely feasible, and
• For any bundle that is built, have a utility that collects all the types and loops over all of the operations emitting an over.ts-style specification for each operation with all of the implementations in one place, and use that (only) to generate a guaranteed-correct .d.ts file that we publish with the bundle, allowing it to be used from TypeScript exactly as desired and without any hassle of manually maintaining TypeScript declarations.

So that's my vote. So I will go back to the benchmarking bit, and presuming that goes fine (I will report in the appropriate discussion on mathjs) I will then just await the determination by "the mathjs council" -- Jos and whomever's opinions he'd like -- as to the direction here: something like my proposal, or a full-blown TypeScript architecture that someone more adept devises, or something else. And in any case, I will of course be happy to contribute my full effort to whatever re-org is settled on.

[josdejong]

Looking into TypeScript support is indeed a high prio to me, thanks Glen for giving this a try! We'll have to see if and how far we can get.

I did some fiddling with your code examples and indeed it seems like this builder only works when having a single chain that is not broken in multiple lines.

How about the following approach, first collecting all objects with signatures into a new object, then merge them, and then pass the merged object to overload:

import { useTypes } from 'over.ts/src/index.js'

const types = {
  number: (x: unknown): x is number => typeof x === 'number',
  bigint: (x: unknown): x is bigint => typeof x === 'bigint'
}

const overload = useTypes(types)

// coming from one file
const negateNumber = {
  'number -> number': (a: number) => -a
}

// coming from another file
const negateBigint = {
  'bigint -> bigint': (a: bigint) => -a
}

// the user merges all signatures that he's interested in
const negateAll = {
  ...negateNumber,
  ...negateBigint
}

const negate = overload(negateAll)

console.log('Negation of 5 is', negate(5))
console.log('Negation of 5n is', negate(5n))

[josdejong]

Here an experiment trying to model the pocomath approach of having a map with function names and for each function name have an object with one or multiple signatures:

import { mergeWith, mapValues } from 'lodash-es'
import { useTypes } from './assets/over.ts/src/index.js'

const types = {
  number: (x: unknown): x is number => typeof x === 'number',
  bigint: (x: unknown): x is bigint => typeof x === 'bigint'
}

const overload = useTypes(types)

// coming from one file
const fnsNumber = {
  negate: {
    'number -> number': (a: number) => -a
  },
  add: {
    'number, number -> number': (a: number, b: number) => a + b
  }
}

// coming from another file
const fnsBigint = {
  negate: {
    'bigint -> bigint': (a: bigint) => -a
  },
  add: {
    'bigint, bigint -> bigint': (a: bigint, b: bigint) => a + b
  }
}

const fnsBigint2 = {
  subtract: {
    'bigint, bigint -> bigint': (a: bigint, b: bigint) => a - b
  }
}

const fnsBigint3 = {
  multiply: {
    'bigint, bigint -> bigint': (a: bigint, b: bigint) => a * b
  }
}

const fnsBigint4 = {
  divide: {
    'bigint, bigint -> bigint': (a: bigint, b: bigint) => a / b
  }
}

// the user merges all signatures that he's interested in
// PROBLEM: types only work with up to 5 objects, with more objects it becomes any
const fnsAll = mergeWith({}, fnsNumber, fnsBigint, fnsBigint2, fnsBigint3, (a, b) => {
  return { ...a, ...b }
})
const fns = mapValues(fnsAll, overload)

console.log('Negation of 5 is', fns.negate(5))
console.log('Negation of 5n is', fns.negate(5n))
console.log('Adding 5 and 2 gives', fns.add(5, 2))
console.log('Adding 5n and 2n gives', fns.add(5n, 2n))

This works, but we're not yet there: it doesn't work for an arbitrary number of source objects

[gwhitney]

Well, suppose you have eight files you want to merge the implementations from. Can you do something like

import { mergeWith } from 'lodash-es'
import fns1 from '../imps1.ts'
import fns2 from '../imps2.ts'
import fns3 from '../imps3.ts'
import fns4 from '../imps4.ts'
import fns5 from '../imps5.ts'
import fns6 from '../imps6.ts'
import fns7 from '../imps7.ts'
import fns8 from '../imps8.ts'

const simpleMerger = (a, b) => ({...a, ...b})

const firstFns = mergeWith({}, fns1, fns2, fns3, fns4, simpleMerger)
const secondFns = mergeWith({}, fns5, fns6, fns7, fns8, simpleMerger)
const fnsAll = mergeWith({}, firstFns, secondFns, simpleMerger)

A potentially more important question: is this how you want the source code for math.ts to look? I mean, imps1 through imps8 etc might really be something like arithmetic, algebra, logical, relational, etc. (and then within each of those might be lists of individual operations), and these files that assemble implementations would consist of a whole bunch of imports, followed by several rows of mergeWith called on all of the imports? As opposed to just the imports as Pocomath has now? I guess maybe the level of redundancy/cumbersomeness is not so high as to outweigh the perceived advantages of having math.ts implemented entirely in TypeScript?

So anyhow, let me know if the "multi-level" mergeWith works and if so I will continue my sequence of steps to get enough of the Pocomath style working in pure TypeScript to make a math.ts feasible; the next up, I think, would be getting the dependencies between implementations working in a jazzed-up version of over.ts.

[josdejong]

Yes, technically it would be solvable like that, though it doesn't feel like neat solution. Ideally, something like the Builder pattern you where playing with would be perfect. Some recursive approach adding a new item to all the previous items. But as far as I understand, we can't use anything dynamic like for loops, then TypeScript loses track.

At this point I'm still not sure if we can create a 100% solution in TypeScript. There is also for example the conversions that typed-function can do, and it will probably be hard to express all of that in a statically typed way. On the other hand, we're already further than I thought was possible, so, who knows. I keep playing around.

I'm also thinking into a direction of having some hybrid solution. One option is to create pocomath+typed-function not or only partially in TypeScript, and has some build script which generates full TypeScript declarations from the typed-functions.

[gwhitney]

Trying to push the 100% TypeScript solution as far as possible to see if we can either make it work or get it to definitively break, it seems as though at the moment we are caught on a sort of syntax issue when lots of "operator name: implementations" objects need to be merged. My last proposal was "multi-level mergeWith()" and I don't think either Jos or I really liked that. What about if we adopted something like:

import merge from './ourSpecialMergeImplementations.ts' // implementation TBD
import fns1 from '../imps1.ts'
import fns2 from '../imps2.ts'
import fns3 from '../imps3.ts'
import fns4 from '../imps4.ts'
import fns5 from '../imps5.ts'
import fns6 from '../imps6.ts'
import fns7 from '../imps7.ts'
import fns8 from '../imps8.ts'

const fnsAll = merge(fns1).with(fns2).with(fns3).with(fns4).with(fns5).with(fns6).with(fns7).with(fns8).implementations()
export default fnsAll // Or build the math instance from it or whatever.

Does that "look nice enough" to try moving forward with? It seems to me that this syntax should allow a type-exact implementation using techniques along the lines of the ImpBuilder above, because here we are deliberately stringing all of the subcollections of functions we want to merge into a single chained call, since there will of course have to be a single file that imports all of the subcollections, and this allows the individual implementations of 'add' to be in different files and be independent of each other. And in that single file that implements the subcollections, there's no problem with making one long merge chain like this.

Anyhow, let me know your thoughts on this organization for collecting up implementations and if it looks OK with you, I will see if I can make something like this work and actually move the ball another meter or so toward the goal.

[gwhitney]

Ok, I got the merge(fns1).with(fns2).imps() syntax working in https://code.studioinfinity.org/glen/typomath -- look in src/steps/five.ts (roadmap for the steps is in the top-level README). While I was at it, I used the typescript-rtti module to determine the types of the implementations at runtime so they can be selected based on the actual argument, to avoid having to rewrite the signature of each implementation (after all, TypeScript figures it out at compile time, so why should we have to redundantly have to write it out in the string key, too?)

So that seems good as long as you are comfortable with that merging syntax.

But then the next item up is dependencies. Now these are less crucial in TypeScript because we have already said in a math.ts there will be no dynamic adding of implementations -- the whole bundle must have all its implementations specified before overloading all of its operators, so that every operator has a well-defined type. So no need to reconstruct functions after dependency changes. But if we want the ability to specify implementations of "add" in more than one file, we will still need a form of dependencies in math.ts because any other operation like "sum" that needs the full generality of "add" has to wait until all of add's implementations have been collected. So we will have to have as an implementation of "sum" some literal typescript entity of some type that indicates that "sum" depends on "add" -- something like a generic function that takes an object with a literal key add with value of (function) type T and returns the actual implementation of "sum", which will have a function type that depends on T in some potentially complicated way.

But now I am concerned, because that seems to mean we want to write something like:

const sumImps = [
    function genSumList<T>(deps: {add: T}) {
       return function sumList(...addends: UnionOfAllowedParameterTypes<T>[]) {
          // body for adding all the addends
       }
    },
    function genSumArray<T>(deps: {add: T}) {
       return sumArray(addends: UnionOfAllowedParameterTypes<T>[]) {
          // exactly the same function body, so hopefully can somehow share
        }
    }
]

So we will need to write this type transformer UnionOfAllowedParameterTypes<T> that takes the intersection of multiple function types, extracts all the parameter types from each of them, and unions them up. And that's just for sum; for more complicated dependencies who knows what type transformers we will have to write. And writing any one of these is no picnic; just look at the "black magic" that is UnionToIntersection<T>. (I am also not 100% sure if it just works OK to stuff generic functions in an array like this...)

Anyhow, suppose we get all the necessary type transformers written. Without dependencies, producing the type of an overloaded function from its implementations isn't too bad, as m93a has shown us -- it's just the intersection of the function types of all of its implementations. But now with dependencies, if any implementation is a generic like this, the type transformer that produces the type of the overloaded function has to detect that situation (how?), and also have as a parameter an object type mapping all the keys of the dependency operations to their resulting overloaded types, and has to apply the type transformer of the generic to those dependencies to produce the final implementation type, and only then can it include that implementation type in its intersection.

In short, basically it looks to me like the whole dependency resolution code has to be also implemented in the TypeScript type system. It's well established that this type system is Turing complete, so in theory this is possible, but it is a somewhat daunting prospect. And at least for me, writing complicated type transformers is a slow, painstaking process. Just these explorations have been a big time commitment for me already.

Does it seem like this is a path worth continuing down? I could plow into dependencies along the lines above, but then there still looming are automatic conversions which throw yet another big complication into generating the type of an overload from its list of implementations.

It is fairly difficult for me to see the light at the end of this tunnel, especially when I worry that any time a new operator with dependencies is implemented, it may mean writing some new type transformer.

So, what are your current thoughts about the trajectory of mathjs and TypeScript? Actually using mathjs in the Numberscope project that I came to mathjs from is currently stalled on full bigint integration, but that is not something that should be attacked before a major refactor if there is going to be one (and also something that is fairly unappetizing with the current organization where nearly every source file must be touched). I can try to devote some more effort to breaking this logjam but it's now not clear to me where that effort will be most effective, and it's not practical for the logjam to simply stretch on for an indefinite time. So any thoughts/advice are very appreciated, thanks.

[gwhitney]

When considering these sorts of issues, I think items like https://github.com/microsoft/TypeScript/issues/42204 in which Max Heiber points out an unsoundness in the type system of TypeScript, and it is simply closed as "working as intended," with Ryan Cavanaugh, the TypeScript lead engineer, saying "I don't find these type specifications to be coherent." This is perhaps an odd statement from a language design point of view: if the expressions are legal, they should denote something, and if they don't denote something, then they should not be accepted as type specifications.

In more detail, the reason this instance of such a viewpoint is particularly relevant to the TypeScript future of typed-function/mathjs is that the difficulty Max highlights is specifically with the treatment/behavior of intersections of function types. And even these first efforts at making a typescript analogue of typed-function are making heavy use of intersections of function types. And if we go down the dependencies path, that use will become much heavier -- it seems like the first thing you need to do to implement a sum dependent on add is take the Parameters<> of the add function type. But the built-in Parameters<> type transform does not work for this:

type InterFunc = ((x: number) => boolean) & ((y: string) => bigint)
type InterParam = Parameters<InterFunc>
let myFunc: InterFunc
myFunc = (a: number | string) => {
    if (typeof a === 'number') return true
    if (typeof a === 'string') return BigInt(a.length)
}
const ip: InterParam = [2.2]

This code segment actually produces two compiler errors. In the assignment to myFunc, it produces the error that type (a: string|number) => bigint|true is not assignable to type InterFunc because bigint is not assignable to boolean. But the entity being assigned to myFunc clearly conforms to Interfunc: for number arguments it produces a boolean and for string arguments it produces a bigint.

In the definition of ip, TypeScript complains that type 'number' is not assignable to type 'string'. However, clearly the InterFunc type specification allows a single number argument. Hence this seems to be a bug.

So I will ask a question about the first one on StackExchange and file a TypeScript bug for the latter. But it seems to me that currently TypeScript can't really reason about function type intersections in a coherent way, which makes it seem unsuitable for implementing an analogue of typed-function. To bring that back to the topic of this issue, it reinforces my recommendation that the best we can do is a JavaScript typed-function with an automated TypeScript .d.ts generator.

[gwhitney]

Wow, I already have an answer on StackOverflow about the first compiler error: TypeScript is deliberately bad at type checking function intersection types because it is deemed that the extra computation time to do so is not worth it. See https://stackoverflow.com/a/70089922/5583443. But this fact about TypeScript does not bode well for typed-function: it seems we would be sailing directly into a region of the TypeScript ocean that has been deliberately left stormy.

[gwhitney]

OK, I have filed the second compile error as https://github.com/microsoft/TypeScript/issues/50975. We shall see what sort of response it generates (but I will be very surprised if it turns out to be as fast as StackOverflow was).

[gwhitney]

OK, eating my words: indeed, I already have a response on the above issue. The gist is that this is a known difficulty in the TypeScript type system but it is considered to be a "DesignLimitation", i.e. it is not deemed that there is any practical way to fix it. So there you have it: none of this gives me any comfort with trying to proceed with a full typed-function analogue in TypeScript with dependencies and automatic type conversions. typed-function is all about intersections of function types, and TypeScript is consciously very limited in operating on such types.

[josdejong]

I love the new project name typomath again :joy:. Thanks for your efforts! Be careful, before you know it you're an TypeScript expert :wink:

One thing that I'm not sure whether it can be solved with inferring types in TypeScript is whether it is possible to use more complicated logic instead of a string replace. We need that to return a more complex type from a simple definition, like when defining a function add(a: number, b: number), and having a conversion from bigint to number defined, the returned type should include all combinations of the possible conversions.

In short, basically it looks to me like the whole dependency resolution code has to be also implemented in the TypeScript type system.
[...] It is fairly difficult for me to see the light at the end of this tunnel

I have the same feeling right now. I don't see a complete picture with all of this working out nicely so far. I feel your frustration. I think what we envision with typed-function and pocomath is too dynamic to play nice with TS. I'm more and more thinking about some hybrid approach, where we let typed-function generate the TS types for us during some build step (similar to how we generate all these entry files in ./src/entry right now), basically automating the manual TypeScript file that we currently have.

it reinforces my recommendation that the best we can do is a JavaScript typed-function with an automated TypeScript .d.ts generator.

yes, I think we're on the same page.

@m93a do you think the conclusions of @gwhitney are right, or are there possibilities of TypeScript that we're still overlooking?

[josdejong]

One more idea in this regard: if we go for an offline script to generate the type definitions, we do not have "on the fly" type checking and that will be a limitation whilst doing dynamic imports and merging and adding new function signatures yourself to your own mathjs instance. Maybe it is possible to write a TypeScript plugin that is capable of doing type checking of your typed-functions on the fly, by trying out creating the typed function in the background, and returning linting errors when there is an issue in how you try to use your own dynamic typed-function?

[gwhitney]

having a conversion from bigint to number defined, the returned type should include all combinations of the possible conversions.

Well, I think automatic conversions by itself can be handled as long as you require that all the conversions are specified before any overloads are specified. That way, you can insert conversions on each individual overload as it comes in. Basically, what automatic conversions means is that if you have conversions from types A, B, and C to D, then any parameter of type D should be substituted by a parameter of type A|B|C|D. With mapped tuple types that should be doable.

But I remain just as worried as ever by the combination of dependencies and automatic conversions because it seems that at some point in there you will need to process intersection types of functions, and as I documented above Typescript is deliberately limited in its handling of such types.

Incidentally I also have no idea how to implement an overload by a template function, e.g

const addImps = [
    (x: number, y: number) => x + y,
    (x: bigint, y: bigint) => x + y,
    <T>(x: Complex<T>, y: Complex<T>) => x.add(y)
]

Admittedly typed-function does not have this feature but Pocomath does and it seems like it would be nice to get it into the next version of the internals, and it would be/would have been very nice if it just came "for free" by typescriptifying typed-function. But I don't at the moment anyway see how it does. As with everything TypeScript, maybe it does if we just did things right but I am not clever/experienced enough to see how to do it.

[gwhitney]

possible to write a TypeScript plugin that is capable of doing type checking of your typed-functions on the fly

I love the creativity in finding possible ways to deal with dynamic aspects in typescript. I am a bit skeptical because plugins run in typescript language services, not in the compiler itself. So I had just assumed that in TypeScript if you want to extend mathjs/mathts (either way we implement it) you have to set up a new instance using the ingredients of the standard one with your extensions before "finalizing" it -- you won't be able to extend it "on the fly" like you can at least in many ways in JavaScript. (I imagine in the mathjs case that means making a module that takes the standard instance, extends it, and calls the utility to write out a new .d.ts, whereas if we could pull off the mathts version it would mean making a new module that pulls in all the standard implementation definitions that assemble to make the standard mathts, incorporating some additional implementations, and then calls the "master assemble" function that properly types all the methods. But once that's done, the resulting math object has no further extensibility or customizability, since those things would change its type, which is impossible.)

[gwhitney]

Just to focus my recent comment about dependencies: I now actually see that there are some workarounds to TypeScript's limitations on working with function intersection types, e.g. https://stackoverflow.com/questions/52760509/typescript-returntype-of-overloaded-function. So the various type operators one needs for doing dependencies in a TypeScript analogue of typed-function may quite possibly be implementable, albeit very complex.

UPDATE: per https://stackoverflow.com/questions/73939196/obtain-the-union-of-parameter-types-for-an-arbitrary-overload, we would need to basically have a case statement in the type system over all possible numbers of implementations to do this; currently I think we have some operations with dozens of implementations, counting automatic conversions, so this would be complex indeed./UPDATE

But I am still concerned about one core obstruction. Suppose by way of example (even though this wouldn't really be a good implementation) that you wanted to define subtract(x,y) basically as add(x, negate(y)). So you would make subtract depend on add and negate by some notation/definition. For Typescript to now know what types subtract can operate on, it will have to know all of the types that add and negate can operate on. So we have to guarantee that all of the implementations of add and negate have been read and assembled into the big, narrowly-typed literal that the TypeScript analogue works with until such time as it assembles that into an object with a real method for each operator. And then the process for creating dependencies has to have access to the add and negate information and be able to process it to be able to produce the correctly-typed literal for subtract.

In the current architecture for mathjs, in which all implementations of add reside in the same file, it seems reasonably plausible to be able to do that. But besides switching to TypeScript, it seems a major goal is to be able to decentralize the implementations of functions like add so that it will be possible to incorporate new types like bigint just by adding more imports, without having to modify many existing files. I don't see how to combine these two goals. If I can just add more imports, how will subtract (or the new TypeScript typed-function-style infrastructure) be able to be sure that it has all of the implementations for add at the point that it encounters the implementation(s) for subtract? Collecting up all of the implementations of all operators into one big narrowly-typed literal before "crystalizing" it into a math object with methods seems to work very nicely in the absence of dependencies, in that it is very tolerant of the pieces being collected up in different orders, but I don't see how to make that work as desired in the presence of dependencies. I mean, one could try to define "strata" and say that the implementations for a new type need to be split into these strata where the implementations in one stratum are only allowed to depend on operations in earlier strata. Then you would "only" have to add your import for a new type to each of the files that import all of the functions of a given stratum, but certainly as mathjs currently stands there would be an awful lot of strata, so it seems like you'd still have to modify quite a few files to add a new type... But maybe I am still missing something here as to how dependencies might work in practice in TypeScript.

[gwhitney]

A further thought: the dependency problem may be even more acute with self-dependencies. Consider the "square of the absolute value" function absquare which should perhaps correspond to something like the following (ignoring for now the issues of not necessarily being able to use + on generic types, and of how self gets filled in):

const absquareImps = [
    (n: number) => n*n,
    (b: bigint) => b*b,
    <T>(z: Complex<T>) => self(z.re) + self(z.im)
]

For TypeScript to type this, it will need the function type of self. That function type will depend on the collection of all implementations of absquare. It's perhaps plausible that could be done if all implementations of absquare are collected into the same literal in a single file. But how could this happen if the implementations of absquare are distributed into multiple files?

[josdejong]

Thanks Glen for the clear explanation of these pain points. I think it's time to come to a proposal and make a decision based on all of the above.

Options

So, basically, there are three possible approaches:

1. JS code base, manually crafted TS definition file. This is the current situation
2. JS+TS code base, with automatically generated TS definitions files for typed-functions. We can extend typed-function to output TS definitions for a typed function, and we can write these definitions to a file so we can auto generate all TS defintions that we currently maintain by hand. We can write the code base of mathjs itself also largely in TS, all the util files and everything.
3. TS code base, expanding on the over.ts POC.

Evaluation

We've had (1) for a long time now, and it doesn't work: the TS definitions are always behind the JS functionality, and it causes quite some work and frustration to try keep the TS definitions on par.

We've done a lot to try get (3) to work. However we're hitting quite some walls there. Some of them may be solvable but we need to jump through hoops and it will be painful and complex. This is a clear sign to me that it is better not to go that route. I think the static nature of TS simply doesn't match the dynamic nature of what we try to achieve: allow a user to dynamically mix&match functions and data types. I also do not see a way where partially sacrificing our goals can result in a solution that does play nice with TS.

Option (2) is simple, straightforward and pragmatic. It will not give a 100% TS experience, but we can get close and it solves the big pain point of the current situation (1) without giving up the flexibility that we want (dynamically mixing&matching functions and data types).

Proposal

Unless we gain new insights within one or two weeks, I propose we go for option (2).

Agree @gwhitney and @m93a? Or do you still see other ways?

[gwhitney]

I agree (1) is broken, also because the prospect of having to change almost every source file to fully integrate bigint is very daunting.

I do feel I have tried to push (3) as far as I am possibly capable, and in the end the most difficult obstacle I encountered is how to get TypeScript to type self-referring typed-functions, as in my most recent comment before this. I don't personally see how to accomplish that, even without contemplating any changes to operators after the math object is put together.

Hence, I am comfortable with option (2) as a general framework.

[ljb630]

It's great to hear that you have explored different options for integrating bigint into your project and have found a solution that works for you. Option (2) can indeed be a viable framework for incorporating bigint functionality into a TypeScript project, especially if you've encountered challenges with options (1) and (3).

In terms of the challenge you mentioned with typing self-referring typed-functions in TypeScript, this can indeed be a tricky issue to tackle. One approach that may work is to use a recursive type definition, where the type of the function refers to itself. Here's an example of how this might look:

type MyFunction = (input: number | bigint) => number | bigint | MyFunction;

const myFunction: MyFunction = (input) => {
  if (typeof input === 'number') {
    return input + 1;
  } else {
    return myFunction(input - 1n);
  }
}

"MyFunction" type refers to itself in the return type, allowing the function to be called recursively with bigint inputs.

[josdejong]

Thanks for your inputs Laljan.