Feature Request: F# style Type Provider support?

[battlebottle]

Could we get support for something like F#'s type providers in the future?

Let me present some use case scenarios for this.. Let's suppose you have a TS project trying to access a REST API, and this REST API uses Swagger or RAML for documentation. Well in this case we could have something like this;

var fooClient = new SwaggerClient<"http://foo.com/swagger.json">();
fooClient.getPeople((people) => {
    people.every((person) => console.log(person.firstName + "," + person.lastName);
});

and this could be completely type safe, as the SwaggerClient pulls down the API return types, method names (such as getPeople) and signatures at compile time and exposes them as methods to the type system. I think that's pretty huge. Such clients could be written by the community for RAML, SOAP or what have you.

another nice one:

var doc = new Document<"index.html">()
doc.titleSpan.text = "Hello World!";
doc.myButton.addListener("clicked", () => { console.log("tapped!")});

In this example the Document type provider loads our index.html file and finds all the elements with id's and figures out what types they are, and exposes them in the type system at compile time. Again, I think that's pretty huge!

F# uses Type Providers for all kinds of things, so there's lots of ways this can get used. It's easy to see frameworks figuring out all kinds of amazing things that these could be used for. It also feels like a nice fit for TypeScript because it helps bridge the benefits on dynamic typing with the benefits of static typing.

one more tiny example:

console.log(StringC<"name:%s/tid:%n">("Battlebottle", 128));

a simple string formatter. This reads "name:%s/tid:%n" and knows that %s and %n must be replaced with a string and number respectively, so at compile time it produces a method that accepts a string parameter and a number parameter. Basically a statically typed string formatter. Just a tiny of example of handy little utilities that could be written with this feature. But the opportunity is pretty huge I think in general.

Any thoughts?

[saschanaz]

:+1: with the example for HTML. I don't like writing many declare vars to access them.

[mhegazy]

The idea is definitely something we have talked about on and off over the past few years. We will need a detailed proposal of how are providers declared, implemented, loaded and discovered.

[battlebottle]

I'm by no means any kind of expert on the TS compiler. I'm not even particularly advanced TS user right now. However, just to get the ball rolling I'll take a stab at a potential implementation for this.

There's obviously a bunch of different ways this could be done, and choosing the best way will depend on a deep understanding of TypeScripts goals as a language, how modern JS runtimes optimise code, among other things.

Here's a proposal for what the type provider might look like for the string formatter example: (Please excuse the string formatting logic here, it's pretty much pseudo code. I didn't test and it's probably full of bugs.)

//Type Provider definition
StringF =
    @TypeProvider (str : String) => {//str eg: "id: %n/tname: %s"
        //Get array of format specifiers eg: ["s", "s", "n"]
        var formats = 
            str.split("%")
            .map((str) => {str.substring(0,1);})
        formats.splice(0, 1);

        //Convert formats to function arguments eg: [{name:"arg1", type: TP.baseTypes.String},{name:"arg2", type: TP.baseTypes.Number}]
        var funcParams = (() => {
            var types = formats.map((p) => {
                if (p === "s") {return TP.baseTypes.String;}
                else if (p === "n") {return TP.baseTypes.Number;}
                else if (p === "b") {return TP.baseTypes.Boolean;}
                else {throw p + " is not a valid string format specifier"}
            })
            var args = []
            for (var i = 0; i < types.length; i++) {
                args.push({
                    name: "arg" + i
                    type: type
                });
            }
            return args;
        })();

        return new TP.Function({
            parameters: funcParams //eg: [{name:"arg1", type: TP.baseTypes.String},{name:"arg2", type: TP.baseTypes.Number}]
            returnType: TP.baseTypes.String
            functionBody:
                @Expression """
                    var stringSplit = str.split("%");
                    for (var i = 1; i < stringSplit.length; i++) {
                        stringSplit[i] = stringSplit[i].substring(1, stringSplit[i].length - 1);
                    }

                    for(var i = arguments.length - 1; i >= 0; i--){
                        stringSplit.insert(arguments[i], i);
                    }
                    return stringSplit.join("");
                """
        })
    }

//Type Provider being used
console.log(StringF<"id: %n/tname: %s">(128, "battlebottle"))
console.log(StringF<"id: %n/tname: %s">(128, "ben brown"))
console.log(StringF<"id: %n/tname: %s/tage: %n">(128, "battlebottle", 30))

In this implementation, a type provider is simply a function that gets annotated with @TypeProvider. Doing this means that this method must be evaluated at compile time. The type provider function may accept parameters. In this example the type provider accepts a str : string parameter. When the type provider is referenced in code, it must be passed this argument as a generic parameter, such as in StringF<"id: %n/tname: %s">, "id: %n/tname: %s" is the str parameter. This parameter needs to be a literal as it must be evaluated at compile time. With this parameter, the type provider function will generate and return an AST node to represent the object, function etc that StringF<"id: %n/tname: %s"> will represent. In this case StringF<"id: %n/tname: %s"> will represent a function with the signature (arg1 : number, arg2 : string) : string.

As you can see, the TP.Function AST node constructor accepts an object with a parameters field, which must contain the name and type of all the parameters, a returnType field, which obviously declares the return type, and then a functionBody field. This one I'm less certain about, but here I currently have this as a string containing TS code representing the function body. Marking this string with @Expression could help let IDE's know that this string contains TS, and should perhaps be validated. This is something of a pragmatic solution. In F# the equivalent of this would be writing <@@ 1 + 1 @@> or what have you, with <@@ converting the expression inside into AST nodes. This is just syntactical sugar though for creating the AST nodes directly in code, and I don't think it does a whole lot for type safety. Evaluating JS string at run time is more normal in JS than doing something similar in C#/F#, and writing the function body as a string like this makes the whole process of creating a type provider a lot less complex than if the whole expression needed to be constructed with AST nodes. The TP.Function constructor defines the type signature of the function being created, which provides a layer of safety for when the type provider method is being evaluated, without making things feel as overwhelmingly complicated as writing F# type providers is today imo. I really have no idea what the best thing to do here. There's probably all kinds of things to consider here I haven't even thought about, but I'll leave this here as a starting point at least.

The type provider method is evaluated at compile time for each time the type provider object is referenced with different parameters. So for example:

console.log(StringF<"id: %n/tname: %s">(128, "battlebottle"))
console.log(StringF<"id: %n/tname: %s">(128, "ben brown"))
console.log(StringF<"id: %n/tname: %s/tage: %n">(128, "battlebottle", 30))

results in the type provider being evaluated twice, since two of these methods have the exact same parameter, they can refer to the same function.

The AST node that the type provider function returns, is converted to a real function (or anything else) at compile time for any code referencing the type provider, and these AST nodes are also used to generate the JS output.

So lastly, here is the JS that would be created from this code if we compiled it:

var StringF = function (str) {
    switch(str) {
        case "id: %n/tname: %s":
            return (function(arg1, arg2) {
                    var stringSplit = str.split("%");
                    for (var i = 1; i < stringSplit.length; i++) {
                        stringSplit[i] = stringSplit[i].substring(1, stringSplit[i].length - 1);
                    }

                    for(var i = arguments.length - 1; i >= 0; i--){
                        stringSplit.insert(arguments[i], i);
                    }   
                    return stringSplit.join("");
                });
        case "id: %n/tname: %s/tage: %n":
            return (function(arg1, arg2, arg3) {
                    var stringSplit = str.split("%");
                    for (var i = 1; i < stringSplit.length; i++) {
                        stringSplit[i] = stringSplit[i].substring(1, stringSplit[i].length - 1);
                    }

                    for(var i = arguments.length - 1; i >= 0; i--){
                        stringSplit.insert(arguments[i], i);
                    }
                    return stringSplit.join("");
                });
        default:
            throw "Could not find match for parameter: " + str;
    }
}

console.log(StringF("id: %n/tname: %s")(128, "battlebottle"));
console.log(StringF("id: %n/tname: %s")(128, "ben brown"));
console.log(StringF("id: %n/tname: %s/tage: %n")(128, "battlebottle", 30));

I've tried to keep to the TypeScript spirit of keeping things "just JavaScript", and making it so the JS output is still highly readable and could comfortably be used if the developer decided to drop TypeScript at some point.

As you can see StringF<"id: %n/tname: %s">(128, "battlebottle") is compiled to StringF("id: %n/tname: %s")(128, "battlebottle"). This keeps it looking largely familiar to the original TS code. The StringF variable, which was originally a type provider, is now a function. When StringF("id: %n/tname: %s") is called, it matches "id: %n/tname: %s" to the function that was generated at compile time for that parameter. Assuming the JS is unmodified, there will always be a function to match for the str parameter passed. I think this keeps things pretty readable, and if a developer wishes to drop TypeScript they could easily refactor this code to make it more idiomatic.

[Lenne231]

Type Providers for TypeScript would be awesome for something like Relay/GraphQL https://facebook.github.io/react/blog/2015/05/01/graphql-introduction.html and FalcorJS https://www.youtube.com/watch?v=z8UgDZ4rXBU

[Arnavion]

Another approach is to have a custom CompilerHost that generates foo-swagger.d.ts dynamically (preferably using some API like the one requested in #9147 as opposed to string concat), and either persists it to disk or services readFile requests for it from memory.

[MartinJohns]

The solution that @Arnavion proposes would also help with importing TS-foreign types in combination with webpack. A few use cases are written in #6615, and that issue is already a great change it doesn't go far enough (in my opinion). It would be great if there's a way to hook up the system and provide custom definitions for code like import styles from './main.less';.

[basarat]

It would be great if there's a way to hook up the system and provide custom definitions for code like import styles from './main.less';.

Agree that something like this would be cool. But I freestyle : https://github.com/blakeembrey/free-style (keep it all in ts for reals).

My opinions on the matter : https://medium.com/@basarat/css-modules-are-not-the-solution-1235696863d6 :rose:

[MartinJohns]

@basarat While that is definitely a possible alternative, I don't think it's for the better.

• By moving stylesheets to the JavaScript, you also delay rendering of the homepage. Commonly you reference the StyleSheet first and use pre-rendering, and reference the JavaScript last. That way the browser can immediately start rendering. By moving it to the JavaScript you delay rendering until the browser has downloaded the JavaScript (which takes longer because it also contains the application logic), the browser needs to parse and JIT the JavaScript and execute it.
• It makes development much more difficult for people who are skilled in CSS, but less so in JavaScript.
• You can't utilize existing tooling as well.

[basarat]

By moving stylesheets to the JavaScript, you also delay rendering of the homepage

Sorted by simply writing out the css https://github.com/blakeembrey/free-style#css-string and putting it in a style tag.

It makes development much more difficult for people who are skilled in CSS

The people I speak of use ReactJs. Also they don't like the cascade + super kick ass deep in a dom tree selectors anyways.

You can't utilize existing tooling as well.

All you needs is TypeScript + TypeScript tooling :rose:

For further discussion please mention me at the blog as I'd rather not hijack this thread. Also type providers are a cool idea and if people want them for css then 💯 x 💕

[felixfbecker]

This would be nice for type declarations of javascript libraries that do dynamic stuff, like Bluebird.promisifyAll()

[elcritch]

Any current discussion on this topic? JSON/REST have become the lingua franca of data communications. Having type provider supports would be an enormous boost in productivity for accessing the plethora of web api's! Is this still just waiting for specs on loading it and providing access?

[weswigham]

I have a partially functional prototype built on the extensibility model work I did over the summer and enhanced type checker APIs; however don't expect any progress in this area for a bit (or any commitments) - things likely need to stabilize for a bit and this will need to be revisited one extensibility as a core feature is nailed down.

On Tue, Sep 20, 2016, 10:11 AM Jaremy Creechley notifications@github.com wrote:

Any current discussion on this topic? JSON/REST have become the lingua franca of data communications. Having type provider supports would be an enormous boost in productivity for accessing the plethora of web api's! Is this still just waiting for specs on loading it and providing access?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/TypeScript/issues/3136#issuecomment-248312756, or mute the thread https://github.com/notifications/unsubscribe-auth/ACzAMj4tfuI7RmUEEL8KjsQ-fAP8qyOTks5qr-mMgaJpZM4EYaDm .

[Ciantic]

I'm throwing this obvious thing in here, any future thoughts about the feature should take in account React components, e.g. think about these massive libraries which has types, but they are in React "propTypes" nonsensical format:

https://github.com/callemall/material-ui/blob/next/src/IconButton/IconButton.js#L75

export default class IconButton extends Component {
  static propTypes = {
    children: PropTypes.node,
    className: PropTypes.string,
    contrast: PropTypes.bool,
    disabled: PropTypes.bool,
    ripple: PropTypes.bool,
    theme: PropTypes.object,
  };
  ...

So type provider should be able to provide types for whole module in this example.

Right now I'm wondering if I should write a d.ts generator for React components alone, I imagine it could typify all properties of this library pretty autonomously.

[mohsen1]

Things like Typed CSS Modules can be done in much better fashion with this.

[jvilk]

While messy, a workaround is to generate interfaces outside of the compiler. I implemented the techniques from F#'s JSON type provider into a tool that generates TypeScript interfaces. Maybe it'll help some of you?

[saschanaz]

Is this possible now as TS added plugin support?

[DanielRosenwasser]

Not quite - we have language service plugins that can augment the experience in something like a string literal or whatnot, but nothing that changes the fundamental type of an expression.

[saschanaz]

Not sure what you mean by "the fundamental type of an expression."

I've seen a repo about implementing kind-of string type enum using custom plugin, I thought adding HTML parser on it may allow:

type identifiers = htmlIdentifiers<"filepath">();
var doc: identifiers;
doc.foo // type checked

[kimamula]

Thanks for mentioning my repository, @SaschaNaz 😄 AFAIK, custom transformers cannot be used to add or modify type definition at compile time (I'd love to know how if they can!), i.e., they cannot be used to automatically provide the return type of htmlIdentifiers<"filepath">() in your snippet. In my example, the return type of enumerate<'green' | 'yellow' | 'red'>() should be { green: 'green'; yellow: 'yellow'; red: 'red'; } just because the signature of enumerate is function enumerate<T extends string>(): { [K in T]: K }; (https://github.com/kimamula/ts-transformer-enumerate/blob/master/index.ts). My custom transformer has nothing to do with this, but it is used to modify emitted JavaScript code so that enumerate actually works as declared.

[saschanaz]

Ah, I see, thanks for the detail :smile:

[Jack-Works]

What about something like this?

Somewhere in Typescript

export type TypeProvider = Function & {
    TypeChecker?: Function,
    IntellisenseProvider?: Function
}

directive.d.ts in Angular

import AngularTemplateType from './ng-template.tp'
export interface Component extends Directive {
    template?: AngularTemplateType.Type
}

ng-template.tp.ts (all type provider need to end with .tp.ts, like .d.ts, and do not emit code for .tp.ts)

import * as Typescript from 'typescript'
export type Type = string 
// ^ Fallback for old version that does not support Type Provider
// AngularTemplateType.Type will be treat as a string

const Type: Typescript.TypeProvider = function (everything) {
    return Typescript.TypeFlags.String
}
Type.TypeChecker = function (everything) {
    // Now impl type checker for angular template
}

Or what about GraphQL

import graphql from './gql'
async function getUser(id: number) {
    return await graphql`
        {
            user(id: ${id}) {
                name, age
            }
        }
    `
}
getUser(2) // <- Type of this is Promise<{ name: any, age: any }>

gql.ts

import GraphQL from './gql.tp'
export default function (s: TemplateStringsArray): Promise<GraphQL> { return ... }

gql.tp.ts

import * as Typescript from 'typescript'
export type Type = object

const Type: Typescript.TypeProvider = function (everything) {
    return Typescript.createTypeLiteralNode(...)
}
Type.TypeChecker = function (everything) {
    // Now impl validator for GraphQL
}

[qm3ster]

corpix/ajv-pack-loader could benefit tremendously, based on something like bcherny/json-schema-to-typescript

[leebyron]

I just wanted to voice my support and desire for this feature. I think it’s potentially incredibly valuable based on how TypeScript is often used these days for rich web apps with Webpack, REST & GraphQL, and other kinds of static but non-JS types.

@weswigham what ever came of your prototype from a couple years back? Did it at least yield a proposed API that could spark further discussion? I hope the TS team seriously considers building out this potential feature, though it could also be an excellent community contribution for a bold person given a clear proposed plan.

[weswigham]

So, speaking from experience, while technically you can make type providers work (and I have - I did have a functional prototype based on my extension model), a better solution (from a ux perspective) is triggered codegen. Rather than writing a library that generates the types from an API or some other file and injecting them directly into a compilation or language service, it's a way better development experience to generate actual code from the API or file. When that's done, you have a source file that actually exists and can be gone to with go-to-definition and inspected for deficiencies.

And I think a lot of people realize this (plus it already works today, since it doesn't need any specific hooks) - that's why there's projects for generating type declarations for things like json schemas and swagger specs.

[vmgolubev]

+1

[xialvjun]

if typescript has a powerful macro system:

gql!(`
query user($id: String) {
  user(id: $id) {
    name, age
  }
}`)

// will emits code:
gql`
query user($id: String) {
  user(id: $id) {
    name, age
  }
}` as GqlTemplate<{ variables?: {id?: string}, data?: {user?: {name?: any, age?: any}}}>

// if this macro system support async code emit, then it can know more: {name?: any, age?: any} :
GqlTemplate<{ variables?: {id?: string}, data?: {user?: {name: string, age: number}}}>

[nikeee]

we have language service plugins that can augment the experience in something like a string literal or whatnot, but nothing that changes the fundamental type of an expression.

Did this change? I'm looking into implementing a prototype myself and I'm evaluating what an appropriate integration would look like.

[JasonKleban]

Another use case is typesafe i18n messages ICUs. We have a distinct build step for this now, but it would be nicer if it could be hooked into the typescript project references and incremental builds mechanisms. Even using a webpack custom loader would run too late for the language service to use it.

[ozyman42]

Type Providers could be used to support many of the same use-cases as the much requested Custom Transformers feature #14419

[maxpain]

Any updates?

[trusktr]

This issue was opened before some things we have today existed. Nowadays, ES Module import / export syntax is totally standard in the JavaScript community, and we also have JavaScript runtimes that allow us to import from any where using URLs, so we don't need to add this dynamic feature to the language.

The Deno TypeScript/JavaScript runtime allows us to import using familiar import syntax from any server.

For example, in Deno we can write the following:

import SwaggerClient from 'https://some-swagger-api.com'

var fooClient = new SwaggerClient()
fooClient.getPeople((people) => {
    people.every((person) => console.log(person.firstName + "," + person.lastName);
});

In Node.js, it is simple to make a URL module loader. This means someone can make one that loads TypeScript in Node.js (go go go!).

I think this issue can be closed, and people should instead focus on building tooling around ES Module syntax.

[Dessix]

I'm not clear that this is a reason to close it - import syntax doesn't really cover many of the code-generation-specific intentions and capabilities of type providers. Do you have an example that would cover cases such as generating types for a database target at design time based on server-side introspection queries?

[infogulch]

@trusktr: like @Dessix, I think you miss the purpose of F#-style Type Providers. Type Providers are not just 'yet another way to import existing code'. Instead, they enable you to programmatically derive new types (that don't exist anywhere) at design time, which the rest of your program can then type check against without ever having them declared in a source file explicitly.