Support for .d.ts files in the compiler

[back2dos]

As pointed out on the list, it would be great to support .d.ts files for haxe/js, much like we support .swc for haxe/flash and .jar for haxe/java.

[Simn]

Before everyone gets too excited about this: I'm currently against this for various reasons which I will point out at a later time. I'll leave this issue open for discussion though.

[nadako]

We discussed it a bit on #haxe, there's still no concensus, but one viable alternative would be to improve ts2hx tool and set up an automatic bot that watches DefinitlyTyped repo, converts it to haxe externs and pushes to a haxe mirror repo. It could also do automatic haxelib releases!

[ncannasse]

I also think that an offline converter is better than a direct one, it also allows to fix the most tricky cases (which can't be decided purely from TS data because of missing information) - if there's any.

@nadako do you volunteer to setup this ?

[nadako]

Maybe, but first we need to fix ts2hx, as it can't handle DefinitelyTyped repo right now.

[back2dos]

On Thu, Mar 5, 2015 at 10:44 PM, Nicolas Cannasse notifications@github.com wrote:

I also think that an offline converter is better than a direct one, it also allows to fix the most tricky cases (which can't be decided purely from TS data because of missing information) - if there's any.

What do you mean by "missing" information? TypeScript's type system is a lot closer to Haxe's than that of the AVM, and if SWC is good enough for us, I don't see why TypeScript wouldn't be.

Also fixing tricky cases by hand and having an automated process doesn't go together well. Anytime there is a change upstream, things will get messy.

[ncannasse]

@back2dos SWC is not good enough, hence https://github.com/HaxeFoundation/haxe/blob/development/extra/extract.patch

[back2dos]

You're moving the goal post. SWC is evidently good enough, since the compiler supports it and people use it. If they want to write patch files, fine, but they don't have to. And with TypeScript, I don't think anyone would bother writing those, if you can actually just fix the source (and make a pull request against DefinitelyTyped). TS has simple enums, generics, function types and so on. The definitions are sooner or later going to converge onto something exact. I don't get why we should do the work, if a bigger community is basically doing it for us ;)

On Thu, Mar 5, 2015 at 11:31 PM, Nicolas Cannasse notifications@github.com wrote:

@back2dos https://github.com/back2dos SWC is not good enough, hence https://github.com/HaxeFoundation/haxe/blob/development/extra/extract.patch

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3977#issuecomment-77466673 .

[nadako]

One important argument agaist ocaml version is that user have to wait for a haxe release to fix library loader bugs or support new TS feature, however with ts2hx we could easily upgrade.

[back2dos]

I invoke "batteries included" as a counter argument that seems to carry a lot of weight around here :P

Also, the argument doesn't really hold. You could set up the bot to watch DefinitelyTyped use the latest haxe nightly to generate externs with --gen-hx-classes. Just saying.

On Fri, Mar 6, 2015 at 12:18 AM, Dan Korostelev notifications@github.com wrote:

One important argument agaist ocaml version is that user have to wait for a haxe release to fix library loader bugs or support new TS feature, however with ts2hx we could easily upgrade.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3977#issuecomment-77473942 .

[nadako]

Yeah but then why one needs it be implemented in compiler, if they need to download latest externs anyway? :)

[back2dos]

Let's recap here: Your argument was that we should go for an external tool, because it has an advantage. I have merely rebutted that argument.

But yes, during a transitory period, we could have .hx externs wherever necessary and once this feature is more less stable, they can be swapped out by .d.ts files.

Things to also consider:

1. The type system of a language usually is among slowest changing parts.
2. There may eventually be changes to the TypeScript type system, but the same is true for Java, which could some day get ADTs for example. The same could also have happened with ABC, if the leap to a significantly richer AS version ever had seen the light of day.
3. When new features are added to the type system, it's not like everybody is immediately going to use them. Just because there is a new kind of type added to TypeScript, it doesn't mean that somebody will go through all of DefinitelyTyped and update them to use it.
4. A relatively forgiving parser should help us in establishing reasonable forward compatibility. And so long as only the exotic things wind up as Dynamic, it doesn't really take away from the usefulness.

I too have mixed feelings about this. It kind of feels like tainting the compiler with an unpleasant dependency on other people's design decisions. But the truth is that we haven't been afraid of that in the past, and right now this would be a really great move. One that could set impulses, create buzz and momentum. And not by just pleasing some personal tastes that are à la mode, but by actually tapping into potential that is right out there for the taking. That's what Haxe does, no?

Intuitively I find a tight on-the-fly integration to be more promising. And I imagine that in the end you find that intuition quite plausible. If you guys think that ts2hx does much better in a cost/benefit analysis, then so be it. You are the only ones who have an idea about the cost side of things, so I will leave this in your capable hands. Enough said, before I start boring Nicolas again ;)

[frabbit]

The problem with typescript is that it has at least one feature which is hard to emulate in haxe.

See section "TypeScript Inherits Differently" - https://msdn.microsoft.com/en-us/magazine/dn890374.aspx

DefinitelyTyped makes use of this, so before talking about integrating such a feature into the compiler or using a tool like ts2hx or whatever we need a solid solution for translating every feature of typescript.

[frabbit]

see section 7.2 and 7.3 in http://www.typescriptlang.org/Content/TypeScript%20Language%20Specification.pdf

[nadako]

easier to link to github spec: https://github.com/Microsoft/TypeScript/blob/master/doc/spec.md#7.2

[skial]

There are two ocaml javascript parsers, one parses es5, called lambdaS5 and facebooks parser, flow, which I think parses es6. I have no idea if its even possible for you guys to use either of these libraries with Haxe.

[back2dos]

Not sure I completely understand the problem with the different type semantics. Would this be accurate?

1. TS has partial classes, which I assume can be dealt with by merging the declarations.
2. TS interfaces can extend classes, which is trickier.

I would assume that we can sidestep the second issue by having interfaces as typedefs. Having this:

class Customer
{
  private id: number;
  get Id(): number
  {
    return this.id
  }
  set Id( value: number )
  {
    this.id = value;
  }
}
interface ICustomer extends Customer
{
  MiddleName: string;
}

We'd get this:

class Customer {
   var Id(default, default):Float;
}

typedef ICustomer = {>Customer,
   var MiddleName(default, default): String;
}

Of course {>A } being deprecated A being a class is a bit of an issue, but that's something I never really understood in the first place.

And if we want to write custom classes explicitly extending typescript interfaces, we can do it like so:

class MyCustomer implements Interface<ICustomer> {}

Where Interface is a @:genericBuild type constructing an appropriate interface on the fly, probably with @:remove. But that's a little extra anyway, that might even be useful outside of this undertaking.

[ousado]

Is there a good, general way how to deal with the outcome of different approaches to JS API design that TS people wrapped in some way they thought would fit TS?

There are always decisions that have to be made for a specific API, so we need to be able to influence the decisions of an externs generator. We need fine-grained, explicit, programmatic control over the process to produce high-quality externs.

Now, you can follow some "batteries included" logic and make people rewrite an existing and written in Haxe tool in ocaml just to then go out of your (or rather their) way to be able to write Haxe plugins for that tool (which then manage the API-specific aspects of extern generation) or you can just extend the existing tool, in Haxe, of course.

Also, there are other formats to consider, JSON, JSCA, JSDuck and whatever else is used to more or less formally describe JS APIs.

A good approach to the problem would be creating a/extending the tool with different frontends for formats that are used with reasonable frequency, a layer for possible API-specific transformations and a common externs generator.

And just like haxelib is being called from the compiler, that tool could be too, at some point, for those who think the "batteries included" experience is that important.

[ousado]

Regarding the TS vs SWC question, at least flash knows the difference between Ints and Floats. I don't know about you, but I consider externs not making that distinction low quality.

[back2dos]

The issue with JS centric solutions is that many of them are quite fuzzy and do little to no static analysis.

Also, if there is a good and popular format out there, I'm pretty sure converters to TypeScript either exist or are only a matter of time.

But more importantly: is there any repository comparable to DefinitelyTyped for any of those formats? They have 786 projects. To put that into perspective, at the time of writing haxelib has a total of 522 projects and GitHub has 1171 JavaScript repos with more than 1000 stars. And currently only 78 projects are requested, so you could naively set their coverage of the popular parts of the JS ecosystem to be between 67% and 91%. I would say that's pretty good. But sure, if there are numbers to suggest a different solution, let's do that.

As for your TS vs SWC comparison, you are cherry picking and the particular cherry you picked I would consider trivial. SWC has only one function type. Pass an incompatible function, you get runtime errors. Pass a float to a JS function that only deals with ints, you can bet the implementation just coerces it an life goes on. The same goes for arrays and what not.

[waneck]

I'm with Juraj. I really like library loaders that are baked into haxe, and I think they add value to the compiler - much more than if you'd need an external library. For the user, it feels much different if you need a third party tool vs only an extra compiler flag. For example, there were externs creators before -net-lib and -java-lib, but adoption was so much more restricted - even I sometimes just ended up not using Haxe instead of using those tools.

[ousado]

@back2dos Please enlighten me what random statistics about DefinitelyTyped, haxelib and github have to do with the choice of an implementation language for an externs generator, and how they suggest a specific one or a specific place where one should be implemented.

[Justinfront]

+1 for -ts-lib since I would love for example to have a -ts-lib for use with creating 2d graphics I think they may have some integration with adobe tools so could be interesting. https://github.com/awayjs/awayjs-core/tree/master/lib

I have not tried ts2hx if it was built in I am sure I would but standalone I agree it's less inviting and less likely to be used but I found something interesting which we may wish to explore. haxe -> Netjs -> typescript https://github.com/praeclarum/Netjs I presume I could feed it some mono from haxe?

In terms of the way the web works with current js "fads" it looks like angular2 is using AtScript which is like a bit more on top of Typescript. some old chit chat here http://sdtimes.com/atscript-googles-new-superset-javascript-runtime/ spec on this subset and how it works.. https://docs.google.com/document/d/11YUzC-1d0V1-Q3V0fQ7KSit97HnZoKVygDxpWzEYW0U/mobilebasic?viewopt=127 then a link for playing with it... https://github.com/angular/atscript-playground Kind of confusing how Angular, AtScript, Typescript all fit together https://github.com/davideast/ng2do

But I think maybe we should add an orange circle around this diagram... ;) http://sdtimes.com/wp-content/uploads/2014/10/Screen-Shot-2014-10-27-at-12.45.39-PM.png

[back2dos]

@ousado: Sure, if you could first explain to me what 1) externs generators and 2) the "other formats to consider" have to do with "Support for .d.ts files in the compiler". If you wish to broaden the scope of the discussion, then don't blame others for it. DefinitelyTyped presents us with an infrastructure that facilitates many things, particularly discoverability. I simply don't care if JSDuck or whatever is "better" if nobody uses it. I also don't care much for TypeScript for reasons other than it exists, has a type system close enough to ours and somebody has done all the tedious work for us, that's it. Unless you can come up with some tangible data on how useful the alternatives you vaguely propose have, it's just unsubstantiated spiel. And it's not like they haven't ever been considered anyway.

On Fri, Mar 6, 2015 at 5:36 PM, ousado notifications@github.com wrote:

@back2dos https://github.com/back2dos Please enlighten me what random statistics about DefinitelyTyped, haxelib and github have to do with the choice of an implementation language for an externs generator, and how they suggest a specific one or a specific place where one should be implemented.

— Reply to this email directly or view it on GitHub https://github.com/HaxeFoundation/haxe/issues/3977#issuecomment-77589494 .

[nadako]

Currently I have no strong position on whether it should be in compiler or ts2hx is enough, but I think right now we should focus on making ts2hx generating actually usable externs, supporting all typescript features in one way or another and generating proper package structure, adding @:jsRequire metadata etc. After that's done, we could think of porting it to ocaml, I think it would be quite easy, as hxparse seems to work pretty the same as ocaml parser and the usage of adt/structures could be well mapped to ocaml.

[ncannasse]

I propose we first make ts2hx working, and then if people start using these hx-from-ts files quite often, we will move it to the compiler, after all bugs fixes and contributions have been made into the haxe version

[Simn]

I have updated the ts2hx parser so it parses everything in DefinitelyTyped again.

I would really like to pass this project on to someone else though because I don't know/use typescript and lack personal motivation to support this further. Also, with the prospect of having to port everything to OCaml at some point anyway I can't shake off this feeling that I'm wasting my time with this.

[nadako]

Thanks for the parser fixes Simon! I don't know/use TS as well, but I'll try to contribute a bit to make ts2hx generate something usable, the good part that the code is very concise and easy to understand and modify! I really think we should invest some time in it, so DefinitelyTyped could serve us. After we get package structure right and everything is usable, we may move further, providing extern bot or porting to ocaml or both.

[ousado]

Just one out of many real world examples demonstrating why an externs generator should be flexible, AKA programmable in Haxe. Quotes from a #haxe IRC conversation: (with permission of the participants)

[18:33] js.html.Promise is kind of dynamiqish ... should it make a generic? [18:35] @aduros ^^^^ [18:43] fponticelli: Type param on Promise would be cool ... [18:52] Promise is definitely in Node and using it through js.html is at the very least misleading ... [18:53] if it's the same es6-related thing, i vote for moving it into the js package [18:53] +1 ... [18:58] is the generator public? [18:58] Yes https://github.com/HaxeFoundation/html-externs [18:59] I can try to PR the generator if you want [19:00] That would be great

[nadako]

anyone interested in ts to hx extern conversion, welcome to https://github.com/Simn/ts2hx/issues, let's discuss current and potential issues and how we handle them.

[sonygod]

ts2hx is no longer maintaining .