Closed Thom1729 closed 3 years ago
It looks like MS maintains a TextMate language definition for TS: https://github.com/microsoft/TypeScript-TmLanguage/blob/master/TypeScript.YAML-tmLanguage
How much modification did you have to do to the original JavaScript language definition you pulled in?
I'm not 100% sure I understand the question, so I'll try to over-answer.
JS Custom is based directly on Sublime's core JavaScript syntax. To produce customized syntax definitions, JS Custom starts with the core syntax and applies the user-selected syntax extensions. Other than what is added or modified by those extensions, the resulting syntax is identical to the core JavaScript syntax. So in a sense, I have not modified that syntax at all. This was one of the key design considerations when designing JS Custom — I didn't want it to diverge from core over time, so it's set up so that I can easily “rebase” by bundling a newer core syntax with the package.
Microsoft's TypeScript syntax definition, on the other hand, is totally separate and incompatible with the core Sublime JavaScript definition. if they ever shared a historical ancestor, then they diverged a long time ago. Microsoft's implementation is an old-style TextMate syntax. It's unsophisticated and not particularly reliable, and it relies on inefficient regexp lookbehinds. Sublime's core JavaScript definition, on the other hand, is effectively a full parser; it's designed to handle any valid JavaScript code, with only a handful of known exceptions. It is also faster.
Adding TypeScript to JS Custom support would mean writing a new extension to handle TypeScript-specific features. In increasing order of effort, this means:
The major hassle is that, while TypeScript is obviously an extension of JavaScript, Microsoft treats it as an entirely separate language. As far as I can tell, there is no official reference describing the new TypeScript syntax in terms of the original JavaScript syntax, so I'll have to reverse-engineer it from the full grammar in the TypeScript spec. This is both tedious and vexing.
If anyone has a good, authoritative-ish reference describing the difference between JavaScript and TypeScript, that would move the expected timeline of this feature from “someday” to “the next time I have a free weekend”.
Thanks for the excellent answer!
JS Custom is based directly on Sublime's core JavaScript syntax
That sounds good – I saw the "Derived from JavaScript Next" comment and thought that maybe you'd had to do some transformations to the original.
I was hoping that I might be able to just fork this module and drop in the TypeScript version of the TM language definition, but your answer makes me think that this module's code might be pretty coupled to the way that the TM definition is laid out? I haven't read this library's source yet.
The only feature I need is custom_templates.tags
– I might try reading the source and see if I can start to noodle out what it would take to get a TypeScript fork working with just that feature.
Before I dive in, can you give me any guidance as to how the library code is coupled to the language definition?
The base syntax definition in JS Custom is an exact copy of Sublime's core JavaScript syntax.
Years ago, the third-party JavaScript Next package offered much better highlighting than the core JavaScript package, so at some point the core package adopted JavaScript Next as a starting point and moved forward from there. By now, I doubt that more than a handful of lines of code remain from JavaScript Next, but the comment remains acknowledging the original contribution.
N.B. The core JavaScript syntax is not a TextMate-compatible definition. It uses the newer, more powerful sublime-syntax format.
I was hoping that I might be able to just fork this module and drop in the TypeScript version of the TM language definition, but your answer makes me think that this module's code might be pretty coupled to the way that the TM definition is laid out? I haven't read this library's source yet.
Yes. Here's an example of how JS Custom works. The core syntax begins as follows:
%YAML 1.2
---
# Derived from JavaScript Next: https://github.com/Benvie/JavaScriptNext.tmLanguage
name: JavaScript
<more stuff>
If a JS Custom configuration uses the hidden
option, then when generating the syntax JS Custom will use the hidden
extension, which is defined as follows:
%YAML 1.2
%TAG ! tag:yaml-macros:yamlmacros.lib.extend:
---
!merge
hidden: true
The implementation details aren't really important; what matters is that the resulting syntax will begin like this:
%YAML 1.2
---
# Derived from JavaScript Next: https://github.com/Benvie/JavaScriptNext.tmLanguage
name: JavaScript
<more stuff>
hidden: true
The hidden
extension is very simple and doesn't depend on the details of the core syntax. But consider the es_pipeline
extension:
%YAML 1.2
%TAG ! tag:yaml-macros:yamlmacros.lib.extend:
---
!merge
contexts: !merge
binary-operators: !prepend
- match: '\|>'
scope: keyword.operator.pipeline.js
push: expression-begin
This modifies the core syntax's binary-operators
context, and it uses the expression-begin
context. It is extremely tightly coupled to the core syntax's implementation. And es_pipeline
is one of the simplest extensions; extensions like custom_templates
, 'flow', and jsx
are a lot more complicated and are tied into the core syntax in many, many places.
This is one reason why JS Custom bundles the core syntax in the package rather than referring to the original copy in the JavaScript package — a given version of JS Custom is tied to an exact version of the core syntax, any changes to the base syntax could break JS Custom, and in order to update JS Custom to a newer core syntax I have to run the JS Custom tests and possibly fix bugs. This is not as inconvenient as it sounds, because I'm also the primary maintainer of the core JavaScript syntax.
If you had a JS Custom-like system that used Microsoft's TypeScript syntax as a base, you would have to write all of the extensions completely from scratch. In addition, because the Microsoft syntax uses the old TextMate system, it may not be possible to port every feature you want. By comparison, adding TypeScript support to JS Custom would be tremendously easier and a lot less work. It would also produce higher-quality highlighting, because Sublime's core JavaScript syntax is of much higher quality than Microsoft's TypeScript syntax.
On the other hand, if you like Microsoft's TypeScript syntax, and you don't mind its bugs and quirks, and all you want is highlighting of a few custom templates, then you could hand-edit that syntax definition to add the specific features you want. I'm not sure how much the old TextMate system would be a limiting factor — I haven't touched that format in years — but it's probably possible to get a couple of custom templates working reasonably well.
Any update on this?
v2.4.0-alpha.2 is out with basic TypeScript support. It is probably buggy and missing important features, but it should be usable and — perhaps more importantly — it shouldn't break non-Typescript highlighting.
Please give it a try and report any issues in this thread. I expect to have time to finish this up over the next couple weeks, and any feedback would be invaluable.
I cloned the repo and checkout the typescript branch.
When sublime started I run JSCustom: Rebuild Syntaxes
,
and I expected to see Set Syntax: JS Custom - typescript
in the command palette,
but I see Set Syntax: JS Custom - react
and Set Syntax: JS Custom - default
How do i set the typescript syntax? :)
this is the output of the rebuild syn taxes command (if it is relevant):
Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to React.sublime-syntax.temp-34kizp. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/React.sublime-syntax.temp-34kizp)
[Succeeded in 3.81 seconds.]
Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to ~embed.sublime-syntax.temp-927x39. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/~embed.sublime-syntax.temp-927x39)
[Succeeded in 7.68 seconds.]
Building JS Custom.sublime-syntax.yaml-macros... (/home/predrag/.config/sublime-text/Packages/JSCustom/src/syntax/JS Custom.sublime-syntax.yaml-macros)
Compiled to Default.sublime-syntax.temp-_h1fml. (/home/predrag/.config/sublime-text/Packages/User/JS Custom/Syntaxes/Default.sublime-syntax.temp-_h1fml)
[Succeeded in 3.94 seconds.]
this ~embed.sublime-syntax.temp
looks suspicious(but I am probably wrong) :)
I haven't added a default configuration with TypeScript enabled. I'll probably add one into an alpha or beta build, maybe when I work on the documentation. For now, you can create a custom configuration with the typescript
option set to true:
"TypeScript": {
"file_extensions": [ "ts", "tsx" ],
"typescript": true,
}
The ~embed
configuration is a sort of hack. It's possible to use custom_templates
to embed JavaScript highlighting within template strings. However, this can lead to infinite recursion and strange errors in certain circumstances. The ~embed
configuration exists so that other first- and third-party syntaxes that embed the source.js
scope (like HTML or Markdown) won't run into this issue.
The temp-
suffixes are there because when JS Custom rebuilds an existing configuration, it replaces the old compiled syntax atomically by writing the new syntax to a temporary file, then replacing the old file with the new one. This prevents a Sublime-generated error popup if something goes wrong with the build.
Thanks for the clarification :)
@predragnikolic Have you had a chance to test out the TypeScript support? I'm not using TypeScript at work right now, so I'm sure there are bugs that I won't run into myself.
I am using typescript for only one personal project, that said I haven't used it that much. :)
I just noticed only thing that can be improved with conditional types
.
type TypeName<T> =
T extends string ? "string" :
// ^^^^^^^ is source.js.typescript variable.other.readwrite.js ... I thing it should be a different scope
T extends number ? "number" :
T extends boolean ? "boolean" :
T extends undefined ? "undefined" :
T extends Function ? "function" :
"object";
function foo<T extends string | undefined>(baz: T): T {
// ^^^^^^^ source.js.typescript meta.function.declaration.js meta.generic.js storage.modifier.extends.js
return bar
}
the extends keyword should be highlighted in conditional types
, but currently it is not:
If you don't mind, I can write on the ST discord channel, I am pretty sure that there are people(who use ts daily) and would like to try out the new ts support?
Personally, I found the JSCustom package really great and find it even more awesome that it wants to support typescript as well.
v2.4.0-alpha.5 should support conditional types.
Feel free to spread the word if you like. I think it's probably usable enough that more eyes on it would help.
v2.4.0-alpha.2 is out with basic TypeScript support
Thank you! I've given it a quick go with a couple of type-heavy files, comparing with VSCode, and it's quite solid! The only thing that stands out is multi-line arrow functions, which I have quite a few of.
I know that they are tricky to scope and it isn't possible to detect their parameters properly right now (right?), but it can probably be assumed that something coming after an :
in a scope like in the screenshot above is likely a type? Or is this not a safe assumption?
And shouldn't b
in the first line be scoped as variable.parameter.function.js
?
Arrow functions argument lists are tricky. The Sublime prereleases have a new branching feature that make them much easier and reliable to handle, but JS Custom is currently using the stable syntax for compatibility. Ideally, I'd like to provide TypeScript support for the stable release, but I haven't decided exactly how to handle arrow function argument lists.
There will definitely be a future-only version that uses branching to provide accurate arrow function parsing (including type annotations).
Looking forward to ST4!
A bit off-topic, but relevant: Is it okay to have both typescript
and flow
enabled in one config? Any downsides to look out for? Performance? Similarly, would I want to avoid using a TS-enabled config for highlighting JS-only files?
TypeScript and Flow would conflict; when it saw a type annotation, it would use one or the other. I wouldn't be surprised if the resulting behavior were glitchy.
Otherwise, there should be no harm in enabling Flow or TypeScript for vanilla-JS files. Performance shouldn't be an issue; Sublime's syntax engine is extremely fast.
v2.4.0-alpha.6 is out with support for namespaces, better object types, and various fixes and improvements.
Incidentally, I have just learned that the language specification linked from the TypeScript project repository, which I've been using as a reference, hasn't been updated in four years. No newer version exists, the TypeScript maintainers do not consider it a priority, and there are no plans to update it (https://github.com/microsoft/TypeScript/issues/15711).
I'm still hoping to get TypeScript support up to beta-quality within the next couple of weeks, but in the absence of a language spec, I expect to be chasing down bugs for months. I'll most likely have to release beta-quality TS support in stable builds for a while with appropriate warnings.
Not sure what your workflow is like, but maybe Babel’s tests could be of help somehow? Or some other part of the project.
v2.4.0-alpha.7 is out, implementing various features from TypeScript versions 1.9 through 2.8.
FYI, currently my workflow is going through the release notes in order (because some features are otherwise undocumented) with AST Explorer open in another tab (because the only authoritative reference for syntax is the parser itself).
v2.4.0-alpha.8 is out, which should be current up to TypeScript 3.9. Known missing features include:
Am I doing something incorrect here? I've cloned and checked out to tag v2.4.0-alpha.8
within $HOME/.config/sublime-text-3/Package
and I cannot use JSCustom.
I got it working now, as in installed, but it seems to not be playing nicely with TypeScript and React, here are my settings:
{
"defaults": {
"custom_template_tags": false,
"flow_types": false,
"jsx": false,
},
"configurations": {
"Default": {},
"React": {
"file_extensions": [ "js", "jsx", "tsx", "ts" ],
"typescript": true,
"jsx": true,
}
},
"embed_configuration": {
"name": "JS Custom (Embedded)",
"scope": "source.tsx",
"hidden": true,
"file_extensions": [],
"custom_template_tags": true,
"custom_templates": true,
"styled_components": true,
},
"auto_build": true,
"jsx_close_tag": true
}
Screenshot:
It doesn't seem to be using TypeScript at all. Try rebuilding your syntaxes.
I've rebuilt syntaxes and reinstalled JSCustom to no avail, this is in ST4 too.
This is the output TypeScript.sublime-syntax
it generates:
Can you please edit your comment so that it doesn't take 2 minutes to scroll it? You can use this syntax to paste your content: https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d
@rchl done, TIL.
Something odd is happening. According to your preferences, your only configurations are “Default” and “React” (plus the special embed configuration), so I'm not sure where that TypeScript.sublime-syntax
is coming from. (The syntax definition you posted does not have any of the TypeScript functionality in it, which explains why it's not working as expected.)
It looks like TypeScript highlighting is enabled in your React configuration. Rebuilding syntaxes should get rid of the extra TypeScript.sublime-syntax
file and ensure that the React configuration will handle TypeScript. If you've already rebuilt with the preferences you posted above, then I'm confused as to where that file is coming from. Is it at exactly Packages/User/JS Custom/Syntaxes/TypeScript.sublime-syntax
?
Can you post your preferences again, just to be sure?
I'm trying it out. Here is a little class with some oddities.
// Missing scopes for `implements` and `Animal`, and they throw off the rest of the class
export class Cat implements Animal {
public hasTail: boolean = true;
furPattern: any;
constructor(
mother: cat,
private father: cat
) { }
jump(height: number): void {
this.y += height;
}
}
Edit: Without the implements Animal
, everything looks good. Note that implements
can be followed by several comma-separated interfaces
@tsujp I think the Git tag may be wrong. (I've tipped off Thom in Discord.) Try checking out the tip of the typescript
branch and rebuilding again.
Just released v2.4.0-alpha.9, using the correct branch. Thanks.
Also, apparently I just never implemented implements
. Whoops. I'll fix that right away.
Fixed in v2.4.0-alpha.10.
Not necessarily your problem (and definitely not your fault), but Package Control does not handle the jump from a one-digit prerelease to a two-digit one correctly.
I think if you called this v2.4.0-beta.11
, it would work until you hit 100.
I dodged the problem by going to beta. The TypeScript stuff is now in master, and in v2.4.0-beta.1.
I'm currently checked out on the typescript
branch (commit 83a692864ced9d149896e7f88f8fbc694e2e1542
) with no TypeScript functionality on built syntaxes.
{
"defaults": {
"custom_template_tags": false,
"flow_types": false,
"jsx": false,
},
"configurations": {
"Default": {},
"React": {
"file_extensions": [ "js", "jsx" ],
"flow_types": true,
"jsx": true,
}
},
"embed_configuration": {
"name": "JS Custom (Embedded)",
"scope": "source.js",
"hidden": true,
"file_extensions": [],
"custom_template_tags": false,
"custom_templates": false,
},
"auto_build": true,
"jsx_close_tag": true
}
{
"configurations": {
"TypeScript": {
"file_extensions": [ "ts", "tsx" ],
"typescript": true,
"flow_types": false,
"jsx": true
}
},
}
I tried those user settings and they work for me. Are you seeing the name of the syntax as JS Custom – TypeScript
?
Hmm working for me now I purged everything again, but I don't have any highlighting on keys for enums for instance or variable names -- what's an easy way to add that?
Can you post example code?
what's an easy way to add that?
Install PackageDev > Open command palette > Edit Current Color Scheme... > Add an entry for variable.other.readwrite
?
Can you post example code?
Here's some screenshots, it's not highlighting prop types, consts, and enums etc with highlighting.
Here it is with TypeScriptReact
from TypeScript Syntax
:
Here it is with JS Custom using the above settings:
what's an easy way to add that?
Install PackageDev > Open command palette > Edit Current Color Scheme... > Add an entry for
variable.other.readwrite
?
Will give this a shot too, thanks.
That const poolReducer = (state: any, action:any): any => {
looks like a bug. The other stuff looks good to me.
I've implemented the string literal enum members, and they'll be in the next build. (I can't find them in the documentation, but AST Explorer confirms that they're valid syntax. Thanks for nothing, Microsoft.)
The arrow function highlighting is a bug. An upcoming Sublime feature will allow me to completely rewrite the arrow function code to work much, much better. I'll probably leave it be in the meantime unless it's really egregiously broken.
Is there anything wrong with the interface highlighting? The only difference I see is that the other syntax highlights the colon as an operator whereas JS Custom marks it as punctuation, and that the name of the interface gets an entity.name scope. In both cases the JS Custom behavior is intended.
It looks like the other syntax is highlighting variable names at the time of declaration. I actually like this behavior, but it's not what the core JavaScript syntax does, so JS Custom doesn't either. You may have success adding a highlighting rule for meta.binding.name
, which should work even with destructurings and so forth. Let me know if this doesn't work or isn't what you're looking for,
Is there anything wrong with the interface highlighting?
Not in Jordan's code, but if you use export interface foo
, there is.
The other thing I've recently found that's not working is optional fields.
export interface Employee {
// ^ breaks here
name: string;
salary?: number;
// ^ breaks here (if `export` is fixed or removed above)
}
Edit: I haven't said so here yet, but I'm very grateful for the work you've put into TypeScript and I've been encouraging people to try it out.
I've fixed export interface
(and export namespace
), and they'll be in the next build.
What's the issue with optional fields? The question mark should be highlighted as storage.modifier
, and otherwise it should look like a non-optional field.
Oops. My fault. Optional fields should also be available in class
es. Serves me right for not testing the snippet in question.
Fixed for next build.
v2.4.0-beta.2 is out with the above fixes.
@Thom1729 does the current beta support for TypeScript work with custom_templates
?
Like it says. I'm not sure how much of an effort this would be.
Known bugs/missing features
Old-style castingImplemented in beta.3(<Foo> bar)
. This conflicts with JSX, so I'll have to make sure that this is configurable.