Closed dummdidumm closed 4 years ago
So none of them are using Svelte parser?
svelte2tsx strips out style/script tags and then uses the svelte compiler to parse the htmlx part. svelte-ts uses the svelte compiler as a second step. The current language server does not use the svelte compiler for parsing. I'll update the initial post.
I wasn't sure if I should share this, but some part of it might be useful. Last year (July/August 2019) I started playing around with typechecking Svelte, knowing that I was doing throwaway work just to learn. It supports some unknown amount of a full solution, maybe 30%, and it has some limitations (more on those later). There aren't any tests or external documentation because I've always considered it to be a dead end.
Svelte compilation goes as it does today with a non-typechecking TS preprocessor. If you wanted types in your HTMLx markup, not just your script tag, you'd need a preprocessor to handle that, or Svelte would need to handle TS internally.
The typechecker is a separate build step that uses the un-preprocessed TypeScript code from script
blocks and the markup AST from svelte.compile
to create a virtual Svelte TypeScript file solely for typechecking purposes. (here's where the Rollup plugin creates the virtual file and then typechecks it) It's similar to @halfnelson's strategy, but different in that it produces normal TS, not TSX, so in the markup it doesn't try to typecheck HTML tags and attributes, only the stuff inside mustache tags and Svelte components. It transforms Svelte components into constructors that get new
ed with their props. I got partway through supporting a lot of Svelte's constructs before realizing I got carried away more than I intended, and further progress should be Serious Engineering.
The TypeScript compiler API is used to construct a program in memory and typecheck the virtual Svelte TS. Unlike Svelte its view of the world is whole programs, not individual files. At the time it seemed like it would be straightforward to add incremental compilation and worker threads as future optimizations.
The virtual Svelte TS file is generated with a sourcemap which is then used to map TypeScript diagnostics back to the original Svelte source. It was pretty cool to see type errors point back to the Svelte source but I had no confidence in my approach overall.
One problem I remember that I couldn't figure out was with diagnostics that weren't specific enough in some cases. IIRC an example was with typechecking Svelte component constructor props - VSCode somehow would point to individual prop errors, but the TS diagnostics I was getting were pointing only to the constructor, not the problem property.
If you look through the code be aware that parts are a mess and some comments may be outdated. I originally used magic-string
and at some point changed over to source-map
. And remember it was a doomed endeavor from the start! Note that all of this is on the ts
branch of that repo, not master
.
Here are the 4 files referenced above:
I did some digging in the source code of the current svelte language server. Here are my thoughts on the typescript part of it:
The language server broadly works likes this:
DocumentManager
handles all documents that get opened.TypescriptPlugin
registers at the DocumentManager
to get invoked on language server events ("doHover" etc). It does not register directly: It gets wrapped with wrapFragmentPlugin
which makes sure TypescriptPlugin
only gets the content inside the script
tag. Mapping of positions (of mouse/text cursor) is done in wrapFragmentPlugin
to ensure the infos are shown at the correct position.TypescriptPlugin
uses service
, a typescript language service wrapper. It basically delegates all events to this language server, with some mapping to adhere to the method return types.service
takes the document to do work on aswell as a createDocument
method from TypescriptPlugin
. service
keeps his own map of documents which are at the moment basically only a delegate to the document of the DocumentsManager
. Everytime service
is invoked from TypescriptPlugin
, service
updates its documents map with the given document (after wrapping it). createDocument
would be invoked if the language service opens a document that is not yet part of the documents map - but this will never happen since typescript language service does not find other svelte modules (see next section)..ts
/.js
files this works, for .svelte
files it does not. Reason: The typescript language service does not know about the .svelte
ending, so it just assumes it's a normal typescript file and searches for files like ../Component.svelte.ts
, which is wrong. In order to fix this, we need to implement the method resolveModuleNames
on LanguageServiceHost
and reroute all .svelte.ts
file lookups to .svelte
. Then the language service will walk all svelte files. NOTE: In order to implement this, we also need to fix the following bug: Since the typescript language service now would walk all files, it would invoke createDocument
from TypescriptPlugin
. But this currently returns the whole document, not only the content inside the script tag. This is a bug inside the wrapFragmentPlugin
not wrapping openDocument
.TypescriptPlugin
(and others) get its document - maybe both extraction of the script tag and preprocessing should be done in one step. This would mean changing or fully replacing the wrapFragmentPlugin
. Another argument for reworking that is that, in order to correctly determine things like "unused function", we have to take the html part into account, so the script tag is not enough. Thoughts?
On resolveModuleNames
, here's how a TypeScript team member did it for Vetur, and here's a similar version I did for Svelte, which includes a comment that mentions an alternate strategy that worked, but copying Vetur seemed better.
@dummdidumm do you mind acting as tech lead and breaking these things out to specific issues? i would be happy to volunteer dev time, but i find alot of this overwhelming and would work better in small, defined issues. Unless there is some big architectural change that needs to be done (looks like no?)
Basically just open up a bunch of issues and let people claim them and manage them as our informal tech lead on this. I'd offer to do it but honestly just havent felt the pain yet haha
also linking to orta's original LSP post for others who are browsing https://github.com/sveltejs/svelte/issues/4518
Sure I would love to help on this, but right now I'm just a random dude who is very excited about the language server, digging into the code and making some suggestions 😄 I also need some more time to get familiar with the code, its architecture and lsp in general. But I'm definitely willing to push stuff forward. @orta since you are the maintainer of this repo, what do you think about this?
👋 Yeah, we both are random people who just care - I don't even have any svelte codebases to trigger my "this should be better" itch. So, I'm very happy to see any movement at all - @dummdidumm - push away and I'll back you up and try make sure everything runs smoothly
Ok great! I will go ahead and make some PRs restructuring some of the code and preparing it so we can then parallelize working on individual items.
Maybe we could combine @ryanatkn, svelete-ts's approach and eslint-plugin-svelte3's variable approach:
injected = true
into instance ts source or ts AST and then use it with source map to provide type check and auto-completion. The script can have three fragments: module <script context="module">
, instance and mustache (template). The mustache part could just be all dump into a template.js for now. But for the long term we could reflect template into tsx or plain ts like ryanatkn's approach.
Module vars, where module = true
, would be injected into template.js and instance but not the other way around.
In this way we don't have to re-implement on how to find reactive variable $: a = 1
or $-prefixed store in typescript AST. These variable injected by svelte compiler derive from top-level statement and we're transpiling it into latest ES. So it should mostly not be rename by typescript.
About reactive variable $: a
it could be written like this:
$: a = 1
let a: number
$: a = 1
or
$: a = someFunction(b) as Type
All it's valid ts that doesn't need transform before transpile
And $-prefixed store we can create a generic function like svelte/store get to extract store type
/** injected */
let $store = getType(store)
- Use svelte preprocess to preprocess script into js. Here we could just tranpile to latest or next version of ES to do as little transformation as possible.
- Let svelte compile the preprocessed source.
- We can then inject result vars where
injected = true
into instance ts source or ts AST and then use it with source map to provide type check and auto-completion.
I think svelte-ts does something like that. Seems like a good idea. On injected = true
: What code exactly has this property?
The script can have three fragments: module
<script context="module">
, instance and mustache (template). The mustache part could just be all dump into a template.js for now. But for the long term we could reflect template into tsx or plain ts like ryanatkn's approach.Module vars, where
module = true
, would be injected into template.js and instance but not the other way around.In this way we don't have to re-implement on how to find reactive variable
$: a = 1
or $-prefixed store in typescript AST. These variable injected by svelte compiler derive from top-level statement and we're transpiling it into latest ES. So it should mostly not be rename by typescript.
So you want to split up the file into a virtual .ts
-file and a .template
-file? Why not have the the mustache-parts on the bottom of the .ts
-file? This way we also would get rid of "unused method"-warnings. On walking the AST: Yes, I think we can be smart here and skip looking into the AST too much because everything that is referenced in the template must be top level in script.
About reactive variable
$: a
it could be written like this:$: a = 1
let a: number $: a = 1
or
$: a = someFunction(b) as Type
All it's valid ts that doesn't need transform before transpile
Is it possible to infer the type? Maybe this is just something we leave as is and if the user wants to use typescript, he has to define let a: number
himself. As an alternative, we could insert let a;
, but then it would be any
.
And $-prefixed store we can create a generic function like svelte/store get to extract store type
/** injected */ let $store = getType(store)
Yeah this seems nice. Another idea I had was to construct getters/setters.
I think svelte-ts does something like that. Seems like a good idea. On
injected = true
: What code exactly has this property?
The compile result has a property vars, which is an array of object that has the following property: name, export_name, injected, module, mutated, reassigned, referenced_from_script and writable you can see svelte compile api for more info
So you want to split up the file into a virtual .ts-file and a .template-file? Why not have the the mustache-parts on the bottom of the .ts-file? This way we also would get rid of "unused method"-warnings.
This is the approach of eslint-plugin-svelte3. Also I just thought if we want to support ts in template because there is no attribute to specify language.
Is it possible to infer the type?
Since this virtual source is only use for type check and autocomplete, I presume it can achieve by just copy the first assign statement to initialize it but it seems to be need more work.
Is it possible to infer the type? Maybe this is just something we leave as is and if the user wants to use typescript, he has to define let a: number himself. As an alternative, we could insert let a;, but then it would be any.
Looks like normal inference doesn't get us all the way there. (did you mean compile-time inference? not sure if that could work)
let a; // type is "any" initially
$: a = 5;
a; // type is now "number", good
const cb = () => a; // type is implicitly "any", noo
The reason is that in the linear code flow, TypeScript can see a
wasn't reassigned, but nested function scopes don't have that guarantee, and TypeScript errs on the side of type safety.
Exactly, these were my worries with that. But as @jasonlyu123 pointed out, we could just copy the definition (everything after the =
).
Before
$: a = true;
After
let a = true;
$: a = true;
But I don't know if that's feasible for all circumstances. Especially with large objects where someone implements an interface, this will not infer everything as the user might expect. Also, the copy-pasta could get pretty big. So I'm still more on the side of "let the user type it".
Thanks for clarifying, I misunderstood what @jasonlyu123 said. It fixes the nested function scope issue I mentioned.
Is there an example of when that's not good default behavior? It seems like it does the ergonomic right thing most of the time, and other cases can be fixed manually. Here's some examples.
Mhm I guess you're right. It is the most ergonomic thing. What I worried about is if the inferred type is wrong, for example property is string
instead of a defined union of strings ('a'
| 'b'
). But as you said, these cases are not that common and then the user still can type it manually.
There's a part of me that's like: "great, the explicit let syntax works awesome" but that would end up not working with JS support out of the box
What about just replace first label to let:
$: a = 1
$: a = 2
get transform to
/** injected */
let a = 1
$: a = 2
I rethink a bit and can't find the advantage of "copy definition" over "replacing $:
with let
". Can you guys think of any disadvantage of replacing $:
with let
?
About inferred type difference to what user expected, can we provide a refactoring actions to convert it to manual define? This action will prompt user to manual type the variable like this, Or refactor to its inferred type if possible.
before
$: a = 'const1'
after
let a : AskUserForType
$: a = 'const1'
Also thinks for the things you point out, Learned something 👍
Ha, smart idea, just entirely replace the $:
through a let
. I also don't know any situations where that would end up bad.
We also could think about always doing the transformation, not only for the first label. Then there would be a "cannot redeclare"-error which hints the user at "woops, I defined this variable twice" - or are there situations where this would be viable and not a mistake?
Nice, I can't think of any reason replacing the label would fail either. (since break $
doesn't ever work inside assignments, right?)
For redeclaring the same reactive variable, it looks like Svelte allows this and it works in order like you'd expect. (see this repl example) I don't personally see myself using this pattern intentionally and would want the error, but it's in the territory of interpreting Svelte's semantics so don't take my word for it!
OK so redeclaring it is a pattern that actually works. I would say we better allow it then - or make a "redeclare not allowed"-feature flag later which is off by default.
@halfnelson has created a language server based off his svelte2tsx approach in https://github.com/halfnelson/svelte-type-checker-vscode . I really love how good this all works already, but I still would prefer to not use tsx as an intermediate output because I fear it could be a little too slow and have some limitations - but that's just my gut feeling. Still, we could take a lot from svelte2tsx: How the flow of parsing-compiling-transformation is done, and also the extensive test suite. How do others see the approach of transforming the code to tsx?
I though about how other transformations could look like. Here are some examples. Feedback welcome.
Original:
<script>
import { createEventDispatcher } from 'svelte';
const dispatch = createEventDispatcher();
export let todo: string;
function doneChange() {
dispatch('doneChange', true);
}
</script>
Converted:
export default class Component {
constructor(props: {todo: string; 'on:doneChange': (evt: CustomEvent<boolean>) => void}) {}
}
import { createEventDispatcher } from 'svelte';
const dispatch = createEventDispatcher();
export let todo: string;
function doneChange() {
dispatch('doneChange', !todo.done);
}
Explanation:
Somehow get all all props (easy) and dispatched events (hard) and add them to one big props object. Adding events with on:
-prefix should minimize collisions and make autocomplete suggestions easier to implement.
(in the following examples, component-definition is omitted for brevity)
Original:
<script>const bla = {bla: 'bla'};</script>
<p>{bla.bla}</p>
Converted:
const bla = 'bla';
const _bla = bla.bla;
Explanation:
Add arbitrary const
elements, maybe prefixed with some obscure unicode character, to check correct usages of variables and to also make the "unused"-warning go away. Maybe we then need to think about how to get rid of the unused _bla
-warning ...
Original:
<script>function bla() {return true;}</script>
<button on:click={bla}>bla</button>
Converted:
function bla() {return true;}
const btn = document.createElement('button');
btn.addEventListener('click', bla);
Explanation:
When using native event listeners, create the corresponding element and then add an event listener. This way we take advantage of the addEventListener
-typings which have overloads for pretty much every listener out there: .addEventListener('click', ...);
-> event is of type MouseEvent
. This pattern would be used after having checked if the element is another svelte component which has the event defined (also see next example). Shortcomings: What if Svelte is used in a non-web-environment (Nativescript)?
Original:
<script>
import Bla from './bla.svelte';
function blub() {}
</script>
<Bla bla={1} on:blubb={blub} />
Converted:
import Bla from './bla.svelte';
function blub() {}
const bla = new Bla({bla: 1, 'on:blubb': blub});
Explanation: Every component gets a class definition, so we just instantiate that class.
Original:
{#each todos as todo,i (todo.id)}
<p>{todo.text}</p>
{/each}
Converted:
todos.forEach((todo, i) => {
const _todoText = todo.text;
});
Explanation:
forEach
seems like the easiest to convert to. Inside, we recursively can apply all other features.
Original:
{#if x === 1}
<p>if</p>
{else if x === 2}
<p>elseif</p>
{else}
<p>else</p>
{/if}
Converted:
if (x === 1) {
} else if (x === 2) {
} else {
}
Explanation: Pretty self-explanatory.
svelte:x
componentsuse:action
, transition
, bind
What do you guys think? Are these transformations viable and doable? Did I miss something?
About component class I would prefer extends svelte's own SvelteComponent class, but i don't know if this is easy to implement or not
import { SvelteComponent } from "svelte";
export default class Component extends SvelteComponent {
constructor(options: ComponentOption<{ propA: string }>) {
super(options)
}
$on(
event: 'input',
callback: (event: CustomEvent<string>) => void
): () => void
$on(
event: 'click',
callback: (event: CustomEvent<number>) => void
): () => void
$on(
event: string,
callback: (event: CustomEvent<any>) => void
): () => void {
return () => { }
}
}
and import these interface
interface ComponentOption<TProps, TSlot = undefined> {
target: Element,
props: TProps | SlotProp<TSlot>
}
type SlotOption<T> = [unknown, unknown, unknown]
interface SlotProp<TSlot> {
$$slots: Record<string, SlotOption<TSlot>>
}
Noted $$slots
is probably svelte's internal api, I just dig it from compiled code
The reason for this is that it can then be use to emit .d.ts
that could be use to type in vanilla ts/js.
Also because, although weird enough,this is valid syntax:
<script>
onMount(() => {
new Component({
target: document.getElementById('foo')
})
})
</script>
Transition
, use:action
and other element directives could be transform like
fade(null as Element, { } /* option in attribute*/)
{#await promise}
<p>...waiting</p>
{:then number}
<p>The number is {number}</p>
{:catch error}
<p style="color: red">{error.message}</p>
{/await}
to
promise.then(number => { number })
promise.catch(error => { error.message })
<script>
let a;
</script>
<div bind:this={a}><div>
to
let a;
onMount(() => {
a = document.createElement('div')
})
needed to wrap it in callback because it's not immediately available
This is a bit tricky
<script context="module">
let count = 1;
</script>
<script>
import _ from "lodash"
let b;
</script>
should be compile to
import _ from "lodash"
let count = 1;
// any block is ok
{
let b;
{
And also what if the user somehow use different language in two script, like js in module and ts in instance?
Nice, looking good. For "Use variable in template" const _bla = bla.bla;
could be just bla.bla;
letting TypeScript do its check without needing to filter out the "unused var" diagnostic. I can't think of any corner cases where this doesn't work.
Some of the Vue team are exploring a typescript language server plugin - https://github.com/znck/vue-developer-experience
It's an interesting angle as today these experiences go from outside TypeScript and in (like vetur does, as it runs it's own tsserver-ish, as well as html/css/etc LSPs) or do you try go from inside TypeScript and work outwards (e.g. this proposal) where you manipulate the TSServer to effectively believe that the .vue files are legitimate TypeScript by masking non-TS code.
Though today it can only work by having patches to typescript
That's an interesting approach. But how would features like autocompletions for html or css work then? Would one have to implement this all "by hand" and not rely on the out-of-the-box-language-servers anymore, or am I missing something?
Do you see any advantages of doing one over the over?
Anything but the JS/TS support would be handled by a separate vscode LSP. It would be like if this extension removes all support for JS/TS and only handled the rest, then the typescript plugin would handle the rest.
Advantage wise you get all of the TS tooling "for free" in this case, jump to symbol, file mappings etc
I'm probably not recommend it, today it requires forks or patches of TypeScript to work which is a pretty high barrier to entry - and the TypeScript team is still not sure if/when compiler plugins could have that kind of access.
Thanks for the insights. I would say we with the current approach then (outside-in).
If we go with the "make a virtual ts file"-approach discussed above, how would we implement this and keep source mappings?
Today, it's easy because it's just "find start of script, find end of script", "extract script part" and for mapping "add offset of svelte file to positions". If we change the script part now, we need a more sophisticated mapping:
Original
<script lang="typescript">
let a = 1;
$: b = a + 1;
</script>
<p>{b}</p>
Mapped:
export default class Bla extends SvelteComponent { ... } // <- prepended, no mapping needed as far as I know
let a = 1; // <- untouched
let b = a + 1; // <- modified in place. How to map? Do we need to adjust all following code positions now to have a new offset/position?
b; // <- appended. Needs mapping from {b} inside svelte-mustache-tag to here. We can get the original position from the html ast which is part of the output of svelte.parse
Any ideas? I never did source mapping stuff before..
Some of the Vue team are exploring a typescript language server plugin - https://github.com/znck/vue-developer-experience
It's an interesting angle as today these experiences go from outside TypeScript and in (like vetur does, as it runs it's own tsserver-ish, as well as html/css/etc LSPs) or do you try go from inside TypeScript and work outwards (e.g. this proposal) where you manipulate the TSServer to effectively believe that the .vue files are legitimate TypeScript by masking non-TS code.
Though today it can only work by having patches to typescript
The remove TypeScript patch PR is merged. https://github.com/znck/vue-developer-experience/pull/7
So ... @orta this means the disadvantages regarding "unofficial" are no longer the case? Or is it still kind of unofficial, just not through a patch but through some package.json
config you have to know of and could change at any time?
Another topic: At the moment I feel we still are in discussion on how to approach this best, and if we decide, then it's going to take some time to implement it. What do you guys think of using svelte2tsx
in the meantime? It could serve as a very good starting point, we could get some tests around it and then we either see "ok this actually works great, let's not change it" or gradually migrate away from it. Thoughts?
Aye, there were two ways it was editing TS's default flow, It's still patching TypeScript - but at runtime by replacing TS' functions. I think it's reasonable, given plugin development isn't on the roadmap yet.
I'd be a bad core team member if I recommended it though ;)
Trying out svelte2ts
sounds like a good avenue for exploration!
You can see what it would be like with svelte2tsx using this vscode plugin: https://marketplace.visualstudio.com/items?itemName=halfnelson.svelte-type-checker-vscode together with Svelte Beta (works best if you disable svelte beta's typescript options so you don't get double the hints :) )
Even if not using svelte2tsx, I see discussion on which transforms need to be made to svelte's JS code to keep typescript happy. The test suite for svelte2tsx covers not only changes needed to be made to the template, but also the script tag, which if you are going for a script tag only solution might be a good starting point: https://github.com/halfnelson/svelte2tsx/tree/master/test/svelte2tsx/samples
the htmlx2jsx
is the folder the template change tests (htmlx), while the svelte2tsx
samples are the svelte specific changes needed for svelte's special abuse of js syntax :)
Just discovered this repo https://github.com/pivaszbs/svelte-autoimport . Maybe worth looking into if we want import intellisense independent of svelte2tsx
.
I think we're at a point where most of the general direction for this project is sorted and seems to be working out neatly. I'll give this a week for feedback, but I'm pretty happy that we don't need to re-think it architecturally now.
Seems noone has objections, so I'm gonna close this one. Thanks everyone for discussing this!
This thread is intended for an overview of approaches on how to implement a language server for svelte files. Discussion on a prefered solution can be done here aswell.
Current state of the language server
Currently syntax highlighting and some basic autocompletion works for svelte files. Rich intellisense however is not provided, does not know about all the typescript files in the workspace, and also has no type information about the svelte components. The same is true for the html part where special svelte syntax is highlighted but intellisense is not working. The vscode-html-languageservice is used which does not know about the special syntax by default. The current language server does use the svelte compiler for parsing files and getting diagnostics like "no alt attribute on this img tag". A more indepth analysis: Comment
Existing approaches/solutions
Disclaimer: I'm not the author of any of the existing solutions, I did look at the code however. If some of the information is wrong or not detailed enough, or you are missing a solution, feel free to comment, I will adjust it.
https://github.com/alexprey/sveltedoc-parser
Uses htmlparser2 to parse the html part of a svelte file. Uses hooks of the html parser to add custom parsing and extract relevant information. Uses espree to parse the script parts of a svelte file. Then walks the AST to extract relevant information. Provides the results in a JSON-Format.
https://github.com/ArdenIvanov/svelte-intellisense uses it to parse svelte files. It uses the result to attach some additional behavior.
Since this uses a javascript parser, it will not work with typescript.
https://github.com/halfnelson/svelte2tsx
Uses svelte's compiler to parse the HTMLx part of a svelte component. Then transforms the HTMLx plus the original script part to a tsx file with a self-written transformer.
https://github.com/simlrh/svelte-language-server/blob/feature/extend-ts-support/src/plugins/ts-svelte/service.ts (a fork from svelte-language-server) uses it to create shadow-tsx-files of svelte-files. It then uses typescript's language service, but proxies the requests: It replaces the original file path with the file path to the generated tsx-file. The typescript language server therefore checks the generated tsx files. Results are transformed back to get correct document positions.
https://github.com/halfnelson/svelte-type-checker-vscode is another language server using the svelte2tsx-library.
https://github.com/marcus-sa/svelte-ts
Description taken from implementer's comment:
Compiler:
It consumes the exported variables and functions from inside a Svelte script, and then generates it as a declaration files which contains a class that extends the runtime SvelteComponent.
Typechecker:
Other approaches
See comments of this thread.