Serve precompiled JavaScript to the browser

KSXGitHub commented 3 years ago

Reason

It would be nice to be able to directly import TypeScript modules from the browser without having to use deno bundle.

Suggestions

Every TypeScript should be compiled once to JavaScript.
If User-Agent is Deno/<VERSION> or Accept contains application/typescript, serve TypeScript code. If User-Agent is one of the browsers and Accept is */*, serve JavaScript code. If Accept contains text/html, serve HTML document. Otherwise, serve TypeScript code.

kt3k commented 3 years ago

jsejcksn commented 3 years ago

or Accept contains application/typescript, serve TypeScript code

(Not starting a media type debate, but just wanted to note that) there are other media types for TypeScript which should be considered:

KSXGitHub commented 3 years ago

Deno treating video/* media types as TypeScript is weird to me.

jsejcksn commented 3 years ago

Deno treating video/* media types as TypeScript is weird to me.

@KSXGitHub It's an artifact of multiple file formats sharing an extension (and historically being served without the [correct] media type).

KSXGitHub commented 3 years ago

@jsejcksn

I have created https://github.com/denoland/deno_website2/issues/1732, you may continue the discussion there.

KnorpelSenf commented 3 years ago

How could this be implemented?

I assume it would rely on a deno script the uses Deno.emit under the hood. Is that correct? If so, this is related: https://github.com/denoland/deno_emit/issues/44
What is the supporting infrastructure? Would it run on the same nodes that execute deno doc?
Is there caching involved? If so, does it re-use the caching infra of doc.deno.land?

I would like to see this feature implemented, and I am willing to PR if this won't be too time-intensive. I just don't know too much about how you'd like to see this designed :)

dsherret commented 3 years ago

This could just type strip or perhaps also transform down to ES6 for compatibility reasons (or maybe some query parameter to specify the target). Unfortunately I don't know anything about the current website's design to provide any guidance on implementing it and I'm not sure what's currently desired for the implementation.

KSXGitHub commented 3 years ago

This could just type strip or perhaps also transform down to ES6 for compatibility reasons (or maybe some query parameter to specify the target).

For now, compiling TypeScript to the latest version of ECMAScript that is supported by all major browsers is enough. Transforming it to ES6 may cause unnecessary performance regressions.

rojvv commented 2 years ago

This would be awesome.

Ciantic commented 2 years ago

Side note, I would not make that difficult tango with User-Agents, I would just serve JS files with .js extension, simple and obvious. Either way I support this.

I think this would make deno.land/x way more usable outside Deno too. In a way this would make Deno a better citizen in vanilla JS context.

For instance sometimes I find a nice Deno library, and because most Deno code follows standards, I could import it in the browser. But I can't because deno.land/x/ does not provide JS files, it only provides TS files, e.g.

<script type="module">
import { pathToRegexp, match, parse, compile } from "https://deno.land/x/path_to_regexp@v6.2.0/index.js"
</script>

Does not work, because https://deno.land/x/path_to_regexp@v6.2.0/index.ts is just a TS file. Then I have to jump hoops to get it in my adhoc script tags.

Third Party Modules library could just compile all TS files to JS files when updated, then they would be usable in the browser as well.

As a bonus, this would make a cool talking point: deno.land/x is the place to get all your ES modules! Everything is an ES module! (Insert Stan S. Stanman here)

Why I like script tags sometimes? They are simple, and for instance I would not like to use TS and all the tools it require for simple demonstration snippets in a blog, I'd just use <script /> tags. Also I write WordPress stuff, and it's horrible as is, I don't want to add any extra tools usually to that workflow, so I just add script tags.

Allowing to use whole deno.land/x in script tags would dramatically increase the knowledge retention:

When I want to do good job I use Deno the tool and TypeScript files
When I don't care, I can use the same imports I used in TS files, but inside <script> tags

vwkd commented 2 years ago

This could also be solved by a more general Web service that transpiles a TS file to JS on the fly, like any XtoY.com/convert?url=…&setting1=…&setting2=… service.

To make this easier, such a proxy worker could be offered by Deno and the second URL integrated into the deno.land/x site to be shown next to the direct URL, maybe in a GitHub-style repo clone drop-down. The worker can simply always transpile without any decision making between JS and TS, because the user explicitly opts in to JS using the new URL. Any impacts to performance and complexity is contained to the proxy worker and don’t impact the current URLs and can evolve independently. It could for example offer additional settings in query parameters, like compile target, etc.

KnorpelSenf commented 2 years ago

I created a service called streif that transpiles TS on the fly to JS. You just need to prefix your URL with https://streif.deno.dev/ and then it will generate code that can be used in the browser.

Code: https://github.com/KnorpelSenf/streif

Disclaimer: I did not actually write any Rust before, and I also only spent a day on this project, so everything is super hacky and bad so far. I am aware of modules that do not work, and https://github.com/denoland/deno/issues/14879 is an annoying problem for this plan, but I still think that this is a good proof of concept. Most importantly, I am very happy about contributions! :)

ayame113 commented 2 years ago

This could also be solved by a more general Web service that transpiles a TS file to JS on the fly, like any XtoY.com/convert?url=…&setting1=…&setting2=… service.

I have considered the case where deno.land/x is imported deep in the dependency. In that case, we can use TypeScript in browser by replacing the URL using import-map.

{
  "imports": {
    "https://deno.land/": "https://XtoY.com/convert?url=https://deno.land/"
  }
}

However, as the number of module registries other than deno.land increases, so does the size of the import-map.

{
  "imports": {
    "https://deno.land/": "https://XtoY.com/convert?url=https://deno.land/",
    "https://deno-modules.com/": "https://XtoY.com/convert?url=https://deno-modules.com/",
    "https://awesome-registry.com/": "https://XtoY.com/convert?url=https://awesome-registry.com/",
    "https://anything-registry.land/": "https://XtoY.com/convert?url=https://anything-registry.land/",
    ...
  }
}

I like the idea of Deno's decentralized module registries, so I'd like to vote for adding a UserAgent-based precompile feature to deno.land so that people can use the various module registries without hesitation.

vwkd commented 2 years ago

@ayame113

I don’t think editing existing code by replacing direct URLs with transpiled URLs is the main use case. Instead, it’s probably writing new code by copy pasting the transpiled URLs. Therefore one doesn’t need an import map. Also import map would prevent using any JS file from that registry because it would map every URL of that host regardless of the path.

Not sure I understand your argument. How would integrate transpilation to deno.land help with any other registries? Instead they each need to implement such a feature themselves. While a standalone transpilation service could be used for any registry and not just deno.land.

KnorpelSenf commented 2 years ago

I agree that import maps won't solve the problem, for the reasons mentioned above. For streif, I implemented import/export statement transpilation in Rust (wasm) that prefixes all URLs automatically. That way, you can import a module through it, and all dependencies will be transpiled through it, as well. (I think there's currently a bug that this does not always work, but I'll try to fix it in the coming days, contributions as always welcome.)

ayame113 commented 2 years ago

Therefore one doesn’t need an import map.

For example, my module imports the Deno standard library internally, so I thought it wouldn't work unless I replaced the import statement with an import map here. In addition, I wanted to say that n module registries would be confusing if they supported on-the-fly transpiling in n ways.

https://streif.deno.dev/https://deno.land/x/jsonlines@v1.2.1/src/parser.ts

However, as @KnorpelSenf said, it seems that the import map will no longer be needed once the ability to rewrite the import statement is added to streif.👍

Due to performance and security and the complexity of using two different URLs, I still want denoland to have on-the-fly transpiling, but I think it's a big step forward if a service like streif works.

KnorpelSenf commented 2 years ago

@wojpawlik pointed me to https://bundle.deno.dev/ by @johnspurlock which looks more mature. It takes a different approach, because it downloads all modules server-side and bundles them up before shipping them to the browser. This creates the overhead of having to bundle things, but that is probably rather advantageous keeping in mind that the browser can load all JS with a single request, rather than one request per module. I'm considering to drop streif again, unless someone can give a good reason why this approach should be pursued further.

Ciantic commented 2 years ago

@wojpawlik This creates the overhead of having to bundle things, but that is probably rather advantageous keeping in mind that the browser can load all JS with a single request, rather than one request per module.

Neat tool, I just think these external bundlers have problem, do they exist five years from now? If they get popular the bandwidth costs etc. start to stack up. It would be nice to have official way to get JS files from deno.land directly.

Bundling as one JS file should be optional, sometimes I want just couple of neat functions from functional libraries and not all.

KSXGitHub commented 2 years ago

Bundling as one JS file should be optional, sometimes I want just couple of neat functions from functional libraries and not all.

My wish was and still is for deno.land itself to serve the JS files to make it convenient for quick prototyping as well as quick experimentation in a browser's console (imagine just copy a link and paste it after fetch( or import from "!). I don't wish for deno.land to serve production ready code. Hence, while having bundling as an additional feature is certainly nice to have, I don't wish to place additional burden on Deno developers.

johnspurlock commented 2 years ago

This creates the overhead of having to bundle things

True, but bundle.deno.dev will send back "cache forever" headers for "pinned" urls (assumed immutable based on commit hashs or release tags), so browsers will only fetch them once. Once Deno Deploy supports edge caching, I'll be able to cache bundle output on the edge itself for these pinned urls without having to re-call bundle at all.

do they exist five years from now? If they get popular the bandwidth costs etc. start to stack up

This is a good point. I'm committed to hosting this as a service, on edge platforms like Deno Deploy to minimize headaches and costs.

sometimes I want just couple of neat functions from functional libraries and not all.

bundle.deno.dev can point to any file that exports items, it does not need to be the top level package module.

e.g. https://bundle.deno.dev/https://deno.land/std@0.145.0/datetime/formatter.ts

However, it's true that it if that file has deps, those will be included in the output.

You prompted me to create a new endpoint called transpile.deno.dev that only does transpilation of the given file, no bundling of deps

e.g. https://transpile.deno.dev/https://deno.land/std@0.145.0/datetime/mod.ts

Hope that helps! - John

johnspurlock commented 2 years ago

[*] Also, each call to bundle.deno.dev and transpile.deno.dev includes Server-Timing response headers, so you can check out how long each part of the bundling process took in dev tools -> network > timing (modulo how accurate Date.now() is on the edge platform)

Once Deno Deploy supports edge caching, for example, you'll be able to tell which responses actually called bundle vs served quickly from the cache.

KnorpelSenf commented 2 years ago

@johnspurlock your service transpile.deno.dev should rewrite import statements to also transpile them. Otherwise a file with external dependencies cannot be used with your service.

johnspurlock commented 2 years ago

your service transpile.deno.dev should rewrite import statements to also transpile them. Otherwise a file with external dependencies cannot be used with your service.

Good point, I agree!

Pushed an update that rewrites remote .ts urls when in transpile mode

To see it in action, start a new codepen: https://pen.new

Use these contents in the html section:

<!-- transpile.deno.dev example -->
<script type="module">
import * as mod from "https://transpile.deno.dev/https://deno.land/x/composium@v0.0.1/mod.ts";
console.log(mod);
document.body.textContent = 'Module exports: ' + Object.keys(mod).join(', ');
</script>

In dev tools, note all .ts files (even the remote ones referenced via deps.ts) are served individually as js

KnorpelSenf commented 2 years ago

Awesome stuff. I'm glad someone finally did it. Now we just need someone from the Deno team to accept a PR that integrates this into /x and then we can close this issue and live on happily :)

crowlKats commented 2 years ago

we are currently doing various architectural changes to the registry and website, so any PRs will get invalid pretty quickly. I will bring this up internally once our reworks are completed

josephrocca commented 1 year ago

@crowlKats Wondering if there are any updates on this yet? A super rough guess at an ETA would be great if possible - e.g. "maybe some time in 2023". (Please feel free to ignore this comment if there are no updates/ETAs yet)

crowlKats commented 1 year ago

The reworks are still somewhat underway, though this could be worked on now. However: this isn't high priority as we have quite a few other things that are being worked on, and especially as there are a bunch of 3rd party solutions for this problem. I cannot give an ETA as this isn't a feature that has been confirmed yet that we want, and in the current state of the new API server, this could be a bit of a problematic feature

josephrocca commented 1 year ago

@crowlKats Thanks for the update! I think I'm at risk of raising points that you and everyone on the team already understand very well, but:

this isn't a feature that has been confirmed yet that we want

Regarding the browser compatibility story of the new npm: stuff, I asked:

Or can we just solve this "browser-incompatible-for-no-good-reason" case with on-the-fly (and cached, of course) pre-compilation/transpilation when a deno.land/x module is downloaded to a browser, https://github.com/denoland/website_feedback/issues/19?

and @dsherret replied:

Yes, I think this is something that could be better solved by registries where if you specify you want a browser compatible module, then it will convert npm specifiers to an esm package, handle deduplication for you, and convert ts to js.

So, given that the npm: stuff (which is Deno-specific) and the TypeScript stuff needs to be transpiled out, isn't it pretty clear that deno.land/x is the best place for the Deno ecosystem to solve this whole browser-incompatibility problem? I find it hard to trust third party services because we've all been burned before by "sunsets" of non-official services before (e.g. rawgit).

I didn't know that this feature was potentially on the chopping block, so I'm wondering what the arguments against it are? The whole value-prop of Deno for me is the browser-compatibility story, and this is a pretty important part of that.

This is the second-most upvoted issue on this repo, so I think other Deno users feel that way too.

KSXGitHub commented 1 year ago

If I understand correctly, npm: scheme can only be used when a local node_modules directory exists, which mean libraries that are published in deno.land/std and deno.land/x shouldn't use it. I don't think it would be a big problem if the transpiler just ignore the npm: scheme.

crowlKats commented 1 year ago

I didn't know that this feature was potentially on the chopping block, so I'm wondering what the arguments against it are?

maybe my wording wasnt right; this isnt on the chopping block, it just hasnt been discussed extensively internally at all yet.

isn't it pretty clear that deno.land/x is the best place for the Deno ecosystem to solve this whole browser-incompatibility problem?

I personally would disagree; this would lock deno.land/x even more to be the defacto and only registry, even though one of the main points for deno to me is it being decentralized. Regardless, I digress; as stated above, this hasnt been discussed yet extensively yet.

crowlKats commented 1 year ago

@KSXGitHub I disagree; if a third party module needs to use a package from npm, this shouldn't be a blocker for it not to be usable in browsers.

KSXGitHub commented 1 year ago

if a third party module needs to use a package from npm, this shouldn't be a blocker for it not to be usable in browsers.

To use an npm package would require --allow-read and node_modules. This would make the idea of writing deno library that depends on npm modules unattractive, therefore, the number of libraries that rely on npm package would be very low.
The transpiler doesn't even concern itself with code that rely on the Deno namespace, there's no reason for the transpiler to care about the npm: scheme.
The purpose of this request is very simple: To make code that can work in the browser work in the browser.

josephrocca commented 1 year ago

it just hasnt been discussed extensively internally at all yet

Ah, gotcha :+1:

this would lock deno.land/x even more to be the defacto and only registry, even though one of the main points for deno to me is it being decentralized.

I agree that hindering deno.land/x by not implementing the second-most-highly-requested feature would increase the chances of competing registries from emerging, but is that the best way to go about this? I.e. making the registry bad enough to encourage other registries? This seems like a very "foundational" feature that all Deno registries should/will have - not a low-priority feature that can/should be a point of differentiation.

Also, I think people want to use deno.land/x because they trust the Deno organisation and have more faith in their vision, security practices, long-term financial stability, up-time, etc. than "random dev with a weekend project". The "market" for registries will have trust-based network effects, and limited ability for product differentiation, so there are inherent forces that are working against the "decentralised registry" vision here. Regardless, using "purposefully make the product bad" as a tool for decentralisation is probably not going to be very effective and will just hinder the whole Deno ecosystem in its battle against competing runtimes.

At this stage I think the main priority should be "make the product good" - which I think was part of the reason for @ry's npm: decision despite it being unpopular from a certain "purist" perspective. I initially disagreed with npm:, but now agree it was a good choice. I think this is a similar case where it would be prudent to put aside this kind of aspiration until the bulk of the friction in the product is gone.

KnorpelSenf commented 1 year ago

I agree that decentralisation is great, but it can already be achieved by keeping the CLI independent of /x semantics (which already is the case). I also don't consider it to be a great plan to build a bad product so that we encourage decentralisation.

Another point is this: The Node compat layer with all its features essentially is a tool to let (m)any package from npm run in the browser. That's a tremendous benefit because by building Deno tooling, we even mitigate the Node/browser incompatibility. In my opinion, that would be an impressive feature which would draw a lot of attention to Deno. Making an npm package browser-compatible rarely was easy, so if Deno would achieve this “accidentally” then I find this very attractive.

ayame113 commented 1 year ago

I still think I need this feature. There is currently no way to write shared frontend and backend code without compiling.

Using things like https://esb.deno.dev is blocked due to the denoland/deno#14879 issue.
Of the dependencies I have, esm.sh, sykpack.dev, etc. can be imported from the browser without any additional work, but only deno.land does not serve JS files. If even one deno.land is included in the dependencies, it will be necessary to build the dependencies.

It would be very useful if deno.land provided this feature.

johnspurlock commented 1 year ago

@ayame113 as a bandaid until they fix this, I just updated esb/est.deno.dev to accept targets of the form: /https/example.com/path.ts

as well as /https://example.com/path.ts

e.g. https://est.deno.dev/https/deno.land/std@0.173.0/encoding/csv/stream.ts

Hope that helps!

denoland / website_feedback

Serve precompiled JavaScript to the browser #19

Reason

Suggestions