search

denoland / dotland

[Archived] deno.land website
https://deno.land
MIT License
957 stars 626 forks source link

feat: on the fly transpile #2669

Closed ayame113 closed 1 year ago

ayame113 commented 1 year ago

continuation of denoland/dotland#1902 close denoland/website_feedback#19

ayame113 commented 1 year ago

I have deployed this branch to https://dotland-transpile.deno.dev/.

Transpiling to js files for browsers seems to work fine.

image

ayame113 commented 1 year ago

Is there anything preventing this PR from being merged?

josephrocca commented 1 year ago

Wondering: RE this discussion, does this handle browser-compatible npm: imports? A large portion of the packages on NPM are web compatible - it has become a kind of default way of publishing a JS package. If npm: ends up being used quite often, then it might hinder the utility of this transpilation feature.

Or, if modules with npm: imports won't currently work even after this is merged, I'm wondering if there's a pathway towards handling it in the future?

(I guess there is a distinction between NPM modules that are browser-compatible "out of the box", vs those that need to have things like Buffer "browserified" - I'm specifically talking about the latter case.)

ayame113 commented 1 year ago

Wondering: RE this discussion, does this handle browser-compatible npm: imports? A large portion of the packages on NPM are web compatible - it has become a kind of default way of publishing a JS package. If npm: ends up being used quite often, then it might hinder the utility of this transpilation feature.

Or, if modules with npm: imports won't currently work even after this is merged, I'm wondering if there's a pathway towards handling it in the future?

(I guess there is a distinction between NPM modules that are browser-compatible "out of the box", vs those that need to have things like Buffer "browserified" - I'm specifically talking about the latter case.)

I think this is a good point.

https://esm.sh seems to do "browserify" npm modules very well as far as I can see. In my opinion, to handle npm: imports, it would be nice to be able to specify in the config file how import statements should be transformed. (e.g. npm:foo->https://esm.sh/foo)

This is (luckily) exactly what is being discussed in denoland/website_feedback#3 and it is on the Q1 roadmap so hopefully it should be available by the end of March this year.

crowlKats commented 1 year ago

We have discussed this internally, however we are unsure we want this feature due to complications including a large increase in costs on our end, especially since these transpiled sources should be cached/stored in an S3 bucket, so for now we are putting this on hold until a proper decision.

josephrocca commented 1 year ago

@crowlKats Wouldn't this only double the size in the absolute worst case? I.e. the case where every single package is transpiled? Presumably it would be something more like 30% increase - since I assume most packages in deno.land/x tend to be used with Deno, and don't need transpilation. Or do I misunderstand something about the type stripping process?

Or is it that you don't think it's worth a 30% increase in storage? Deno is the bastion for browser runtime compatibility - so this seems like something that should have been implemented on day one. Is the recommendation to publish two separate packages on deno.land/x?

Unless I'm misunderstanding something, this is super disappointing. It's been so long, and this is such an obvious win (again, unless I'm misunderstanding something).

crowlKats commented 1 year ago

The storage cost isnt the main issue, but the requests are. Lets say someone starts using a module in a popular website by embedding said module in a script tag. Thats going to massive increase the traffic to dotland (and our system) just for that. now quantify that by multiple extremely popular websites. and unsure if we can handle that traffic without potentially degraded experience for normal usage. sure, thats just a worst-case scenario, but thats what we need to prepare for. And lets say this isnt a issue whatsoever, this isnt a high priority project right now for someone on the team to invest time to investigate & working on this.

On a personal note: I like this feature, I want it, but I see that this needs further investigation. This feature basically makes us a full-fledged CDN alike other large js CDN services, which have insane amount of traffic.

josephrocca commented 1 year ago

@crowlKats Ah, okay, that makes perfect sense - bandwidth could definitely become huge, and it would be tragic to have to pull down the CDN after it becomes broadly used, and in so doing, cause a lot of sites to break. I wonder if there's a way to partner with Cloudflare in some way to solve this - since they already serve an absurd amount of traffic for cdn.js and jsdelivr.

On another random note, it's possible to serve directly from NPM using services like jsdelivr (like this: https://cdn.jsdelivr.net/npm/packageName@version) - perhaps they'll eventually get a /deno/ handler. Or maybe unpkg/skypack/etc, since they are in the business of transpilation already. It would be great to somehow be able to use deno.land/x to publish packages, and then to use them on both the server and the client.

ayame113 commented 1 year ago

The storage cost isnt the main issue, but the requests are. Lets say someone starts using a module in a popular website by embedding said module in a script tag. Thats going to massive increase the traffic to dotland (and our system) just for that. now quantify that by multiple extremely popular websites. and unsure if we can handle that traffic without potentially degraded experience for normal usage. sure, thats just a worst-case scenario, but thats what we need to prepare for. And lets say this isnt a issue whatsoever, this isnt a high priority project right now for someone on the team to invest time to investigate & working on this.

Makes a lot of sense. Also, I'm happy to hear that the core team has discussed this internally. However, I currently already have a library for browsers published on deno.land. (Transpiled with deno bundle in advance.) example: https://deno.land/x/gbfs_map@v2.3.1, https://deno.land/x/embed_pdf@v1.2.0

Currently there is esb.deno.dev as a workaround for doing on-the-fly transpiling, also hosted by deno deploy. Since the requests eventually reach deno deploy, I think the problem of increased traffic exists whether or not you introduce this feature. I really like the way libraries are distributed by hosting ES Modules on deno.land and loading them directly from the browser, so I would love to see this feature introduced.

The idea of storing the transpiled results in s3 packets sounds good, but swc updates can change the transpiled results. Maybe we need some logic to delete and update the transpiled results after a certain period of time? (Maybe we can use the cache API?)

crowlKats commented 1 year ago

Thanks for the PR. We don't use this repository anymore, as the website went closed-source, and there are conflicting opinions internally about whether to land this or not, so for now I will be closing this PR and hopefully in the future we can reference it if we do plan to implement it.