search

typelevel / feral

Feral cats are homeless, feral functions are serverless
Apache License 2.0
176 stars 41 forks source link

Wrap the Cloudflare Worker runtime APIs #138

Open armanbilge opened 2 years ago

armanbilge commented 2 years ago

Many of these are very similar to existing Web APIs, but slightly different. So we should probably facade and wrap them ourselves :(

Typescript available here: https://github.com/cloudflare/workers-types/tree/master/overrides

zetashift commented 2 years ago

What about using ScalablyTyped to generate the Scala defs as an initial start? Even the cloudflare/workers-types repo says those types are generated as well. (And then making it typelevel-friendly)

armanbilge commented 2 years ago

As an initial start, we can grab most of these from existing places. I've already done some in https://github.com/typelevel/feral/pull/4. Fetch, WebCrypto, and WebSockets are available via scala-js-dom.

(And then making it typelevel-friendly)

My preference would be not to expose the facades at all if possible. I think that's what I was doing in https://github.com/typelevel/feral/pull/4, but it's been a while.

Would be great if you'd like to pick this up :) my main concern was figuring out a way to actually test this.

zetashift commented 2 years ago

I don't know much about Feral, or Cloudflare Workers :P.

But for testing, am I misunderstanding something? We should be able to test with https://developers.cloudflare.com/workers/wrangler/ right?

armanbilge commented 2 years ago

I don't know much about Feral, or Cloudflare Workers :P.

Sounds like a great learning opportunity! :P

But for testing, am I misunderstanding something? We should be able to test with https://developers.cloudflare.com/workers/wrangler/ right?

Yes, actually I've had my eye on https://miniflare.dev/ for testing. But wiring it up is non-trivial :) I started a branch doing that here: https://github.com/armanbilge/scala-js-env-jsdom-nodejs/commit/7f6081ce40e6bf9b5a7790ebadcc35009fc59a67

zetashift commented 2 years ago

I read up on Miniflare recently and from Cloudflare's blog: https://blog.cloudflare.com/workerd-open-source-workers-runtime/

workerd is also designed to facilitate realistic local testing of Workers. Up until now, this has been achieved using Miniflare, which simulated the Workers API within a Node.js environment. Miniflare has worked well, but in a number of cases its behavior did not exactly match Workers running on Cloudflare. With the release of workerd, Miniflare and the Wrangler CLI tool will now be able to provide a more accurate simulation by leveraging the same runtime code we use in production.

zetashift commented 2 years ago

I'll check out the #4 PR, but from a cursory glance it looks hella intimidating :P.

So it uses facades, but provides a very functional-programming-y interface on top of it using cats effect and fs2?

armanbilge commented 2 years ago

Interesting! Well I guess we should toss that miniflare branch then 😂

Thanks for sharing, that's pretty cool. I'll have to look at it more closely to figure out how we might use it.

It also reminded me that cloudflare workers supports WASM, and Scala Native will support WASM in the next major release. So that's something to keep in mind as well 🤔

So it uses facades, but provides a very functional-programming-y interface on top of it using cats effect and fs2?

Yes, that's the idea. http4s-dom does a similar thing, as does the History API in https://github.com/armanbilge/calico/pull/68. In fact, fs2-io JVM does the exact same thing for JDK APIs (instead of JS facades). So it's a very common pattern.

Feel free to comment directly on the PR with questions!

zetashift commented 2 years ago

I asked in the scala-native discord channel about Scala Native wasm examples, but I didn't know it was slated for next release, thought it was in already haha.

I guess the hard part of contributing to that PR is, the fact that I've never written/worked on a FP library, I've only consumed them! I'll read more into it, thank you for the help.

armanbilge commented 2 years ago

Yes, check out https://github.com/shadaj/scala-native-wasm

I didn't know it was slated for next release, thought it was in already haha.

Yep it was already merged into the 0.5.x branch, in theory you can publish a snapshot and start playing it.

I guess the hard part of contributing to that PR is, the fact that I've never written/worked on a FP library,

Personally I find it quite fun to figure out how to wrap some (god awful) non-FP API with CE3/FS2 😃

zetashift commented 2 years ago

Personally I find it quite fun to figure out how to wrap some (god awful) non-FP API with CE3/FS2

It does sound quite fun, I just have never done it :P. So let's try this out! Here's what I have in mind, let me know if I'm close:

armanbilge commented 2 years ago

@zetashift personally I merge instead of rebase 😜 but yes!! 🚀

integrate wrangler instead of miniflare with the current code

I'm quite interested about this part too :)

zetashift commented 2 years ago
  • integrate wrangler instead of miniflare with the current code

So this seems to be a bit of a rabbithole. This is as far I know the current situation.

Miniflare now uses Cloudflare's open-source Workers runtime, workerd, to run your code! tada This is a massive change, and should mean your code runs locally almost-exactly as in production.

Breaking Changes

Miniflare's CLI has been removed. We're still discussing whether it makes sense to keep this, given wrangler dev has additional features such as automatic bundling and TypeScript support. For now, use wrangler dev --experimental-local.

So I think we should still use Miniflare, but for the developer experience that will mean actually using wrangler.

armanbilge commented 2 years ago

So I think we should still use Miniflare

I agree.

but for the developer experience that will mean actually using wrangler

I'm less sure about this. I wonder if instead of working with the wrangler CLI, we can directly use the Miniflare JS API. This is essentially what I was exploring in https://github.com/armanbilge/scala-js-env-jsdom-nodejs/commit/7f6081ce40e6bf9b5a7790ebadcc35009fc59a67.

zetashift commented 2 years ago

I'm less sure about this. I wonder if instead of working with the wrangler CLI, we can directly use the Miniflare JS API. This is essentially what I was exploring in https://github.com/armanbilge/scala-js-env-jsdom-nodejs/commit/7f6081ce40e6bf9b5a7790ebadcc35009fc59a67.

Ah yea! You posted this link before, but I couldn't really grasp the implications.I'll look into the Miniflare API more (the docs seem solid from a first glance) and figure out what this whole scala-js-env-jsdom-nodejs (what a naming :P) exactly does.

armanbilge commented 2 years ago

Right, I cheated by starting with a fork of this project because it does a similar thing and trying to modify it for miniflare:

https://github.com/scala-js/scala-js-env-jsdom-nodejs

Basically, JSDOM is a thing that lets you emulate browser APIs inside of Node.js.

https://github.com/jsdom/jsdom

Similarly, Miniflare is a thing that lets you emulate Cloudflare APIs inside of Node.js. (Or at least it used to be, not sure if workerd has changed that.)

The scala-js-env-jsdom-nodejs lets you run your Scala.js tests inside of the JSDOM environment.

So in theory seems like we should be able to do basically the same thing except with Miniflare.

zetashift commented 2 years ago

Right that clears a lot up, thank you! So Miniflare (the v3) currently using workerd, which is a custom runtime and I don't think using scala-js-env-jsdom-nodejs will cut it?

Look at miniflare's code it, it just seems to talk to a local workerd runtime https://github.com/cloudflare/miniflare/blob/v3.0.0-next.2/packages/tre/src/index.ts

Wouldn't it be better to have bindings to Miniflare(which does all the hard work of talking to workerd) then that are used for running tests locally?

EDIT: I guess that's what you did in your scala-js-env-jsdom-node-js fork, but it still seems somewhat redundant? Like it's a nodejs process emulating a DOM, that calls a miniflare library talking to a workerd runtime

armanbilge commented 2 years ago

Like it's a nodejs process emulating a DOM, that calls a miniflare library talking to a workerd runtime

scala-js-env-jsdom-node-js is a Node.js process emulating a DOM. My fork removes JSDOM and replaces it with Miniflare, so we end up with a Node.js process emulating Cloudflare (no more DOM).

With the new Miniflare, you are right that we would have a Node.js process calling to Miniflare calling to workerd. Which raises the question, why don't we just build a JSEnv that directly runs on workerd? And I think we've come full circle 😅