[dev-server] Use dev-server as middleware for Express

mercmobily commented 4 years ago

I am using dev-server to serve a local project without having to build each time. In order for my application to actually work, I need to define routes within the server.

I wrote this tiny function which will basically enrich app (which is an Express object) so that I add es-dev-server as Express routes. I do that by creating the server manually with createConfig, and then creating a Koa app, and use koaApp.callback() to convert the koa app into an Express one. It works beautifully, and we are on the verge of releasing all of this under a free license.

However, as I understand es-dev-server is no more... I was hoping you could let me know if this code is still functional (that is, es-dev-server and es-server share the same API) or if I need to change things around. Or, if what I am doing can be done with es-dev-server.

Thank you for the awesome work.

exports = module.exports = function (app) {

    const Koa = require('koa')
    const config = esDevServer.createConfig({
      nodeResolve: true,
      appIndex: 'index.html',
      moduleDirs: ['node_modules'],
      preserveSymlinks: true
    })
    const middlewares = esDevServer.createMiddlewares(config)
    const koaApp = new Koa()
    middlewares.forEach(middleware => {
      koaApp.use(middleware)
    })

    app.use(koaApp.callback())
  }

LarsDenBakker commented 4 years ago

Right now we don't expose only the middleware of the server. This is because we the core server consists of plugins and middleware, so you'd only get a part of the functionality.

The node API we have right now is documented here: https://modern-web.dev/docs/dev-server/node-api/

It returns a DevServer instance which has a koaApp and server property, which you can use. Does that fit your use case?

mercmobily commented 4 years ago

I don't think it does, because I have an existing Express app object, which I need to enrich with Express' app.use() This way, an app will 1) Define several routes 2) Add dev-server as a middleware. All I can see in the API is a way to start the server, which is not what I want.

What I would need, is the Koa app fully designed, but before starting Koa. That way, I can get koaApp.callback() which will give me what I need.

LarsDenBakker commented 4 years ago

I will check how we can support this without bleeding too much internal knowledge.

mercmobily commented 4 years ago

Hi,

Thanks. I don't know how this is not considered a totally necessary feature... How else do you test a server with custom routes?

I mean, every client software needs a server to talk to!

Merc.

On Wed, Sep 23, 2020, 4:18 PM Lars den Bakker notifications@github.com wrote:

I will check how we can support this without bleeding too much internal knowledge.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/modernweb-dev/web/issues/615#issuecomment-697210468, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQHWXVTPFUZLKS3X5YY3VLSHGVL5ANCNFSM4RVG5Y4Q .

LarsDenBakker commented 4 years ago

In essence web dev server is a dev server for local development serving static files on the file system. It's not a plugin for other servers. This is also because there are features which the server offers that require being in control of the server environment.

I'm not sure what you're describing exactly, but do you mean you want to mix both the API backend and the file server in one? Typically what people do is run the two servers side by side, and proxy only the necessary requests to the API server.

mercmobily commented 4 years ago

In essence web dev server is a dev server for local development serving static files on the file system. It's not a plugin for other servers.

Well, it's basically something that serves static files, and it creates Koa middleware to do that...

This is also because there are features which the server offers that require being in control of the server environment.

I am not sure what they could be. However, essentially what I think should happen is the exposure of the "routes" side of the story.

I'm not sure what you're describing exactly, but do you mean you want to mix both the API backend and the file server in one? Typically what people do is run the two servers side by side, and proxy only the necessary requests to the API server.

Oh ouch. That sounds very much unnecessarily painful Lars! Please have a look at what I am doing at the moment -- I am adding comments to it.

With the "old" es-dev-server, I simply call createMiddlewares based on the config:

const config = esDevServer.createConfig({
      nodeResolve: true,
      appIndex: 'index.html',
      moduleDirs: ['node_modules'],
      preserveSymlinks: true
    })
    const middlewares = esDevServer.createMiddlewares(config)

At that point, I create a Koa app that uses all those middlewares:

    const koaApp = new Koa()
    middlewares.forEach(middleware => {
      koaApp.use(middleware)
    })

koaApp to me is golden because it's the full server enclosed in a Koa app. This is most likely what esDevServer does too. Now, consider that app is my Express 4 application which implements the full backend of my application. All that it's missing, is the serving of static files -- which for SPAs they need some magic so that node resolution is handled properly. And voila':

app.use(koaApp.callback())


All I am after, is the Koa middlewares that make up the Koa app which is then run by dev-server.
Proxying the requests feels clunky...

mercmobily commented 4 years ago

Just letting you know that I am trying... I am getting lost in the code for the core-server. I don't really understand typescript, so I am a little lost more often than not. However, the key is to expose server/createMiddleware in such a way so that it "just works" -- it returns an array of middleware objects, which I can then manually add to a Koa app.

More neatly, you would provide a function that returns one middleware that represents the whole thing.

I am nearly there, but I am getting File Not Found at the moment.

mercmobily commented 4 years ago

One thing I don't quite get... we have gorgeous ES6 which doesn't require building or anything. I can understand Typescript in cases where you have a huge software project and typing actually helps, but... why typescript for something like the es-dev-server...? I guess it's totally the dev's choice, and it depends on what they are comfortable with. But I do wish I were looking at ES6 code right now.

mercmobily commented 4 years ago

UPDATED with some minor fixups

Alright, I did it. This took quite a while, since I am 100% unfamiliar with the codebase and unfamiliar with Typescript. Hopefully, this will give an insight on what I had to do, and whether the whole thing can be improved.

Considering that app is an Express application, here is what's required to add dev-server as an Express middleware:

    const { createMiddleware } = require('@web/dev-server-core/dist/server/createMiddleware')
    const { nodeResolvePlugin } = require('@web/dev-server/dist/plugins/nodeResolvePlugin')
    const { transformModuleImportsPlugin } = require('@web/dev-server-core/dist/plugins/transformModuleImportsPlugin.js')
    const FSWatcher = require('chokidar')
    const Koa = require('koa')

    const config = {
      plugins: [],
      appIndex: 'index.html',
      rootDir: '/home/merc/Development/superSoftware'
    }

    // Create file watcher
    const fw = FSWatcher.watch([])

    const nrp = nodeResolvePlugin('/home/merc/Development/superSoftware)
    nrp.serverStart({ fileWatcher: fw, config })
    config.plugins.push(nrp)

    const tmip = transformModuleImportsPlugin(config.plugins, '/home/merc/Development/superSoftware')
    config.plugins.push(tmip)

    const middlewares = createMiddleware(config, { debug: (...params) => { console.log(params) } }, fw)

    const koaApp = new Koa()
    middlewares.forEach(middleware => {
      koaApp.use(middleware)
    })
    app.use(koaApp.callback())

Please note that I am listing the following issues in the most humble possible way, and always being thankful for your hard work.

Issues:

1) One glaring issue here is that I need the nodeResolvePlugin, a plugin from dev-server. This means that I have to manually call the serverStart() hook, in order to set rootDir in the plugin's scope. This is because core's own transformModuleImportsPlugin doesn't actually do the work: it offloads it to whichever plugin is already loaded and exports the resolveImport function (hence the 2-stage adding of plugins), which happens to be dev-server's nodeResolvePlugin. Ideally, the core server should be able to actually do the transformation. I realise that this would imply having rollup in the core server (yeah...). But at the same time, is the core server actually functional without the ability to resolve modules? The solution where there is a plugin (in core) that looks for other plugins (from non-core) and uses them feels very convoluted.

2) Again, the createMiddleware function expects a logger object, which implements the usual suspects (warn, error, debug) but also some funky ones (group functions). The core server here is totally dependent on the main dev-server, which shouldn't be the case. The core server should provide a basic implementation of the logger object, probably using the debug module by default. For example pluginTransformMiddleware calls logger.syntaxError, but it's not actually defined.

3) Plugins don't seem to have a standard interface. Each plugin seems to have different parameters. I would suggest always and only passing config to them, regardless, and expect them to figure out what to do with it.

My code above works: you can have your own Express or Koa server, define your routes, and then simply offload the static files to dev-server/dev-server-core. No forwarding required. I think this is a good enough use case to provide a better way of achieving what I did.

It would be nice, for example, to add the websocket plugin, or the watch plugin. I guess it's possible, but I feel the more I add to it, the more fragile it will become. Having a documented, official way of doing this would be fantastic.

mercmobily commented 4 years ago

Another thought: it would probably be easier if dev-server exposed its Koa server (rather than running it). This would mean that I would simply require that one, rather than the core one.

LarsDenBakker commented 4 years ago

If we write es6 we would be stuck in the past by 5 years, there are so many nice things like async functions and optional chaining that came out since then that are helpful to this codebase.

At modern web we're big advocates for buildless development, this is what we recommend to developers to start out. We also recommend making educated decisions to add tools on top of that if it makes sense for a project. The dev server is used in the test runner and together they are quite a large codebase - using typescript has been a tremendous help in documenting and maintaining the public APIs of the packages as well as manage large refactorings.

People expect typings from packages, so we have to do something there anyway. And because this is a node package, we also need to compile it down to the lowest version of node we support (v10 right now) and we need to make a commonjs version of the codebase available as well. A tool needs to come into play here at some point. Typescript nicely packages that all in one tool.

I'm happy to further discuss the technical decisions of this project, but please refrain from judgements like "this feels convoluted" and "should" and "should not". We have been maintaining and triaging issues for es-dev-server for a few years now, we set up the new project this way for a reason.

Our goal is to keep the API surface minimal, I've had issues in es-dev-server where I exposed too much and was either unable to implement certain new features or had to do some crazy workarounds. Therefore while we can technically do anything we want, I want to understand the use cases people have and evaluate is that is worth the technical cost.

mercmobily commented 4 years ago

I just want to make this dead clear: I really do understand that you know what you are doing, and I totally see that whatever decisions you have made stemmed from a "lesson learned' perspective. I have been there before, and I know how irritating it is to then get somebody who hasn't lived those lessons, is not maintaining the code, and tells you how the project "shouldn't" or "should" be.

I want to make it clear that ALL of my comments, all my "shouldn't", all my "should", weren't meant in the sense of "how the project should be", but "how the project should be to facilitate my specific use case". I wrote this ticket with the full understanding that the answer could be "Sorry, not a strong enough use case". And if that is the answer, I wasn't going to challenge it and wasn't going to have a spat.

So, to go through them: Lars, I realise that picking Typescript must have benefits, and you as the developers know better than anybody else. I mean, I wouldn't be still using Javascript today if I couldn't use async/await; I was convinced it was part of ES6 too! (It's not, but it is really supported by 92% of browsers and any half-recent node). I still never liked it, and stayed away from any library/framework that used it (see: Dojo 2, Angular 2, etc.). It's a choice that makes my life interesting at times, and at one point I will finally get it and set up a dev environment that uses it.

Please allow me to go through my points again.

Issue 1: dev-server-core doesn't know how to transform modules. To do that, it needs a plugin from dev-server (which is an outer layer). Again, from a perspective of a developer that just wants to use dev-server-core, this makes things hard. I am not saying "it shouldn't be like this", and I am not telling you how to code your own software. If this was a very deliberate, strict design decision, then it is what it is, and I will do my best to work with it. However, if it is something that can be changed (that is, you have strongly considered giving the core module the ability to resolve the module's names, and you were 50-50 about it), then yes, absolutely: from the perspective of a developer wanting to use dev-server-core, it absolutely should be like that.

Issue 2: dev-core-server doesn't actually know how to log. Once again, my writing did come across like I was telling you how things should be. What I meant, once again, is from a perspective of a developer who wants to use dev-core-server. If you don't feel that using dev-core-server straight is something you want to support, then it's obviously fine as it is. My "should" and "shouldn't" is uniquely from the perspective of a developer who wants to use it. If you think it would be valuable, I would be totally happy to write a mini-logger which would be super basic and embedded in the dev-server-core, and it would be used as default if there is no logging plugin.

Issue 3: plugins don't have a standard API At the moment, the nodeResolverPlugin is included by calling codeResolvePlugin('/home/merc/Development/directory' whereas the transformModulePlugin is included by calling transformModuleImportsPlugin(config.plugins, '/home/merc/Development/directory') ). Other plugins have different instancing signatures. From a perspective of an outside developer who might want to add several plugins, this is not ideal. Once again, if you think this is a worthwhile change, I'd be totally happy to submit a change where we uniform this. But again if there are design reasons for this, then all good.

Please understand that I had been battling this for a good 3 hours, unfamiliar with the codebase, unfamiliar with Typescript, and after getting lost a number of times. I wrote my message above at the end of that session, and some of the spirit obviously didn't convey (and in fact some of the fatigue/frustration did -- I apologise for this).

Thanks for the great work!

UPDATE: Formatting

mercmobily commented 4 years ago

I guess the short hand version of the question is:

Are you guys interested at all in having dev-server-core as an independent module that doesn't rely on plugins that are found in the outer dev-server (which happens to also use dev-server-core)? Logging is easy, but module resolution is way less trivial and seems to be a deliberate design issue.
Are you guys interested in having a uniform API for plugins?

mercmobily commented 3 years ago

I am closing this. You obviously set up the new project this way for a reason. I would have loved to hear the reason, and would have loved to help. But, all good. Best of luck and thank you for everything! Open-wc was a great inspiration. Closing.

mercmobily commented 3 years ago

Pure ES6, no typescript. Minimal, functional code. About to be turned into a literate program. Very simple architecture. It can be used as a stand-alone server or as middleware. It does caching (with expiry if the file changes). 6 hours start to finish.

https://github.com/mobily-enterprises/es6-dev-server

mercmobily commented 3 years ago

Wooops, apologies, the repo was set as private. It's public now!

mercmobily commented 3 years ago

Sorry about the noise. I wanted to add that I added literate documentation to the ~150 lines of code that make up the program.

Since you have vast experience in writing this, and if you have spare time, I would love to know where I can improve on it (e.g. use cases I didn't predict,possible pitfalls, etc.). If not, all good.

If others are requesting such a feature (see: using web-dev-server as Express middleware), feel free to distribute/rebrand es6-dev-server as you like.

I hope this helped!

https://github.com/mobily-enterprises/es6-dev-server/blob/main/index.js

modernweb-dev / web

[dev-server] Use dev-server as middleware for Express #615