Flask Multi-Server Development Hooks

[mitsuhiko]

I don't have a good description for the issue yet but one of the issues I keep running into myself in 2017 is that it's not convenient to marry Flask and webpack. The issue here is that one wants to both have a server app as well as a client app and each uses a hot reloader for code.

There is a lot of boilerplate to make two services work and there is absolutely no support on the side of Flask to make it work. So here is what Flask most likely should do:

Environment Awareness

Flask needs to know about environments. For now I would propose to mirror the node model where we have an environment flag and depending on the environment flag some flags get flipped automatically. In development for instance we would turn on debug by default. This discourages apps to make too many assumptions about the debug flag but instead use an environment flag instead.

This means debug can be flipped on and off in development mode. A potential webpack integration extension can then forward that flag to webpack.

Proxy or be-proxied Capabilities

Right now one either needs front/backend to be running different servers on different ports or use some manual proxy setup to make this work. Right now the webpack server can pass through to the Flask server but it's quite hacky and non trivial.

In an ideal case Flask in development mode can spawn another process (like a webpack server, ngrok etc.) and manage this process. It would work independently of the Flask reloader but shut down together.

If it spawns a process on a different port Flask should itself be proxied through that other server or proxy to another server if that's easier. This would most likely require that the Werkzeug server learns HTTP proxying as we might not be able to do this on the WSGI level.

[mitsuhiko]

For now this issue is an unclear goal but I have a few implementations of parts of this to make Flask and neutrino work together. My goal would be to make this work good enough that getsentry/zeus can start using it.

[davidism]

How would the development env var differ from the FLASK_DEBUG env var?

[davidism]

It sounds like the server proxy something similar to DispatchingMiddleware, except for any process, not just WSGI. We'd probably also want to handle subdomains or subpaths (api.myapp vs myapp/api). localhost can't have subdomains, so we'd still need to document how to setup /etc/hosts.

[mitsuhiko]

@davidism the main goal is to have this:

• FLASK_ENV=production for production mode (the default), defaults to FLASK_DEBUG=0
• FLASK_ENV=development for development mode, defaults to FLASK_DEBUG=1

Additional environments can be created as wanted (for instance staging etc.). The environment is then forwarded to other development tools as well (such as node based ones as NODE_ENV).

The main motivation is that you want to have software run in development without having debug on or to have debug on in production like environments (staging).

The FLASK_ENV would be used to turn on the webpack live reloader etc. / turning minifiers off without having to have debug mode enabled. This is important for when you want to collaborate with others through things such as ngrok without worrying about potential security issues coming up from debug turned on.

[mitsuhiko]

We might even go as far as having a flask build command which kicks off build processes if a webpack extension or similar is loaded which then builds the assets to dev or prod mode based on the FLASK_ENV var (which is forwarded to NODE_ENV) and have flask run automatically spawn the webpack server and proxy through etc.

The proxy feature however generally should also be able to dispatch to other systems such as async python servers for websockets etc. if needed.

[mitsuhiko]

For the issue with /etc/hosts I'm kinda curious how other development servers are dealing with this now. In particular we're at the point where we need to also provide support for local SSL setups in some cases. The moment you're no longer using localhost as name you need to setup SSL even locally.

[davidism]

I personally haven't had much issue doing things with Flask and Webpack. All I do is add project and api.project as aliases for 127.0.0.1, then have Flask bind to api.project and Webpack bind to project. Everything seemed to work.

[mitsuhiko]

@davidism how do you bind them to the same ip but different hostnames? I assume you use different ports or do you use a local proxy setup for this?

[davidism]

It's been a while, so maybe I'm misremembering. Maybe it was still different ports.

[lepture]

I'm using Flask and webpack together to develop Typlog. Here is my solution:

I create a JSON file assets.json, it has something like:

{
  "base": {
    "styles": [
      "http://localhost:8080/assets/base.css"
    ],
    "scripts": []
  }
}

This assets file is loaded into context processor, so that I can use it in template:

{% for src in assets.base.styles %}
  <link rel="stylesheet" href="{{ src }}">
{% endfor %}

Then, I will generate a new assets.json for production when deploying.

---

1. I use localtest.me for development. http://readme.localtest.me/
2. I use hitch to create a SSL forwarding proxy for localtest.me.

[lepture]

I don't care if I'm using webpack or other tools. What I need to do is taking care of assets.json file.

[mitsuhiko]

@lepture yeah. My plan would be to have an assets file for production builds and a helper to generate links to them.

[ThiefMaster]

BTW, if you want to add anything directly related to webpack (either in flask itself or as an addon) and not just generic support to run stuff like webpacks autobuild, maybe having a look at Flask-WebpackExt could save some time.

That extension is actually developed by some of my colleagues and it's used by the two major flask apps at CERN (indico, invenio). Maybe they'd be even interested in contributing to webpack stuff in Flask itself,, (pinging @lnielsen, @pferreir)

[mitsuhiko]

Right now my thinking process is something like this: Flask gets a buildprocess decorator which registers a thing by name as a build process. This can either be an external build process or just a thread. It gets a unique key it's identified with.

Upon reload new build processes are loaded and old ones are discarded.

Something like this maybe:

@app.buildprocess(key='webpack')
def webpack(process):
    config = 'config.%s.js' % app.env
    p = subprocess.Popen(['build-process', config], cwd=...)
    process.register_wait(p.wait)
    process.register_exit(p.terminate)

There might be nice helpers for making this nicer when actual subprocesses are involved. When the app is first loaded all build processes are registered with the development server. When the app is reloaded all processes that are no longer needed are closed by invoking the exit functions. A build process either has to quit immediately or watch in the background. Still need to decide on how this is controlled, maybe a flag is passed that indicates background watching mode. Likewise the process should pass ports over if proxying is wanted.

[mitsuhiko]

This actually is unlikely to work because we want these processes to be owned by the outside reloading process. So more likely we need to be constraint to something that can be serialized to that process. Maybe we just limit this to processes like this:

@app.buildprocess(key='webpack')
def webpack(build_config):
    cmdline = ['build-process', 'config.%s.js' % app.env]
    # the build_config holds also config parameters that can customize
    # how the build is done.  In this case we check if watch is enabled
    # to turn on the watcher of the external build process
    if build_config.options['watch']:
        cmdline.append('--watch')
    # this communicates the parameters upwards to the reloader which
    # owns the processes.
    build_config.spawn(cmdline, cwd=...)
    # we also want to proxy everything from /static/foo to the other
    # server listening at 5001 that the build process spawns.
    if app.env == 'development':
        build_config.proxy('http://127.0.0.1:5001/', prefix='/static')

[lepture]

@mitsuhiko @davidism should this be put into milestone 1.0?

[mitsuhiko]

Yeah. I want to get this finished for 1.0

[davidism]

@mitsuhiko we have FLASK_ENV support and Werkzeug has ProxyMiddleware now. What else do you want to get in?

[pferreir]

Maybe they'd be even interested in contributing to webpack stuff in Flask itself,, (pinging @lnielsen, @pferreir)

Definitely interested in lending a hand, if needed.

EDIT:

From what I understood, there are three main ideas here:

1. Proxying asset requests to webpack-dev-server;
2. Generating a JSON manifest for production deployment;
3. Automatically executing webpack --watch when the dev server is run;

Is there a particular reason why you'd like to use webpack-dev-server over serving the resulting bundles normally? The work that @lnielsen did in flask-webpackext, and which I've been building on to integrate webpack into Indico, doesn't involve the dev server at all. Bundles are served from Flask, from a URL path that is customizable. It also creates a config file that webpack.config.js can read and which includes path/URL information.

As for 2, we're using ManifestPlugin, which the extension can parse and conveniently use from templates. E.g. {{ webpack['common'] }} will generate a corresponding <script> tag.

[lnielsen]

Definitely interested in lending a hand, if needed.

Same here.

As far as I can decode from the discussion it should be fairly easy to integrate Flask-WebpackExt with the @app.buildprocess() decorator and FLASK_ENV variable.

Once you have the buildprocess() decorator I'm happy to give it a try and integrate it with Flask-WebpackExt.

Would the buildprocess() decortaor run the proccess during a flask run or via something like flask build?

Spawning build process Flask-WebpackExt is depending on pynpm and pywebpack to do most of the work. We would likely need to change pynpm to allow outputting the cmdline for build_config.spawn(cmdline, ..) instead of actually running the command.

Something like:

# FROM:
from pynpm import NPMPackage
pkg = NPMPackage('path/to/package.json')
pkg.run_script('build', '--report')
# TO:
from pynpm import NPMPackage
pkg = NPMPackage('path/to/package.json', run=False)
cmdline = pkg.run_script('build', '--report')

I see it as positive that Flask could provide some API to run the command line.

FLASK_ENV to NODE_ENV

Flask-WebpackExt (optionally) allows you to inject a config.json into your webpack build directory, so you can provide webpack with access to debug flag, paths and URLs. It should be trivial to add the FLASK_ENV there or on the cmdline.

You can see an example application using Flask-WebpackExt.

Basically, you point Flask-WebpackExt to your Webpack project's package.json, and then commands like flask webpack install and flask webpack build, will run the corresponding NPM commands/scripts.

If your Webpack project generates a manifest.json (via some of the available plugins) Flask-WebpackExt can read this manifest.json and make them available in templates (e.g. {{ webpack['main.js'] }}). This is only useful if your Flask app needs to serve the assets, and works similar to the way @lepture described.

[lnielsen]

Would the buildprocess() decortaor run the proccess during a flask run or via something like flask build?

Answering myself here: via flask build :-)

[davidism]

Not sure if this would be good, but perhaps we could add a method to the cli objects, connect_external or something. So you could do:

@build_command.connect_external()
@run_command.connect_external()
def webpack(config):
    ...

When the command runs, it will execute any connected methods, possibly gated behind flags (Lektor does -f webpack).

[mitsuhiko]

I played around with it a bit now and the main complexity is that we are still missing a bit of an interface from the werkzeug reloader to Flask. In particular there is no good way to keep things alive between reloads that might need to at least partially interface with the app.

Right now I'm thinking we might set up a communication channel with the reloader on the app object. So when the app gets initialized it asks which build commands exist. Each that is not registered with a thing on the reloader gets spawned. Each that has a different version gets restarted. The reloader itself is obviously in a different process so it might just use a channel to talk with the app.

So in pseudo code we might have this:

app = Flask(...)
app.dev_controller = DevController()

And on that controller there are things such as:

CURRENT_VERSION = 1
if app.dev_controller is None:
    return
service = app.dev_controller.get_service_process('foo')
if service is None or service.version != CURRENT_VERSION:
    service = app.dev_controller.spawn_service_process('foo', version=CURRENT_VERSION, cmdline=['cmdline', 'goes', 'here'])

Something like this.

[TimotheeJeannin]

Is there a particular reason why you'd like to use webpack-dev-server over serving the resulting bundles normally? @pferreir

Probably because of hot module replacement among other things. Each time a source file changes, corresponding modules are swapped without a page reload.

[TimotheeJeannin]

I use Flask and Webpack with this config:

module.exports = {
    // config stuff here ...
    devServer: {
        proxy: {'/': {target: 'http://localhost:8000'}}
    }
    // other config stuff here ...
};

Basically, all requests are proxied to localhost:8000 but the webpack-dev-server actually does not proxy requests matching files generated by webpack. Both webpack hot module replacement and flask reload works correctly with this 1 line setup.

I did a small experiment based on the Vue.js webpack-simple template here : https://github.com/TimotheeJeannin/vue-webpack-playground before using this setup on bigger projects.

Is there anything wrong with this setup in your opinion ?

[pferreir]

Probably because of hot module replacement among other things. Each time a source file changes, corresponding modules are swapped without a page reload.

I guess you could have some WSGI middleware take care of it, in the same way https://github.com/webpack/webpack-dev-middleware does.

[j-walker23]

@TimotheeJeannin are you saying HMR works with just that simple proxy? Any chance you could tell me about any external env settings that can't be found in your repo? External like nginx config would be. Really any missing pieces would be greatly appreciated.

[TimotheeJeannin]

@j-walker23 Yes, it does work very nicely. I have been using this configuration for months and I didn't have a single issue with it so far. Note that the webpack-dev-server is used for development only. The main benefit for me is that I can have a multiplage web application and still benefit from hot module replacement and all the other nice things webpack has to offer. I have one entry point per page as recommended by webpack.

For developpement my stack is Webpack Dev Server -> Gunicorn (with reload on change) -> Flask Application And I switch to Webpack Dev Server -> Flask in Debug Mode when I need to debug.

And for production : Nginx -> Gunicorn -> Gevent -> Flask

I don't really understand why you're looking for an nginx config and how this relates to the webpack-dev-server. Can you elaborate ?

[j-walker23]

@TimotheeJeannin Oh man, thanks for the quick reply. I wish i would have seen this. Mine is almost the exact same minus one key part. My debug env uses the exact same stack as when deployed. Including ssl cert and all. It was a pretty massive pain to get working since ssl support is not usually high priority for local develop services, but i was dying from reported bugs that were only with prod setup.

Anyway, the why isn't important. My problem seems to be two things. First, because WDS compiles in memory, i don't understand how to get it so flask/jinja can process my index.html from the templates folder. That's the one i've never been able to figure out. The other difficulty, and this is why i asked about nginx, is getting the WDS proxy working because of the local dev ssl setup.

Lets say my domain is coolapp and prod site is at https://app.coolapp.com. Pre webpack, my development url is https://local.coolapp.com. And it's routed through nginx to my flask server running on http://localhost:8000.

So now i have to run WDS, and make it so my dev url is still https://local.coolapp.com. Then WDS needs to send me to nginx and then on to flask. Which i can do after great nginx pain. But even then, WDS is giving me it's in memory compiled index.html, not the one i need processed by flask/jinja.

So yeah, it's good fun. Even writing that out was painful : ). Thanks for any help!

[davidism]

Closing this for now. Based on all the issues I've had maintaining the dev server, reloader, and CLI for Flask itself, I really don't think more complexity in Flask is the right answer. This would probably be more appropriate as a separate package for managing a WSGI + other services dev environment.

[j-walker23]

Thanks, @davidism.

FYI to anyone landing here. I solved this with https://github.com/shellscape/webpack-plugin-serve