Optimized serving of static assets of a (bundled) Meteor 1.6 application

[serkandurusoy]

_I believe this can go into the docs repository but given that this issue I'm opening is part of an ongoing discussion at the Meteor forums I'd like to get your official opinion before starting a pull request on the docs, if necessary._

Consider a meteor app with the following layout

|-- my-application
    |-- .meteor
    |-- client // some app directory
    |-- imports // directory containing bundled/dynamically loaded js files
    |-- packages // atmosphere/local packages, possibly containing
    |-- public
        |-- foo // directory/file as public static asset
        |-- bar // directory/file as public static asset
    |-- server // some app directory

When a meteor app is bundled using meteor build command, it creates a directory layout that resembles:

|-- bundle
    |-- programs
        |-- web.browser
            |-- app // maps to the root
                |-- foo // directory/file contained within the meteor application's public directory
                |-- bar // directory/file contained within the meteor application's public directory
            |-- dynamic // maps to the root
                |-- imports // directory containing dynamically loaded js files
            |-- packages // used for loading atmosphere packaged static assets
            |-- body.html
            |-- head.html
            |-- program.json // a map of the files and their root relative urls and some other info (not served to the client)
            |-- xxxxxxxxxxxxx.js // application bundle
            |-- xxxxxxxxxxxxx.stats.json
            |-- yyyyyyyyyyyyy.css // style bundle

So as you see, we don't have the typical nodejs app layout with a public directory, which per the docs we manually create within the bundle directory ourselves.

But then, that directory is empty. And the docs does not mention anything about a requirement for the bundled static assets to be copied over to this directory.

Furthermore, as you see from the rather complicated directory layout mapping from above, there is no single public directory where we can symlink to.

Now, as per suggestion from @dr-dimitru on the forum thread, it is possible to do:

cp ./bundle/programs/web.browser/*.css /var/www/myapp/public/
cp ./bundle/programs/web.browser/*.js /var/www/myapp/public/
cp ./bundle/programs/web.browser/*.html /var/www/myapp/public/
cp -R ./bundle/programs/web.browser/app/* /var/www/myapp/public/
cp -R ./bundle/programs/web.browser/packages /var/www/myapp/public/

yet, this is neither complete, maintainable, nor scalable as meteor's app structure evolves (eg the latest addition of dynamic imports) and now a possible near future merger of the imports directory directly into the app root.

So, my personal impression from my earlier deployments (if you please have the time to skim through that thread for some more context and information) was that as soon as passenger proxies a static asset response without any explicit response headers, it applies any and all "nginx static file serving features" and we would not have to do all this copying of static assets to the public directory.

Now, I doubt that based on the input from that thread.

Nonetheless, it is also evident that this file copying approach - although scriptable - is nowhere maintanable and is kind of a weak spot at passenger's mantra as reiterated in one of your recent blog posts: "...Passenger's philosophy is to put user experience at the forefront. We believe that writing and managing apps should be easy, hassle-free and productive, and that software should serve users..."

TL:DR;

• To be able to let nginx serve static files, do we need to copy all assets to the bundle/public directory
• If yes, should we perhaps update the official library documentation (and the wiki article)
• If yes, how do you think we should reflect this information in the docs? (We are willing to initiate a PR)

[OnixGH]

@serkandurusoy you mentioned that program.json is a (json) map of the files, does this match exactly with the files that you copy to public? If yes, is that a guaranteed match?

The docs indeed need an update as mentioned in the referred thread.

Btw. it's absolutely within our mantra to make things as easy as possible; the Meteor app will still "just work" even with the changed layout, just not as optimized as possible. I certainly see room for improvement in this case, but we would need a safe and robust way to determine what is supposed to be public (the last thing we need is to create a way where private files can accidentally end up public, so it should also be a way that is very obvious for the user).

[serkandurusoy]

Hey @OnixGH sorry for missing this post.

program.json looks like it does contain an extensive list but it does also contain non-public files.

Although, examining a few of them gives me the impression that we can assume that values with the url property are all we need (perhaps even the complete list) and we can ignore the rest.

I also cannot locate the html that constitutes the initial response, although it probably makes sense that I can't, give it may be getting dynamically generated.

[dr-dimitru]

@OnixGH @serkandurusoy parsing program.json and making public only "public" files seems like a great idea, I may write/add needed lines of code to do so. Could someone point me to the point at the existent codebase where this action should be performed?

[OnixGH]

@dr-dimitru awesome that you'd want to contribute. Before starting to code I think we first need to make sure that parsing program.json is the way to go, @serkandurusoy also sees the potential but isn't 100% sure, what do you think?

An additional snag is that we're talking about a bundled Meteor app, which technically means it looks just like a "normal" Node app; I'm not sure Passenger could conclude from the mere presence of program.json that the "meteor workaround of mapping files to public" should be activated.

I could imagine various ways to go about this that need some further thought, for example there could be a new app type meteor_bundled (possibly autodetected with a recognized program.json) that would trigger the workaround, and that a user could override if it was wrongly detected (or set if we don't enable autodetection).

[dr-dimitru]

@OnixGH well, I'm still thinking what it's better to let decide about it to developer. Initially I'm up for better docs with well explained behavior:

• You wish to serve static files with Nginx - move "those files" into "that directory"
• You don't care (as everything works out of the box) - you're good to go, no need to change everything

With explanations or references to comprehensive info about differences, pros and cons, and etc. I'm thinking about it more like it's an optimization step, rather than "setup" step.

If you and Passenger team believe what this should be "magically" managed by Passenger - I can do it.

[serkandurusoy]

There's a file called star.json right beneath the bundle directory whose content looks quite a lot like::

{
  "format": "site-archive-pre1",
  "builtBy": "Meteor METEOR@1.6",
  "programs": [
    {
      "name": "web.browser",
      "arch": "web.browser",
      "path": "programs/web.browser/program.json"
    },
    {
      "name": "server",
      "arch": "os",
      "path": "programs/server/boot.js"
    }
  ],
  "meteorRelease": "METEOR@1.6",
  "nodeVersion": "8.8.1",
  "npmVersion": "5.4.2"
}

This might be a good signal that this is a bundled meteor app!

Oh and by the way, the location of program.json is /bundle/programs/web.browser/

[serkandurusoy]

Regarding the "magic" I think that already is passenger's mantra so it makes sense to me to follow suit.

[OnixGH]

@serkandurusoy interesting; I was wondering what the status of that file was. This meteor blog post seems to recommend depending on that file:

Non-Galaxy deployment options may offer manual, freeform Node.js version selection.
In those environments it’s recommended to use the nodeVersion field from the star.json file
at the root of Meteor application bundles.

So that seems pretty safe for now, is this something that's new in 1.6?

[dr-dimitru]

So that seems pretty safe for now, is this something that's new in 1.6?

No, it's common for a few years already.

[OnixGH]

@dr-dimitru aha, okay, thanks. I'd like to discuss this with the team as well.

It definitely falls into the category of optimization rather than setup, but I also like the idea of having it optimized out of the box. It seems that with star.json we'd have a good way to detect Meteor bundled, but we still need to know how to reliably establish candidate files.. @serkandurusoy any more ideas on getting those locked down? I guess maybe the meteor list?

[dr-dimitru]

program.json let's easily to find publicly available files, take as example this program.json, we look for:

where: client
type: asset|js|json|css

[serkandurusoy]

Hmm what about type dynamic js @dr-dimitru do you know how they're loaded? AFAIK they're loaded through websockets, though that might change with the latest http-2 support. In any case, are they candidates to end up in top level public you think?

[dr-dimitru]

@serkandurusoy dynamic js should remain there it is, as we can mistakenly publish private part of codebase (at least in case of careless developers). Also Meteor needs to work with those files via WebSockets in 99% cases. If it will be changed to http2 we can update Passenger to meet new needs.

[serkandurusoy]

as we can mistakenly publish private part of codebase

those files are afaik public files as they are already candidates to be transferred to the client as is.

the only thing different about them is their transport mechanism, which I mentioned only because I don't know that if that mechanism being websockets/http2 makes any difference.

[dr-dimitru]

@serkandurusoy there is Meteor's "magic" inside. It's not simple requests for exact modules, but all modules Client needs as batch request, except already cached modules. Plus it utilizes auto-update mechanism only replacing updated modules/files in a cache.

And, yes WebSockets with fallback to XHR via SockJS are currently used for dynamic modules.

[serkandurusoy]

Ok, so it is the "batched requests" - which I had not known about - that makes the difference I guess. Good insight. Thanks!

[dr-dimitru]

@serkandurusoy you're right about upcoming shift towards HTTP/2, see this PR, but:

Although the implementation of the /__dynamicImport endpoint is a bit too complicated to allow serving dynamic modules from a CDN, fetching modules individually from a CDN remains a possibility for future experimentation. In other words, how modules are fetched is still just an implementation detail of the meteorInstall.fetch callback.

My guess - same applies to Nginx, due to "magic" of batch requests, sadly we neither be able to cache those requests as each request will be unique.

[bolaum]

This thread is exactly what I was trying to figure out today. Did this evolve? Now meteor has an even more complicated bundle structure with static files for different contexts:

• web.browser
• web.browser.legacy
• web.cordova

For anyone using passenger with meteor with the latest version, what do you copy to the public dir? Does passenger somehow magically parses this structure? The tutorial doesn't mention this problem for bundled apps.

[dr-dimitru]

@bolaum this is part of our Bash deployment script where we copy all assets to public directory, it works with latest meteor (as of today v1.10.1) and modern/legacy JS assets, which is set as root in our Nginx/Passenger configuration:

nginx:

server {
  server_name example.com;

  root /var/www/app-dir/public;
  passenger_app_root /var/www/app-dir;

  # ... other settings
}

during deployment, assuming bundle directory is where we extracted "meteor build" archive:

## create 'public' directory
mkdir -p ./bundle/public

## copy assets
cp ./bundle/programs/web.browser/*.css ./bundle/public/
cp ./bundle/programs/web.browser/*.js ./bundle/public/
cp ./bundle/programs/web.browser.legacy/*.css ./bundle/public/
cp ./bundle/programs/web.browser.legacy/*.js ./bundle/public/
rsync -qauh ./bundle/programs/web.browser/app/ ./bundle/public
rsync -qauh ./bundle/programs/web.browser.legacy/app/ ./bundle/public
rsync -qauh ./bundle/programs/web.browser/packages/ ./bundle/public
rsync -qauh ./bundle/programs/web.browser.legacy/packages/ ./bundle/public

## set access permissions for assets
chmod -R 744 ./bundle
chmod 755 ./bundle

## this step might be redundant
mkdir -p /var/www/app-dir/public/
mkdir -p /var/www/app-dir/programs/web.browser/

## copy bundled app to /var/www/app-dir
rsync -qauh ./bundle/ /var/www/app-dir --exclude=".git"