Support side-by-side compressed files?

[shepmaster]

If I have a directory with the following contents:

foo.css
foo.css.gz

It would be nice to serve the gzipped version when the client asks for foo.css. Does this seem like something that is in-scope for this library?

[reem]

I think that's probably too much magic, unless we can figure out a more general way to express this idea.

[untitaker]

I don't think staticfile should simply assume that those two files have the same content. What's wrong with compressing on-the-fly?

[shepmaster]

What's wrong with compressing on-the-fly?

Nothing is wrong with it, but there has to be increased overhead (not that I've benchmarked it though). If someone uses a compression algorithm like zopfli, which uses more time to compress in exchange for smaller files, then the delay would be even more noticeable. Additionally, compressing the same static file millions of times is inefficient compared to compressing it once.

On-the-fly compression is invaluable for operations where the server is generating the content. Ahead-of-time compression is useful for static content.

[untitaker]

From what I understand most of this most performance concerns are obliterated by implementing the X-Sendfile header and offloading everything to the reverse proxy that is likely to sit in front of servers like Hyper's (nginx).

[shepmaster]

the reverse proxy that is likely to sit in front of servers like Hyper's (nginx).

Ah, I was not aware that Hyper / Iron had no desire to participate at this layer of the stack. I can certainly replace or supplant Hyper / Iron with nginx. That does seem to place the staticfile middleware in a strange place though, if serving those types of files is better served by a different piece of software.

[untitaker]

X-Sendfile is particularly popular with webframeworks in slower dynamic languages as there is little point executing some Python code just to serve static files. I don't know if Rust as a systems programming language is in a similar position, but it seems wasteful to reimplement support for compression, range requests, ETags, If-Match, ... in Rust when all this can be done in a framework- and language-agnostic way, at least for the subpaths where it does matter.

In Flask [1] it is implemented the following way: By default, static assets are served by the application itself, such that the development server itself is functional (but slow). In production you would turn on X-Sendfile support which makes Flask's implementation, as you say, almost a no-op. But it still has a usecase. It's just that I don't think we should spend a lot of time on performance optimizations on it.

[1] http://flask.pocoo.org/

On Sun, Sep 11, 2016 at 09:12:19AM -0700, Jake Goulding wrote:

the reverse proxy that is likely to sit in front of servers like Hyper's (nginx).

Ah, I was not aware that Hyper / Iron had no desire to participate at this layer of the stack. I can certainly replace or supplant Hyper / Iron with nginx. That does seem to place the staticfile middleware in a strange place though, if serving those types of files is better served by a different piece of software.

You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/iron/staticfile/issues/74#issuecomment-246188413

[shepmaster]

It's just that I don't think we should spend a lot of time on performance optimizations on it.

This is 100% a valid thing to aim for, but I would suggest updating the README with text along these lines to help people understand from the start that this project has an explicit non-goal of replacing solutions such as Nginx etc. when it comes to performance.

it seems wasteful to reimplement support for compression, range requests, ETags, If-Match, ... in Rust when all this can be done in a framework- and language-agnostic way, at least for the subpaths where it does matter.

From my end of the stick, that's part of the appeal of Rust — reimplementing things. Specifically, I would have wanted Iron (perhaps powered by staticfile) to be able to replace Nginx for this specific usecase (and more and more, eventually). Again, if that's not what the maintainers want, that's absolutely fine, but stating it upfront should help fend off future feature requests.

Philosophically, it does get tricky when this is extended to other aspects, such as SSL termination. SSL can be provided by Nginx as a frontend before Iron, yet Iron appears to support it (I haven't used it myself).

[untitaker]

This is 100% a valid thing to aim for, but I would suggest updating the README with text along these lines to help people understand from the start that this project has an explicit non-goal of replacing solutions such as Nginx etc. when it comes to performance.

This is my personal perspective, not sure if it's in line with Iron's.

From my end of the stick, that's part of the appeal of Rust — reimplementing things. Specifically, I would have wanted Iron (perhaps powered by staticfile) to be able to replace Nginx for this specific usecase (and more and more, eventually). Again, if that's not what the maintainers want, that's absolutely fine, but stating it upfront should help fend off future feature requests.

Perhaps I'm routine-blinded, but high-level web application frameworks in general are not designed for the purpose of implementing a web server. That isn't to say that a simple nginx-clone couldn't be implemented in Iron though.

Flask's builtin server doesn't scale at all, is vulnerable to a wide range of DoS attacks (single-threaded server). For production, other server implementations are used. However, it does have SSL support (primarily for testing things such as HTTPS redirect behavior), and despite all the listed shortcomings, it is sometimes used for low-traffic intranet sites just because it's already there. Stable Hyper is also just a synchronous threaded server and unlikely to be any better. But development Hyper is different in ways I don't fully understand, so maybe the situation will be completely different and maybe sometime Iron can be used for large-scale websites without nginx.

[shepmaster]

Perhaps I'm routine-blinded, but high-level web application frameworks in general are not designed for the purpose of implementing a web server

And perhaps I might be blinded the other way 😉 Perhaps the "right" solution is to implement something like this atop Hyper (or whatever Tokio grows into) and continue to put it before something more application-focused.

Flask's builtin server doesn't scale at all, is vulnerable to a wide range of DoS attacks (single-threaded server). For production, other server implementations are used. However, it does have SSL support (primarily for testing things such as HTTPS redirect behavior), and despite all the listed shortcomings, it is sometimes used for low-traffic intranet sites just because it's already there.

Ditto for Rails. You generally switch to an alternate and put Nginx in front, as you mentioned.

However, that doesn't necessarily strike me as The Way Things Must Be. Having never implemented a production-quality web application framework or web server, my opinion doesn't carry much weight, but I'd love it if Rust could basically consolidate those two layers of the modern stack.

[serprex]

In another project I resolved this by caching the compressed version. Most clients will support gzip so falling back to reloading the file for clients who will only accept plaintext seems reasonable (In the project I've linked I opted to ignore checking the Accept-Encoding header, a shortcut decision that isn't recommended for a general purpose library obviously). This has the bonus of reducing the size of the cache

[shepmaster]

I also implemented this myself for the playground