Warn on very large assets

[Michael-F-Bryan]

I accidentally tried to include about 400MB of stuff the other day and rustc ended up using stupidly huge amounts of RAM trying to compile it all. It's probably a good idea to emit a warning if the embedded assets will be larger than about 100MB or so...

[ExpHP]

Even 100MB is way too big.

Today I had a reported issue where rustc used over 8GB of memory because a single 12MB file was misplaced into the directory.

[Michael-F-Bryan]

I'm quite surprised. If you look at the implementation, files are embedded by creating a normal &'static [u8]. I'm not sure why it's using so much memory when this is just another constant...

[ExpHP]

I do admit it that it is a fairy simple implementation (and I would not have expected the immensely popular quote! macro to have such problems!), but I can confirm on my own machine that stable or nightly, release or debug all use over 8GB of memory to embed a 15MB file.

On linux:

# Make a 15 MB file
dd if=/dev/zero of=src/item bs=1M count=15

# Limit this shell process and all children to approx 8GB for ease of testing
ulimit -v 8000000

cargo build
# eventually dies with 'memory allocation of 171872600 bytes failed'

I notice that quote is a macro_rules! macro, which surely must expand to many smaller helper macro invocations. Perhaps this is the root of the inefficiency? I haven't looked too far into it.

---

What about having the proc_macro expand to a call to include_bytes? Something like:

impl ToTokens for File {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let path = self.path.display().to_string();

        let tok = quote!{
            File {
                path: #path,
                // FIXME: might need adjustment to account for CARGO_MANIFEST_DIR
                contents: include_bytes!(#path),
            }
        };

        tok.to_tokens(tokens);
    }
}

I'm going to try playing around with this.

[ExpHP]

Okay, so I tried my suggestion above, and:

• changing it to include_bytes! drops the memory usage to "only" 700MB. (but this is is still a tenfold decrease, and it's probably the best we can get!)
• unfortunately, rustdoc tries to render the whole thing as one giant b"\x00\x00\x00\x00\x00... without line breaks which puts immense strain on the browser; but I think that's really a problem that ought to be fixed in rustdoc. (and it's largely due to the fact that the file I generated contains no whitespace characters, which is unnatural)

[ExpHP]

Sorry for hijacking this issue, by the way. I've created #33 as a separate actionable item or for continuing this discussion.