Takes a long time to build runes which have a large number of labels

[Mi1ind]

Building the landmark_na rune on a macbook air with rune build Runefile.yml took 2h 21m 15s. The build process spent most of it's time on the following process:

[2021-05-26T19:28:34.923Z DEBUG rune_codegen::environment] Executing "cargo" "+nightly-2021-05-09" "build" "--target=wasm32-unknown-unknown" "--quiet" "--release"

This issue persists on a windows laptop as well.

Rune details:

• Rune has 99424 labels which is by far the largest amount (next largest runes we have are mushroom and food with 4128 and 2024 labels respectively).
• Rune size is 55.8 MB, however it is unlikely that the size of the rune is what's causing the issue (Building mushroom rune takes 1m 2s - Its size is 29 MB)

Additional Information: Created landmark_na rune causes runic_mobile app to crash (on both iOS and Android). This has been outlined in issue #23 in the runic_mobile repo.

Discussion Points:

• An idea that @akshr had was to move the labelling processing block out of the landmark_na rune into a second rune and chain them together.
• @kthakore proposed creating a labeling only service where the rune points to the label file in the host device to use for labelling (avoids loading the labels within the rune entirely).

[Michael-F-Bryan]

Copying across my response on slack:

I have a feeling I know what the issue will be... Basically, with 100,000 labels we'll be creating a Rust array literal with 100,000 elements. At that scale, the generated lib.rs will probably be megabytes in size, take forever to parse and the internal data structures managed by the Rust compiler will be massive. Hence why it takes forever to build, or doesn't complete at all.

The way we get around this is by moving it to a file, using include_str!() to embed the file into the binary as a string literal, then doing a bit of processing to turn it back into an array when running the _manifest() function.

The Runefile itself is probably massive, so we may want to figure out a way to move the labels out and into a separate file. A while ago, @kartik and I were throwing around the idea of saying "read this argument from a file". Maybe we should revisit that.

This seems quite similar to several issues I had when writing the include_dir!() crate:

• Pathological compile times (https://github.com/Michael-F-Bryan/include_dir/issues/22)
• Extremely large memory overhead (https://github.com/Michael-F-Bryan/include_dir/issues/33)
• Unbounded memory growth (https://github.com/rust-lang/rust/issues/56900)

[Michael-F-Bryan]

As mentioned in #320,

This should be resolved by #310 because you'll be passing a single (albeit long) string literal to the label proc block.