libexpr: serialize ASTs to avoid parsing many files each time per evaluation

[inclyc]

Is your feature request related to a problem? Please describe.

This is the profiling data for eval my nixos configuration. And there are actually 3450 calls to nix::EvalState::parseExprFromFile, which means we will read the filesystem 3450 times, parse the file using a GLR parser, on each evaluation (e.g. the user types nixos-switch rebuild ...).

Describe the solution you'd like

Serialize ASTs into some files (for example, module files in nixpkgs, we are parsing these files each time, for nixos config), and get ASTs directly from that file, instead of parsing files again. Similar things like pre-compiled headers, or modules, in C++ language.

Maybe we can explicitly make serialized ASTs at first, and introduce caches to libexpr, for backward compatibility.

Describe alternatives you've considered

Additional context

callgrind.out.1466508.zip

Priorities

Add :+1: to issues you find important.

[edolstra]

IIRC, parsing is a relatively small part of the total evaluation time, so caching ASTs probably isn't worth the complexity. An eval cache hit also avoids parsing, so it's probably better to improve the eval cache.