Add registry indexer

[davidanthoff]

Very WIP. Some things that still need to be done:

• [x] Only index and save one package per call to the main script
• [x] Fall back to older Julia versions if there is an error (deferred)
• [x] Record failed trials so that we can skip things
• [x] Skip things that are already cached
• [x] Come up with a file structure for the cached results
• [x] Figure out how to handle stdlib and base (deferred)
• [x] Don't write separate cache versions for different Julia versions
• [x] Compress things

[ZacLN]

I'm not sure of the benefits of changing how we store the caches for Base/Core at the moment outweigh the costs (i.e. within the SymbolServer module, cached as part of the compilation process). Loading from disc would be slower. Arguments for doing this would include running type inference on all methods which is almost certianly too time consuming to be the default option for most users.

I can see the possibility in the future of wanting to use SymbolServer loaded in a vXX Julia process as part of an environment representing Julia vYY, but we're not there yet.

We also have the option of caching the stdlibs alongside Base/Core in the SymbolServer module.

[davidanthoff]

I'm not sure of the benefits of changing how we store the caches for Base/Core at the moment outweigh the costs (i.e. within the SymbolServer module, cached as part of the compilation process). Loading from disc would be slower. Arguments for doing this would include running type inference on all methods which is almost certianly too time consuming to be the default option for most users.

Ah, great, so we can skip that for now!

I can see the possibility in the future of wanting to use SymbolServer loaded in a vXX Julia process as part of an environment representing Julia vYY, but we're not there yet.

Agreed.

We also have the option of caching the stdlibs alongside Base/Core in the SymbolServer module.

The downside of that is probably that during precompile we would now precompile all the stdlibs, right? But maybe the best strategy for now to just get all of this out, right?

[davidanthoff]

I'm going to merge this as is, given that it can't break anything. And then we can iterate with more improvements in future PRs.