Add skip indexing phase options to hiedb

josephsumabat commented 11 months ago

Being able to skip certain phases of indexing can be very useful in practice - particularly skipping the type refs phase which is a massive bottle-neck.

Skipping indexing type refs on Mercury's codebase takes the indexing from over 7 minutes to under 50 seconds while maintaining most IDE features. Being able to do this as part of an argument would be incredibly useful. Notably I think that phase could be sped up outside of this but I think this feature would be useful as well.

josephsumabat commented 11 months ago

@wz1000 seems like CI is broken on master

wz1000 commented 10 months ago

TBH, 7 minutes doesn't sound that bad to me for an initial index of a project the size of Mercury's. Future indexes will be incremental and shouldn't take nearly as long if you aren't modifying all the files in the codebase or substantially changing something deep in the dependency graph.

josephsumabat commented 10 months ago

Thanks!

TBH, 7 minutes doesn't sound that bad to me for an initial index of a project the size of Mercury's. Future indexes will be incremental and shouldn't take nearly as long if you aren't modifying all the files in the codebase or substantially changing something deep in the dependency graph.

In practice though:

We have one bottle-neck module in particular that takes over 60 seconds that is very high up in the module graph (though this is an outlier)
People pull from our main branch relatively often and that will typically require a full re-index
most IDE features avoid using the types table (you could read the hie file directly to get the type at point) so avoiding it is very beneficial having it be less than 1/7th the time is pretty nice

I think on less performant machines it hits something like 15 minutes as well and this number is only growing for us so having the option is very helpful!

josephsumabat commented 8 months ago

Went ahead and rebased to fix the merge conflicts on this if you're still open to including this feature! I have a separate branch that maintains the interface of addRefsFrom too if that's something you'd like to maintain

wz1000 commented 7 months ago

Thanks @josephsumabat. I'll include this in the next release.

wz1000 / HieDb

Add skip indexing phase options to hiedb #62