sourcegraph / srclib

srclib is a polyglot code analysis library, built for hackability. It consists of language analysis toolchains (currently for Go and Java, with Python, JavaScript, and Ruby in beta) with a common output format, and a CLI tool for running the analysis.
https://srclib.org
Other
942 stars 62 forks source link

Change srclib metadata import to write new refs/defs/indexes in serial. #272

Closed samlavery closed 8 years ago

samlavery commented 8 years ago

This change removes the parallel writes that were in place previously. This is only a change at the source unit level, multiple source units can still be stored in parallel, but this change makes the writes atomic within a single source unit. In theory the old parallel writes should have made import faster, but on slower storage systems, this was causing write contention and data corruption, slowing things way down. Serial writes eliminate this corruption.