Code Intelligence Sprint 5 Tracking issue

[efritz]

Plan

This sprint covers 2021-02-01 through 2021-01-12.

Availability

If you have planned unavailability this iteration (e.g., vacation), you can note that here.

Eric: 10d Noah: 10d Garo: 6d Owen: 8d Olaf: 10d Maria: 10d TJ: 10d

Tracked issues

@Strum355

• [ ] Cross-language API/IDL support (GraphQL, Thrift, Protobuf) (#981)
• [ ] ~Failed LSIF uploads not reflected in uploads UI~ (#18188) 🐛

Completed

• [x] (🏁 18 days ago) Improve debuggability (~#27~)
• [x] (🏁 18 days ago) Emit stacktrace on segfault with instructions for core-dump only scenarios (~#32~) :shipit:
• [x] (🏁 13 days ago) introduce documentation to aid users in the case of a segfault (~#43~) :shipit:
• [x] (🏁 12 days ago) WIP: Documenting lsif-java ft. minor refactorings (~#83~) :shipit:
• [x] (🏁 12 days ago) cross-package type alias definitions have self-referencing semantic definition results (~#130~; PRs: ~#131~) 🐛

@efritz

• [ ] Remove long-running transactions from dbworker (#14920)
• [ ] blog: Write about the code intelligence commit graph optimization effort (#17379)

Completed

• [x] (🏁 19 days ago) Search-based code intel hovers are broken when definition token and docstring are separated by multiple lines (~#17925~; PRs: ~#567~)
• [x] (🏁 18 days ago) Add clarity to MinIO credential settings (~#16412~; PRs: ~#17965~)
• [x] (🏁 17 days ago) codeintel: Ensure txn is used for bulk inserts (~#18020~) :shipit:
• [x] (🏁 12 days ago) cross-package type alias definitions have self-referencing semantic definition results (~#130~; PRs: ~#131~) 🐛
• [x] (🏁 10 days ago) UI: Fix cache key for document highlights (~#17997~) :shipit:
• [x] (🏁 10 days ago) Fix pings for lsif WAUs (~#17923~; PRs: ~#17959~)
• [x] (🏁 10 days ago) chore: Add correct size hints when making slices (~#18211~) :shipit:
• [x] (🏁 10 days ago) chore: Fix order of driver.Value declarations in dbutil (~#18212~) :shipit:
• [x] (🏁 10 days ago) chore: Distinguish stores in error messages (~#18213~) :shipit:
• [x] (🏁 10 days ago) chore: Update bloom filter package interface (~#18223~) :shipit:
• [x] (🏁 10 days ago) codeintel: Add GetDumpsByIDs to dbstore (~#18214~) :shipit:
• [x] (🏁 10 days ago) codeintel: Add BulkMonikerResults to lsifstore (~#18217~) :shipit:
• [x] (🏁 10 days ago) codeintel: Standardize limit/offset parameters in lsifstore (~#18218~) :shipit:
• [x] (🏁 10 days ago) codeintel: Add DefinitionDumps and ReferenceIDsAndFilters to dbstore (~#18221~) :shipit:
• [x] (🏁 10 days ago) codeintel: Add PagedReferences to lsifstore (~#18215~) :shipit:
• [x] (🏁 10 days ago) codeintel: Do not return nil scanner (~#18226~) :shipit:
• [x] (🏁 3 days ago) codeintel: Remove dead code (api package, dbstore, lsifstore) (~#18231~) :shipit:
• [x] (🏁 today) codeintel: Add a request-level data cache (~#15943~; PRs: ~#17964~)
• [x] (🏁 today) codeintel: Batch lsif store requests (~#15945~; PRs: ~#17964~)

@gbrik

• [ ] Disconnect between hover for the same symbol (#39)

Completed

• [x] (🏁 21 days ago) make code intel alerts non dismissible (~#566~) :shipit:
• [x] (🏁 21 days ago) RFC 201: Do not auto-index things being manually indexed (~#14572~; PRs: ~#17893~)
• [x] (🏁 19 days ago) codeintel: prepopulate index configuration field with auto-configurator output (~#17489~; PRs: ~#17736~)
• [x] (🏁 18 days ago) write script to combine multiple LSIF dumps of the same repo locally before upload (~#17960~; PRs: ~#24~)
• [x] (🏁 14 days ago) escape the project root we compare paths against (~#42~) :shipit:

@tjdevries

Completed

• [x] (🏁 18 days ago) ci: Use https:// rather than ssh for clone (~#41~) :shipit:
• [x] (🏁 12 days ago) cross-package type alias definitions have self-referencing semantic definition results (~#130~; PRs: ~#131~) 🐛

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• :shipit: Pull Request

[github-actions[bot]]

Heads up @macraig - the "team/code-intelligence" label was applied to this issue.

[shrouxm]

(going to start updating this as I go since it doesn't happen otherwise)

Week 1 Done

• Made all code intel hover alerts non-dismissible: sourcegraph/code-intel-extensions#566
• Merged PR for prepopulating autoindexer config field: #17736, closed #17489.
• Merged PR to skip auto-indexing for repos with manual indexes: #17893, closed #14572.
• Reviewed RFCs 293, 294, 307
• HRT call!! got an apparently successful LSIF dump produced for a file
• Merged PR for script to combine LSIF dumps: sourcegraph/lsif-test#24, closes #17960
• Opened PR for bugfix around escaping file paths in lsif-clang: sourcegraph/lsif-clang#42

Week 2 Done

• Merged PR for bugfix around escaping file paths in lsif-clang: sourcegraph/lsif-clang#42
• Peer reviews
• Revised C++ related code intel goals
• Opened PR for pulling from upstream in lsif-clang and document the process

Week 2 Planned

• Update docs based on issues found during HRT call
• Investigate a weird range mismatch in precise hovers for lsif-clang (and other langs?) sourcegraph/lsif-clang#39
• Since I have some extra time, maybe write an RFC about some overdue code reorganization we've been putting off

[macraig]

Last week:

• RFC 293
• Customer info aggregation: CE syncs, dashboard scraping, Sales/CE information surfacing project
• Customer reach out and follow-ups

This week:

• Planning prep
• RFCs 294 and 307
• Fill in alignment doc
• Kick off design collab
• Customer follow ups
• OKR review and brainstorming

[olafurpg]

This week:

• Reviewed protoc brainstorm by @Strum355
• Investigated lsif-java and the Kythe Java indexer
• Started working on a new Java compiler plugin based off lsif-java, but a clean rewrite in Java
• Addressed several comments on RFC 307

[macraig]

Last week:

• Code intel OKRs and strategy
• Customers follow ups
• RFCs 294 and 307
• Alignment doc
• Dashboard get back WAUs
• Review cycle
• Design collab draft doc

This week:

• Q1 OKRs alignment and roadmap adjusting
• Continue customer reach out and delivery follow up
• Planning prep
• Chorus AI digging
• Draft mid-long term code intel roadmap

[efritz]

Last week:

• Looked in to database size issues at customer
• Looked in to performance issues at prospect customer (will continue investigating)

This week:

Going to continue working on backend performance optimizations.

[macraig]

Last week:

• PR on updated goals/milestones
• Iteration on roadmap: https://sourcegraph.productboard.com/roadmap/2288108-code-intel
• Continue customer reach out and delivery follow up
• Planning prep
• Some Chorus AI digging
• Hover re-design review

This week:

• Customer followup
• Merge updated goals/milestones
• More Chorus AI digging

[efritz]

This week:

Investigated a few customer performance issues. Refactored the code intelligence resolvers in https://github.com/sourcegraph/sourcegraph/pull/17964 to make it obvious where we need to optimize. Made a bunch of issues with the code-intelligence-performance label.

Next week:

Will plan/prioritize the issues with the label above and start digging in to that for the next few iterations.

[olafurpg]

Last week:

• lsif-java reboot https://github.com/sourcegraph/lsif-java/pull/85

This week:

• Gradle integration for lsif-java

[efritz]

Continued in https://github.com/sourcegraph/sourcegraph/issues/18290.