sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.12k stars 1.29k forks source link

Search 3.20 Tracking issue #13065

Closed nicksnyder closed 4 years ago

nicksnyder commented 4 years ago

Plan

@keegancsmith will be working on an initial implementation of streaming search. This is a backend change which gives us more flexibility. Immediately it will improve time to first result for slower searches other issues we have with timeouts. In general it will improve the user (and machine) experience around searches which are not interactive (unindexed, diff/commit, etc).

@rvantonder is taking some time off this iteration and will be doing less. He'll (1) ship search expressions and (2) enable the new query parser for 3.20 onwards. These will build a foundation for expressing new queries and enable future extensions for long-awaited features to pipe, filter, and customize powerful searches for understanding code.

@stefanhengl will wrap-up his work around @ and rev: and then move on to adding support for negation to searcher and, if time allows, introduce the concept of .sourcegraphignore to our search engines.

@limitedmage and @attfarhan will finalize planning for the Enterprise Home feature and file necessary issues related to this, then start working on implementation.

Availability

If you have planned unavailability this iteration (e.g., vacation), you can note that here.

@rvantonder proposed time off: 31 August to 9 September (8 days off) @limitedmage holiday: 7 September (Labor Day)

Tracked issues

@Unassigned

@attfarhan: 1.50d

@keegancsmith

@limitedmage

@rvantonder: 3.00d

@stefanhengl: 10.00d

@uwedeportivo

Legend

keegancsmith commented 4 years ago

Planning done. I added only one issue: Stream based search API #13067. I'm sure I'll have many other things come up along the way, so this is why it is scoped to just this goal. I plan on taking one day off this iteration, but unsure when.

rvantonder commented 4 years ago

My planning is done, but the tracker issue action is failing. When it's working again the issue should update.

I plan to take some time off this iteration (see edit to description), so will be doing less. I will focus mainly on adding search expressions https://github.com/sourcegraph/sourcegraph/issues/13126 (which should mostly be done this week, since I actually wanted this feature flagged already), and then remaining fixes for changing the parser by default in 3.20 https://github.com/sourcegraph/sourcegraph/issues/13128. Then, there are smaller pieces of work for alerts and such, time allowing.

nicksnyder commented 4 years ago

@rvantonder I see you added a few unassigned issues to the milestone. I assigned one to @limitedmage because it seemed like it would be a good onboarding task. For the other two, @stefanhengl might be the best candidate. @stefanhengl can you take a look and see if you think it would make sense for you to do either of those this month (including assessing whether they seem important to do)? If so, assign them to yourself and if not remove from the milestone.

rvantonder commented 4 years ago

@rvantonder I see you added a few unassigned issues to the milestone

Yes--should have clarified: these are just work item proposals after I looked at some of our open issues that could be nice to work on (important, not urgent). Next time I'll explicitly tag those who might be interested and they can either remove it from the milestone or assign themselves.

@stefanhengl #6870 was another onboarding task I thought had a good amount of impact, and #13110 came up recently which is just to support negating repohascommitafter for now. Feel free to pick up or remove those from the milestone. You can also pick them up and consider them lower priority for this iteration.

limitedmage commented 4 years ago

Last week I joined the search team and continued to ramp up on the codebase by fixing a few minor issues (show alerts before results, navbar search button not working, show tab keyboard hint). I also started working on the Enterprise Home feature: synced up with the rest of the team on this and started writing an RFC.

Next week I'll continue fleshing out the Enterprise Home RFC and start planning out the work required to implement it.

attfarhan commented 4 years ago

Last week Mostly focused on getting the search tour merged. Additionally, spent time in planning meetings for enterprise homepages for next iteration, and in 1-on-1s to help with search goals. Also spent time fleshing out a solution for Stanford repogroup pages with Mimi and Beyang.

This week Kicking off enterprise home page planning and work with Juliana.

rvantonder commented 4 years ago

Last week

Finished planning, triaged and closed some older and outdated issues. Continued work on search expressions. Reviewed some user-testing walkthroughs for our proposed wildcard syntax. I also spent some time on structural search examples and have some good content for an impromptu blog post.

This week

Main item: merge search expressions PR and document some examples. I'll probably write an impromptu blog post about some structural search things (stay tuned). Then there are some things I want to tidy in the parser code (we'll be making that the default in 3.20).

keegancsmith commented 4 years ago

Last week I had a single goal of streaming search on Sourcegraph.com. This quickly changed direction. I ended up exploring some refactorings since our search codebase is so tied up into graphqlbackend. This hasn't had a happy ending yet, but spending today on it again.

I spent a significant part of the week hacking with people and then following where that took me. Best win was shaving ~200ms off search queries with repo filters on Sourcegraph.com search #13171.

This week Same goal as last week. Streaming search on Sourcegraph.com behind a user configuration flag.

stefanhengl commented 4 years ago

Last week

I spent most of last week investigating, extending, and fixing our revision syntax. @rvantonder and I added rev: as a new keyword (#13206, #13133) and I fixed a couple of bugs we had related to revisions (https://github.com/sourcegraph/sourcegraph/pull/13271).

This week

I will improve the user docs for our revision syntax and start working on adding support for negation to searcher.

nicksnyder commented 4 years ago

Please update the # Plan section with a sentence or two that describes your goal for the iteration.

@attfarhan @limitedmage @keegancsmith @rvantonder @stefanhengl

stefanhengl commented 4 years ago

Last week I added support for negated patterns to searcher. Now, users can use NOT and -content: on indexed and unindexed repositories. Apart from that, I improved the user docs w.r.t. our revision syntax and fixed (not yet merged) a bug where ref-globs (part of revision syntax) were not resolved for results of type "repository" leading to unexpected behavior.

This week I will probably spend most of the week on adding support for .sourcegraphignore (or similar). Additionally, I would like to define possible next steps for the wildcard syntax and add documentation for our new keyword, rev:.

attfarhan commented 4 years ago

Last week I spent significant time last week on planning work, reviewing several RFCs around saved searches, and continuing to iron out details for the enterprise homepage. I also worked on and merged several PRs to address QA feedback for the onboarding tour. I also worked on an improvement to tracking issue tool to support issues with multiple assignees.

This week I'll spend this week on enterprise homepage implementation, specifically working on implementing the repositories and saved searches panels.

limitedmage commented 4 years ago

Last week I finalized the enterprise homepage RFC and filed all the planned work items to start implementation. I started coding the feature and implemented some of the first infrastructure-related changes.

This week I plan to continue working on the enterprise homepage and start bringing some of the key features to life, specifically the recent searches and recent files.

keegancsmith commented 4 years ago

Last week Was AFK most of the week due to being sick, and then having to deal with the rest of my family also getting sick. Didn't have any good focus blocks, so spent the time I did have reviewing or tackling smaller tasks. What I did ship that I like is using fsnotify for watching config file overrides.

Previous week Investigated and fixed the goroutine spikes plaguing production (root cause was saved search + diff search limits). The next big chunk of my time was spent hacking with others. I did 7 hack sessions, I am scaling them back a bit now since it impacted my main focus of streaming.

This week Focus on shipping for streaming. Starting shipping much smaller and PoC quality changes to ensure forward progress. Scaled back hack sessions. See plan at https://github.com/sourcegraph/sourcegraph/issues/13067#issuecomment-688350371

stefanhengl commented 4 years ago

Last week Added support for .sourcegraph/ignore to zoekt, fixed a bug where we wouldn't show any results for repo-searches, and documented our new keyword rev:.

This week I want to add support for .sourcegraph/ignore to searcher and update the documentation and changelog accordingly. After aligning with Keegan on Wednesday, I will probably start working on performance-related issues.

limitedmage commented 4 years ago

Last week Completed infrastructure work for the enterprise homepage panels and started work on implementing the recent searches panel

This week I will continue working on the recent searches panel and the rest of the enterprise homepage features.

keegancsmith commented 4 years ago

Main: Context switched capturing lines of code metrics which helps unblocks an important deal. This was the task I spent the most time on this week. Made good progress on streaming search, so close to it working end to end.

Other: Planning with Stefan on search performance and search dataset in honeycomb.

Retro: Felt like a really productive week, but still didn't achieve my goal around streaming search. Roughly tracked my work: 25% on capturing LoC task, 20% on streaming search, 20% on meetings and hack sessions, 20% on slack and rfc/code review, 15% on other code tasks. The best work I did was on the day I had no meetings/hack sessions or bigger code reviews/etc. I could fully focus for a full day on just tracking lines of code. I haven't had a day like that yet for streaming search. Next week I will cancel as many hack sessions/meetings as possible to get that again.

Next: Streaming search v1. The most amount of work is in the webapp which I intend to pair with on someone (will ask in sync). Additionally there will be lots of tweaking on the protocol/etc which will be informed by the webapp. Will likely also spend some time on perf work with Stefan.

attfarhan commented 4 years ago

Last week Implemented recent repositories and saved searches panels for the enterprise homepage.

This week Get the recent repositories panel merged, and tie up any loose ends (any bugs that arise, code quality) with the enterprise homepage.

stefanhengl commented 4 years ago

Last week Implemented .sourcegraph/ignore for searcher. This took longer than anticipated because we had to make sure to find a solution that doesn't put too much burden (CPU or memory) on searcher. I started working with Keegan on performance. We created a new service, search-blitz, that will provide us with a baseline for our search performance.

This week Deploy search-blitz, determine the first groups of representative queries, align with Keegan, and start working on the first performance improvements.

davejrt commented 4 years ago

Dear all,

This is your release captain speaking. πŸš‚πŸš‚πŸš‚

Branch cut for the 3.20 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

Thank you

limitedmage commented 4 years ago

Last week I continued working on the enterprise home panels and finished most of the implementation.

This week I will do some polishing on the enterprise home panels and start evaluating the work required for code monitoring.