search

sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.1k stars 1.27k forks source link

Core Services: 3.13 Tracking Issue #7719

Closed tsenart closed 4 years ago

tsenart commented 4 years ago

Status

This tracking issue is approved.

Availability

Period is the January 21st until February 17th. Please put in the days you won't be working and the number of working days for the period.

Planned work

@keegancsmith: 8.00d

@kzh: 6.50d

@mrnugget: 10.00d

@rvantonder: 6.00d

@ryanslade: 8.00d

@tsenart: 3.50d

@unknwon: 16.00d

Legend

mrnugget commented 4 years ago

Here's what I'd mainly like to work on:

Besides that, I also talked to @efritz and @chrismwendt about helping out the @sourcegraph/code-intel team and writing an LSIF indexer for C#. Since writing an indexer takes much more than 5d (our maximum estimation value) and I haven't written one before and Automation is, when in doubt, higher priority for me, I'll treat this as a 20% project in the next iteration and hope to work on it when I find the time:

I'd also be happy to assist @rvantonder with his work on boolean operators and his ideas for a regex-based search/replace campaign type (which could potentially require some architectural changes).

ryanslade commented 4 years ago

After speaking with @mrnugget we decided I'd take a stab at the issues around additional PR details:

Stretch:

I'll try to spike one of them this week so that I have a better estimate of how much work they

We also spoke about potentially adding a button to force a sync. This could be a quick win which allows us to make background syncs less frequent but allow users to sync manually when needed.

tsenart commented 4 years ago

@ryanslade, @mrnugget: Do all of these need spikes to estimate between 0.5 and 4 days? Perhaps together you can roughly estimate each of these without needing so much work on it?

tsenart commented 4 years ago

Please label those issues accordingly also, and ensure the right owner is assigned. (Those seem like roadmap items to me).

ryanslade commented 4 years ago

@ryanslade, @mrnugget: Do all of these need spikes to estimate between 0.5 and 4 days? Perhaps together you can roughly estimate each of these without needing so much work on it?

I'm planning to spike one of them as I think they should be approximately the same size

mrnugget commented 4 years ago

@ryanslade, @mrnugget: Do all of these need spikes to estimate between 0.5 and 4 days? Perhaps together you can roughly estimate each of these without needing so much work on it?

Sure, but since @ryanslade is not familiar with thart part of the codebase yet I think it makes sense for him to at least take a closer look. We can do an estimation round tomorrow/day after to roughly put numbers on them.

keegancsmith commented 4 years ago

I'm at 19 days currently, but need to be at 13.5days. I am putting these 5 days worth as up for grabs / to backlog:

I've just sent out a PR for

which gives me another 0.5d.

So I am now at my target of 13.5 days.

However, there has been some more work added to the milestone which is not reflected in this tracking issue yet:

These each represent one days worth of work. @efritz has volunteered to do #7812, I have to do #7803, and #7804 could possibly be backlogged / picked up by someone else. Given that, I am just going to commit to #7803 which puts me 1 day over budget which I am fine with :)

tsenart commented 4 years ago

@keegancsmith: Do you consider these new issues you mention more important than some of the other issues we already pulled in? Regarding the "up for grabs" issues, I think we'd need to identify an owner now, or else they should be backlogged. Any ideas about who else on the team could do that work in similar time estimate?

tsenart commented 4 years ago

@mrnugget: @kzh got a taste of working on automation stuff with the Bitbucket Server webhooks feature and is asking for more :-) What do you think he could help out with in 3.13?

rvantonder commented 4 years ago

@mrnugget just want to chime in and say that we have a data point from a customer who wants this and I only imagine the demand for 'let me do arbitrary things' will grow. My strong impression is that it is worth prioritizing this over stacked diffs based on product feedback and immediate use cases. I'm pretty sure @christinaforney would be on board, maybe check in with her?

unknwon commented 4 years ago

My 75% availability is ~14days, currently 8.5 days with following items:

Technically, I could take 5.5 more days.

Seems I can take @keegancsmith 's two items.

Added:

Now I have 14.5 days.

tsenart commented 4 years ago

@sourcegraph/core-services: Since we know ensure we have an issue for each item in the tracking issue, you can easily see your personal work in the issues view. Here's @unknwon's work for instance: https://github.com/sourcegraph/sourcegraph/issues?q=is%3Aopen+is%3Aissue+assignee%3Aunknwon+milestone%3A3.13+label%3Ateam%2Fcore-services

rvantonder commented 4 years ago

My proposal:

Total days allocated: 16d (84%). Would like the rest of the days for padding, progress on spike items, or reprioritizing issues.

mrnugget commented 4 years ago

@mrnugget: @kzh got a taste of working on automation stuff with the Bitbucket Server webhooks feature and is asking for more :-) What do you think he could help out with in 3.13?

Well, I'm happy to share! What a coincidence! 😄

If you want, you can pick up these four from me:

I think they're pretty well documented, but there are some fuzzy areas to each of them. If you decide to take them on, let's jump on a call and I'll give you an intro (and show you the official, secret Automation handshake).

Then there is our tech debt list: https://github.com/sourcegraph/sourcegraph/issues/6572 I think, based on 3.12, the biggest problem of those mentioned there are the hard-to-write GraphQL API integration tests.

Besides that I think @ryanslade can also come up with things that @kzh can work on, but that will only happen once Ryan got into the weeds so to say and knows what's what.

If @kzh decides to take these on, that would free me up a lot and give me time to spent on:

  1. the "spike & research" tickets that are high priority
  2. writing documentation, UI polish, help Web team/Ryan/Kevin
  3. helping @keegancsmith with the flaky e2e infrastructure
  4. helping @sourcegraph/code-intel with the C# LSIF indexer
  5. helping @rvantonder with his work on search and Automation (we should set up regular calls, Rijnard!)

What do you think @kzh?

@mrnugget just want to chime in and say that we have a data point from a customer who wants this and I only imagine the demand for 'let me do arbitrary things' will grow. My strong impression is that it is worth prioritizing this over stacked diffs based on product feedback and immediate use cases. I'm pretty sure @christinaforney would be on board, maybe check in with her?

Agree 100%. I created a ticket, put an estimation on it and put it on the top of my list. Working on this PR would also give me much more insight into how to run 'arbitrary things' on the server.

tsenart commented 4 years ago

If you want, you can pick up these four from me:

Good from my side. Just adjust the estimates appropriately to account for @kzh still getting up to speed on this part of the codebase.

tsenart commented 4 years ago

@rvantonder: Can you create a separate issue for this?

Dart LSIF ("20% time") part of parent tracking issue #7625 4d hammer_and_wrench @rvantonder

mrnugget commented 4 years ago

Good from my side. Just adjust the estimates appropriately to account for @kzh still getting up to speed on this part of the codebase.

Done! Assigned the tickets to Kevin and increased estimation. Doesn't mean that we don't jump on a call and talk about them, @kzh :)

rvantonder commented 4 years ago

W02 Plan [2020-01-20]

Last week I felt I had to do some things that I didn't explicitly plan for, because they are important.

This week I must deliver some delayed items:

and I plan to fix up our query parsing to prepare for AND/OR/NOT and squash bugs #7604, #6490.

mrnugget commented 4 years ago

W01 Recap

Last week I didn't do a lot of research and investigate the topics I want to work on 3.13, instead I spent my time on higher-priority things:

W02 Plan [2020-01-20]

My plan for this week:

Besides that I want to

ryanslade commented 4 years ago

W01 Summary

I mostly worked on some regressions introduced by the move to background workers and on backend support for draft campaigns (with help from @mrnugget )

W02 Plan [2020-01-20]

tsenart commented 4 years ago

@nicksnyder, @christinaforney: Please review this tracking issue and give us any input you may have. You can also review it using the issues page for easier filtering and exploration: https://github.com/sourcegraph/sourcegraph/issues?q=is%3Aopen+is%3Aissue+milestone%3A3.13+label%3Ateam%2Fcore-services

unknwon commented 4 years ago

I've removed and added following items because I believe two added items have much higher priority:

- Migrate Bitbucket Server code host authz provider to use the new authz.Store for CRUD permissions #6845 **2d** 🧶
+ RFC 40: Move call to the Authz methods into application layer #7878 **1d** 🧶
+ RFC 40: Add unit tests for GraphQL APIs #7748 **1d** 🧶

W01 Summary

Last week, I focused on shipping RFC 40 with 3.12, which I did.

W02 Plan [2020-01-20]

keegancsmith commented 4 years ago

W01 Summary

The great thing about a weekly plan is the whooshing sound it makes when the week is over and half of it is not done. What I actually ended up working on:

Notably the TLS/SSL work and code host fingerprinter I ended up moving into 3.13 cycle. This was done early on in the week to make time for pressing 3.12 work.

W02 Plan [2020-01-20]

This week my theme is to focus on work which unblocks other work. So anything that is a spike or blocking a customer:

tsenart commented 4 years ago

Done this for you @unknwon

Others: I would like to time-box 1 hour to hack on the internal/cmd/tracking-issue to group items by assignee, right now is so hard to find my own items in the list by eye scan 😢

PTAL #7891

kzh commented 4 years ago

W01 Summary

W02 Plan [2020-01-20]

This week, I will focus on the automation tasks assigned to me. Thorsten and I will sync tomorrow to kickstart this.

tsenart commented 4 years ago

@kzh: Can you please list which tasks you'll be focusing on once you know?

nicksnyder commented 4 years ago

@tsenart how do I know what the relative priority is for each person's work? Is everything in stack ranked order? There are certain things that I want to ensure get done but I don't know how to communicate that to the team. I would be happy to shuffle around the order of checkboxes but I only want to do that if the ordering won't get destroyed the next time the automated update script is run.

@tsenart This security issue is not yet represented in the tracking issue: https://github.com/sourcegraph/security-issues/issues/55 Who owns this?

@keegancsmith volunteered to own e2e tests but I don't see that work represented here. This is important work to get done, so if this team is going to hold the semaphore on that work then I want to see time explicitly allocated for it (and I want the allocation to be generous so we are confident that we will make meaningful progress). I would also like to review the proposed solution. https://sourcegraph.slack.com/archives/C07KZF47K/p1579181724117400

@keegancsmith I am worried that you have too many items assigned to you, even if they are all small. Some of them will certainly take longer than expected and the others won't get done. Can we prune the less important things off your list?

tsenart commented 4 years ago

@tsenart how do I know what the relative priority is for each person's work? Is everything in stack ranked order?

No, there's no order in each teammate's work list. This is intentional, and is meant to complement our efforts to move away from "wishlist" planning towards more accurate planning. As documented in the handbook, all planned work is meant to be finished in each iteration (not 80% or 90%, but 100%).

Thus, relative prioritization of items (what gets done first) within each iteration should be done in a way that increases the chances of team mates finishing everything they planned, not necessarily finishing something to the detriment of another less important thing — if it is planned, it needs to get done.

We (managers), and other teammates should certainly advise on how teammates can approach the task list they have, and sometimes that means asking for something to be done earlier for some valid business reason, but we should strive to prevent such advice from impacting the completion of all planned work (whenever possible).

That advice can come 1:1, 1:N, or through whatever other means — ultimately, teammates are responsible for handling that prioritization input appropriately and keeping track of the order in which they want to do things.

@tsenart This security issue is not yet represented in the tracking issue: sourcegraph/security-issues#55 Who owns this?

I am personally taking this one. Regarding security issues in that repo, I am unsure how to represent them in the tracking issue. Do you think that it is OK to have the title of the issue be public?

nicksnyder commented 4 years ago

Thank you for the added context on prioritization. My only input for this iteration is that all-else-equal, I would like to see the security issues addressed sooner rather than later.

I am unsure how to represent them in the tracking issue. Do you think that it is OK to have the title of the issue be public?

Let's link to them but not inline the title.

tsenart commented 4 years ago

Let's link to them but not inline the title.

@nicksnyder: I implemented this here: #7982. You can see the result in my section in this tracking issue.

unknwon commented 4 years ago

W02 Summary

  1. RFC 40: Move call to the Authz methods into application layer #7878 1d 🧶
  2. authz code logs warnings about Redis returning nil values #7912 0.5d 🐛

I ended up spending some time learning time on 1., but helps my planned items this week.

W03 Plan [2020-01-27]

Side note: 14d/16d left.

Added:

sqs commented 4 years ago

I am deprioritizing (backlogging) these 2 items and their related frontend work because none of the automation private beta customers has told us they are necessary:

@ryanslade @tsenart FYI after rerunning the tracking-issue cmd and updating the issue, @ryanslade went from 15d to 12d of work.

rvantonder commented 4 years ago

W02 Summary

W03 Plan

mrnugget commented 4 years ago

W02 Recap

Pretty busy week with a lot of "invisible" work like reviews, documentation, researching, interviews, etc.

W03 Plan [2020-01-27]

After #8008 has been merged, our focus going forward will not be on running campaign types in Sourcegraph, but rather allowing customers to run arbitrary code.

That's why these two items are now my highest priority:

I want to make the src actions exec as good as it can be, but taking into account that it's better to ship this sooner to the customer rather than later. I want to get it into their hands by mid-week.

ryanslade commented 4 years ago

W02 Summary

W03 Plan

keegancsmith commented 4 years ago

W02 Summary

Finished all the work I planned for last week! "Finished" is relative though, I'll dive into details here to summarise it for interested parties, but it's hidden behind a details tag to prevent wearing out the scroll on your mouse :)

> gitserver should auto-recover from corrupt repository clones #6676 I implemented a way to know if inspecting the stderr is a viable solution. We now have a log on sourcegraph.com for all repos which would be regarded as corrupt with this detection. Once we have collected some data, this can then be hooked up into some sort of auto-recovery logic (later this cycle, 0.5d work). Early signs are that it works well. > gitserver: Rebalance after adjusting replica count #2485 This work was a spike that found some other bugs around how Sourcegraph behaves when adjusting the number of gitserver replicas. The spike work ended up in its own issue (so we could track it) and was fixed ( gitserver: Correctly report if repos are cloning after rebalance #7970). The follow-up work is documented in the original issue. > Index multiple (non-master) branches #6728 This work expanded as I investigated, and turned out to be much tricker that I first thought. Just making our zoekt stack support indexing multiple branches ended up being much tricker due to us using archives everywhere and how zoekt does indexing of branches. I ended up splitting out the work we wanted to do for this milestone into zoekt: support indexing multiple branches in zoekt-archive-index #7930. I contributed a few changes to upstream zoekt, and have branch (with a hacky commit) which implementes this. However, I need to clean that hacky commit up but I've already sunk quite a bit more time than planned into this. So I am deferring this work till later in the milestone to ensure I tackle more pressing issues first.

Reprioritizing

The expanded work from last weeks spikes + taking on e2e work means I don't have enough time for the work I planned. As such I have removed 3 tasks from this milestone:

W03 Plan [2020-01-27]

This week my theme is external code host communication. This was some work I started last milestone, but will do more broad work this milestone:

kzh commented 4 years ago

W02 Summary

Last week was mainly focused towards becoming more familiar with the automation codebase.

a8n: Move "default branch" check to execution of CampaignJob #7725 2d has been removed (context: https://github.com/sourcegraph/sourcegraph/pull/8008#pullrequestreview-347914223)

unknwon commented 4 years ago

W03 Summary

Others

W04 Plan [2020-02-03]

Planned/available = 6.5d/11d

rvantonder commented 4 years ago

W03 Summary

W04 Plan

ryanslade commented 4 years ago

W03 Summary

W04 Plan

unknwon commented 4 years ago

VIM :q! test-run? :trollface:

mrnugget commented 4 years ago

VIM :q! test-run?

It's embarassing how close to the truth this is :( My new Vim-in-Firefox confused me for a second and accidently hit the "Close" button.

W03 Recap

W04 Plan [2020-02-03]

NOTICE: As planned, I'll not be working on Thursday and Friday. If there's something urgent/important, I'll be reachable though.

I'll use the three days this week for to make some progress on

Besides that, I expect to also be

kzh commented 4 years ago

W03 Summary

W04 Plan

rvantonder commented 4 years ago

W04 Summary

W05 Plan

unknwon commented 4 years ago

W04 Summary

W05 Plan [2020-02-10]

ryanslade commented 4 years ago

W04 Summary

W05 Plan

kzh commented 4 years ago

W04 Summary

I became ill towards the end of the week.

W05 Plan

mrnugget commented 4 years ago

W04 Recap

Only worked Monday to Wednesday.

W05 Plan [2020-02-10]

Meeting up in London with @sqs, @tsenart, @ryanslade, @keegancsmith until Wednesday. Due to travel, somewhat limited capacity.

uwedeportivo commented 4 years ago

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.13 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly. When in doubt, reach out!

Thank you

rvantonder commented 4 years ago

W05 Summary

I worked on:

I would have liked to:

I didn't get to this because the implementation issues behind blog post took priority and took an unexpected chunk of time.

W06 Plan

Note: there is too much to do here to get the functionality in for branch cut, but the upside is we can almost certainly just as effectively analyze and improve search latency using sourcegraph.com data.