search

seneubert / maven-review

Proposal for the methodology of a 2-tiered collaborative review process of scientific analyses.
4 stars 1 forks source link

Suggestion: Bring in reviewers only when substantial work has been done #6

Open seneubert opened 9 years ago

seneubert commented 9 years ago

Feedback by Mike S.: Before the Maven is assigned at least an analysis plan has to be there. Need not be "day one" of the analysis. Tier 2 review need not be done in parallel to the working group review. Ideally when they come in the analysis is in a state, through the work of the tier 1 reviewers, that tier two can proceed quickly. Intermediate steps of the analysis should be reviewed by "external" reviewers if there is an important milestone which you don't want to change later, such as the final selection in a blind analysis, before you unblind. In such a case it makes sense to get the selection approved by external, unbiased reviewers. In BaBar there was a successful model: for the unblinding approval a member of the WG who was not involved with the analysis up to that point (i.e. NOT the maven) does the review and approval to go to unblinding. This reviewer then also becomes an additional member of the tier 2 review committee.

Take into account that there is a rule in LHCb the one should only be on one review committee at a time. People expect "substantial" amount of work to be done before they start reviewing. They want enough material for discussion.

seneubert commented 9 years ago

I appreciate that it makes not much sense to get a reviewer assigned before the analysis has started. However, one of the cornerstones of the presented concept is that the review process is intentionally broken down into smaller, manageable chunks. I maintain that reviewers need to be available before the analysis has produced results. They should review the analysis strategy as laid out in the backlog. An obvious case is met when you have a blind analysis + blind approval.
Another issue is, that while in an ideal world the tier 2 reviewers should not have a lot of work, in practice this does not happen (as explained in the proposal). The current proposal accepts this reality and tries to make the best out of it.

In summary, I guess I simply disagree with Mike on this point.

The experiences of BaBar with a Working group reviewer continuing as one of 3 members on the collaboration review committee are very interesting. This is already very similar to my proposal where the tier1 reviewer is similarily meant to be available throughout the tier2 review.

betatim commented 9 years ago

We need a balance between there not being anything to review and everything being done. Taking into account that the amount of time/effort required to review 2*N units of work is more than two times higher than reviewing N units of work, you want to start early to have a chance to do useful reviewing.

Not sure what Mike calls "analysis plan" (which is his minimum to start reviewing), to me it sounds like a small hurdle to clear: "have you thought about what you want to do, and written it down?", "yes", "good let's start the review".

apuignav commented 9 years ago

After reading the document I was going to suggest the same thing as Mike did, but I would say it in a different way: when is an analysis started? When it's first shown in the WG, or when you start making the ntuples?

Sometimes we make ntuples, start playing a bit and we want to be sure we know where we can go or if we can actually go where we want, and for that you don't want a formal structure. I think the risk of this review procedure, which I like, is to make things too rigid in places where they shouldn't be. There should be some space for playing around, even when you have a very well laid-out idea, and biweekly meetings and a backlog is not encouraging this.

Maybe a solution is to have the WG conveners judge when an analysis is "mature enough" (whatever this means) to include a WG reviewer. I think this gives enough freedom to the analysts at the beginning, while at the same time leaving margin to incorporate the WG reviewer early enough.

seneubert commented 9 years ago

I agree. "From day one" means as soon as the analysis is being presented in the WG. Of course people are free to try things out on their own.

Having meetings biweekly is just a suggestion. I should make this more clear. There should be no formal rule on this. There will also be periods of time where a lot get's done and one wants to sync more often. And there will be times when things slow down. All this is no problem.

seneubert commented 9 years ago

I think when to bring in the collaboration reviewers is actually the most controversial point.

Having spoken to people who design other review procedures I maintain my original point that they should be brought in early. The review should run in small, manageable iterations while the analysis is build up by the team. Getting rid of the big final review at the end when all is done already is the main point of this proposal. All the other things, public code, a public analysis logbook, a reviewer looking at the technical side of things are important and nice, but they are only there to enable and facilitate this one crucial change of how we think about a review.

apuignav commented 9 years ago

I agree, but I think this is what is going to get the most resistance. In my experience, a lot of reviewers are not "happy" to be doing the job; even if most of them do it fast and in a reasonably enthusiastic manner, being a reviewer is considered by many as "a waste of time" (even if everybody agrees reviewing is necessary). I've seen more than one case in which one of the reviewers was not even participating in the review at all, but that is a separate issue.

Asking people to enter in a review even earlier in time, even if it represents a small chunk of work every week instead of a big one in a single seating, may find a lot of resistance. In this sense, I think reviewers should be introduced as early as possible, but not too early, so the least possible resistance is met. Afterwards, we can start bringing them in earlier and earlier :-)

seneubert commented 9 years ago

@apuignav What would the optimal time be in your opinion?

I suspect this attitude among the reviewers (which I have not experienced) comes from the fact that they feel (if they don't understand it) that a review after all work has already been done is a outdated model. Of course they are unhappy getting dropped a 120 pages ana note on their desk and are now supposed to give expert opinion on that work. When they just hear "spend more time on this" there will be resistance.

It is the challenge of this proposal to make people see that by chopping the work up into manageable chunks it will be more engaging for the reviewers and allow them to make a more meaningful contribution to the analysis. This is the change of mental model that we aim for here.

seneubert commented 9 years ago

I have created #8 as a possible solution to the question, what should be done before the tier 2 reviewers are called in.

apuignav commented 9 years ago

@seneubert I think what we were discussing of the first RD presentation is a good moment to look for the reviewers.

I agree completely with the other points: in fact, I think the new proposal is much better in terms of the amount of time you have to spend and in the quality of the review, but I wanted to point out that I am 99% sure people will complain a priori that the new procedure will take much more time. We should be ready to face this resistance, maybe there are some studies on the subject?

seneubert commented 9 years ago

Yes there are tons of studies on this. I am looking at the section on "overcoming resistance" in http://www.succeedingwithagile.com/chapter-6-overcoming-resistance This table from a 2007 study shows the primary reasons for resisting change:

Number Employees Managers
1 Lack of awareness Fear of losing control and authority
2 Fear of the unknown Lack of time
3 Lack of job security comfort with status quo
4 Lack of sponsorship No answer to "What's in for me?"
5 No involvement in solution desgin

quite interesting actually!

Another study categorizes individuals according to their disposition to change

Conservers Pragmatists Originators
25% 50% 25%
Prefer change that maintains the current structure Prefer practical change Prefer change that challenges exisiting structures
Enjoy predictability Open to both sides of the argument Will challenge accepted assumptions
Honor tradition and established practice More focused on results than strcuture Little regards for accepted policies

Obviously it will be easier to convince pragmatists. So we need to identify those.

Basically resistance can be grounded on two stance:

Examples of Liking the status quo:

Examples of Not Liking the new Process:

Mike cohn in the above mentioned book then further categorizes resistors in four quadrants

How they resist Like Status Quo Don't like proposal
Active Diehards Saboteurs
Passive Followers Skeptics

He then suggests how to approach the different types of people. I her cite only the ideas I like the most:

Skeptics:

Saboteurs:

Diehards:

Followers:

Finally as we already try to do here is to understand resistance as something useful that allows us to hone the proposal and discover problems we might have missed.

apuignav commented 9 years ago

Brilliant! This is what we need :-)

On Fri, Oct 2, 2015 at 3:21 PM, Sebastian Neubert notifications@github.com wrote:

Yes there are tons of studies on this. I am looking at the section on "overcoming resistance" in http://www.succeedingwithagile.com/chapter-6-overcoming-resistance This table from a 2007 study shows the primary reasons for resisting change: Number Employees Managers 1 Lack of awareness Fear of losing control and authority 2 Fear of the unknown Lack of time 3 Lack of job security comfort with status quo 4 Lack of sponsorship No answer to "What's in for me?" 5 No involvement in solution desgin

quite interesting actually!

Another study categorizes individuals according to their disposition to change Conservers Pragmatists Originators 25% 50% 25% Prefer change that maintains the current structure Prefer practical change Prefer change that challenges exisiting structures Enjoy predictability Open to both sides of the argument Will challenge accepted assumptions Honor tradition and established practice More focused on results than strcuture Little regards for accepted policies

Obviously it will be easier to convince pragmatists. So we need to identify those.

Basically resistance can be grounded on two stance:

  • They like the status quo
  • They don't like the new proposal

Examples of Liking the status quo:

  • I like who I work with.
  • I like the power / prestige that comes with my current role (e.g.convener, reviewer).
  • I was trained to do it like this and it is the only way I know how to do it.
  • I don't like change of any sort.
  • Change initiatives always fail anyways

Examples of Not Liking the new Process:

  • I think it's a fad and we will have to switch again in three years
  • The process is bad for our analysis output/quality
  • I got into this field so that I could get headphones on and not talk to people
  • Highly collaborative concepts don't work with distributed teams like ours

Mike cohn in the above mentioned book then further categorizes resistors in four quadrants How they resist Like Status Quo Don't like proposal Active Diehards Saboteurs Passive Followers Skeptics

He then suggests how to approach the different types of people. I her cite only the ideas I like the most:

Skeptics:

  • Provide training / awareness
  • Appoint a champion skeptic
  • Put the skeptic in charge of a part of the transition to the new idea

Saboteurs:

  • Show off your success
  • Fire them
  • Form thriving communities that communicate that they want change

Diehards:

  • Acknoledge and confront fear
  • Create dissatisfaction with the status quo

Followers:

  • Involve them
  • Provide a model yourself how change can work
  • Identify what is the true issue that makes them stick to the status quo (awareness?)

Finally as we already try to do here is to understand resistance as something useful that allows us to hone the proposal and discover problems we might have missed.

— Reply to this email directly or view it on GitHub https://github.com/seneubert/maven-review/issues/6#issuecomment-145018772 .

Dr. Albert Puig Navarro Laboratoire de Physique des Hautes Energies Ecole Polytechnique Fédérale de Lausanne (EPFL) BSP 614.4 (Cubotron UNIL) CH-1015 Lausanne EPFL Phone: 021 6939808 CERN Phone: 72518