feature/lg3.conf: Not in sync with master branch

[HenrikBengtsson]

The feature/lg3.conf branch has not incorporated updates that have been done to the master branch. It is currently unmergable and needs manual attention to fix. So, don't diverge much more before fixing it - otherwise there's a risk that you'll end up with yet another LG3 project code base.

[HenrikBengtsson]

The fix involves merging master into the feature/lg3.conf branch and the resolving merge conflicts manually.

[ivan108]

I examined conflicts, most of them are caused by a typo error: ${LG3_HOME?}/scripts/utils.sh instead of ${LG3_HOME:?}/scripts/utils.sh This error is observed in master and develop branches, but fixed in feature/lg3.conf branch, which creates conflicts. I guess I need to fix the code in both master and develop branches...

[HenrikBengtsson]

I see. So, a few comments:

If your find mistakes that are not specific to the branch you're working on, then:

1. create an issue for the record - that way it can be tracked and discussed if needed
2. move to the develop branch and fix the mistakes there.
3. go back to your branch and git merge develop.

I guess I need to fix the code in both master ...

We never commit to the master branch. That is our release branch and that one needs extra care. It will only be updated via a develop -> release/<version -> lots of testing -> master merge.

I examined conflicts, most of them are caused by a typo error: ${LG3_HOME?}/scripts/utils.sh instead of ${LG3_HOME:?}/scripts/utils.sh This error is observed in master and develop branches

Hmm... even when using ShellCheck 0.7.0, I don't get an error. make check is clean. How did you manage to get an error/detect this? Interestingly, both ${LG3_HOME?} and ${LG3_HOME:?} produce an error if LG3_HOME is unset. It looks like should be ${LG3_HOME:?} but do you have a reference we can stand on?

[HenrikBengtsson]

Ok, I found the answer regarding ${LG3_HOME?} vs ${LG3_HOME:?}. I've created Issue #134 to track this.

[HenrikBengtsson]

@ivan108, I think it'll be easier for you if you do:

$ git checkout feature/lg3.conf
$ git merge develop

and then fix up any conflicts before anything else is done - including what needs to be done on the develop branch. You know you're done when you can run the above to commands without any git errors/warnings.

[ivan108]

OK, I fixed LG3_HOME? on develop branch. Trying to merge with origin/feature/lg3.conf

>git status On branch feature/lg3.conf Your branch is up to date with 'origin/feature/lg3.conf'. nothing to commit, working tree clean

>git merge develop merge: develop - not something we can merge

Did you mean this? 9-07-22

I am confused....?

[HenrikBengtsson]

Yes, that is what I meant by don't update anything else before resolving that merge problem first. The problem is that you basically allowed yourself to work on an old branch without making sure to merging in develop on a regular basis.

I'm not sure how to fix this easily; I don't know that much git. I'm sure there are ways. Whatever you do/find, do not**** use --force in any of your git command. Do not do that. That can corrupt everything.

If your diff isn't to big, you might be better of creating a fresh branch from develop:

$ git checkout develop
$ git checkout -b feature/ivans-favorite-tweaks

and then copy in the files you've updated.

[HenrikBengtsson]

If you goto https://github.com/UCSF-Costello-Lab/LG3_Pipeline/compare/feature/lg3.conf...develop, it makes it clear that it cannot be merged. But they say it's still possible to do a pull request. Maybe that is a way forward.

[ivan108]

I figured out why

git merge develop

doesn't work when I am at origin/feature/lg3.conf branch! The right command should be

git merge origin/develop

Now I am getting a list of conflicts on which I can work on.

[HenrikBengtsson]

You need to make sure your branches are up-to-date. You missed to do:

$ git checkout develop $ git pull

first. Now, forget that 'git merge origin/develop' hack.

/H

On Thu, Feb 13, 2020 at 1:48 PM IVAN SMIRNOV notifications@github.com wrote:

I figured out why

git merge develop

doesn't work when I am at origin/feature/lg3.conf branch! The right command should be

git merge origin/develop

Now I am getting a list of conflicts on which I can work on.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/UCSF-Costello-Lab/LG3_Pipeline/issues/133?email_source=notifications&email_token=AAMKXUSMTJYAPAZ47XAOS43RCW5ZXA5CNFSM4KSPCVN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWXP5A#issuecomment-585988084, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMKXUQDTW5VF7OEN7HSI6DRCW5ZXANCNFSM4KSPCVNQ .

[ivan108]

I always do git pull before modifying local code..

I was trying to follow your suggestion from 2 days ago

@ivan108, I think it'll be easier for you if you do:

$ git checkout feature/lg3.conf
$ git merge develop

and then fix up any conflicts before anything else is done - including what needs to be done on the develop branch. You know you're done when you can run the above to commands without any git errors/warnings.

On success this will merge remote develop code with local feature/lg3.conf code and put merged code into local feature/lg3.conf branch, right? Is this what we want to do?

[HenrikBengtsson]

I always do git pull before modifying local code..

Hmm... if you do, the develop and origin/develop should be identical and point to the exact same commit;

$ git status
On branch develop
Your branch is up to date with 'origin/develop'.   <<===

nothing to commit, working tree clean

or equivalently:

$ git log -1
commit 2d705766035c2a33bf2ecf8dd615c667f3125737 (HEAD -> develop, origin/develop)
Author: Ivan Smirnov <Ivan.Smirnov@ucsf.edu>
Date:   Tue Feb 11 15:12:20 2020 -0800

    Fix ${LG3_HOME?} --> ${LG3_HOME:?}

...and therefore there should be no difference between git merge develop or git merge origin/develop.

On success this will merge remote develop code ...

So, 'remote' should not be needed here. See above.

with local feature/lg3.conf code and put merged code into local feature/lg3.conf branch, right? Is this what we want to do?

$ git checkout feature/lg3.conf 
Already on 'feature/lg3.conf'
Your branch is up to date with 'origin/feature/lg3.conf'.

$ git merge develop
Auto-merging scripts/trim_galore.sh
CONFLICT (content): Merge conflict in scripts/trim_galore.sh
Auto-merging scripts/runLOH_ivan.sh
CONFLICT (content): Merge conflict in scripts/runLOH_ivan.sh
Auto-merging scripts/pindel_setup.sh
CONFLICT (content): Merge conflict in scripts/pindel_setup.sh
Auto-merging scripts/pindel_filter.sh
CONFLICT (content): Merge conflict in scripts/pindel_filter.sh
Auto-merging scripts/pindel_annotate.sh
CONFLICT (content): Merge conflict in scripts/pindel_annotate.sh
Auto-merging scripts/mutect_coverage_intersection_mutations.sh
CONFLICT (content): Merge conflict in scripts/mutect_coverage_intersection_mutations.sh
Auto-merging scripts/mutdet_submit.sh
CONFLICT (content): Merge conflict in scripts/mutdet_submit.sh
Auto-merging scripts/mutation_overlaps.sh
CONFLICT (content): Merge conflict in scripts/mutation_overlaps.sh
Auto-merging scripts/libID_to_patientID.sh
CONFLICT (content): Merge conflict in scripts/libID_to_patientID.sh
Auto-merging scripts/combine_snvs.sh
CONFLICT (content): Merge conflict in scripts/combine_snvs.sh
Auto-merging scripts/chk_pindel.sh
CONFLICT (content): Merge conflict in scripts/chk_pindel.sh
Auto-merging scripts/chk_mutdet.sh
CONFLICT (content): Merge conflict in scripts/chk_mutdet.sh
Auto-merging scripts/annotate_mutations_from_bam.sh
CONFLICT (content): Merge conflict in scripts/annotate_mutations_from_bam.sh
Auto-merging scripts/Recal_step.sh
CONFLICT (content): Merge conflict in scripts/Recal_step.sh
Auto-merging scripts/Recal_pass2.sh
CONFLICT (content): Merge conflict in scripts/Recal_pass2.sh
Auto-merging scripts/Recal.sh
CONFLICT (content): Merge conflict in scripts/Recal.sh
Auto-merging scripts/Mutect2_TvsN.sh
CONFLICT (content): Merge conflict in scripts/Mutect2_TvsN.sh
Auto-merging scripts/MutDet.sh
CONFLICT (content): Merge conflict in scripts/MutDet.sh
Auto-merging scripts/MutContext.sh
CONFLICT (content): Merge conflict in scripts/MutContext.sh
Auto-merging scripts/Merge_QC.sh
CONFLICT (content): Merge conflict in scripts/Merge_QC.sh
Auto-merging scripts/Merge.sh
CONFLICT (content): Merge conflict in scripts/Merge.sh
Auto-merging scripts/Germline.sh
CONFLICT (content): Merge conflict in scripts/Germline.sh
Auto-merging scripts/Align_mem.sh
Auto-merging scripts/Align_fastq.sh
CONFLICT (content): Merge conflict in scripts/Align_fastq.sh
Auto-merging pindel_all.pbs
Auto-merging lg3.conf
CONFLICT (content): Merge conflict in lg3.conf
Auto-merging bin/lg3-test
Auto-merging bin/lg3-status
CONFLICT (content): Merge conflict in bin/lg3-status
Auto-merging bin/lg3-cleanup
CONFLICT (content): Merge conflict in bin/lg3-cleanup
Auto-merging Recal_pass2.pbs
CONFLICT (content): Merge conflict in Recal_pass2.pbs
Auto-merging PSCN.submit_all.pbs
CONFLICT (content): Merge conflict in PSCN.submit_all.pbs
Auto-merging Mutect2_TvsN.pbs
CONFLICT (content): Merge conflict in Mutect2_TvsN.pbs
Auto-merging Merge_QC.pbs
CONFLICT (content): Merge conflict in Merge_QC.pbs
Automatic merge failed; fix conflicts and then commit the result.

and

$ git diff
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
modified: Merge_QC.pbs
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ Merge_QC.pbs:11 @
#PBS -m ae

# shellcheck source=scripts/utils.sh
<<<<<<< HEAD
source "${LG3_HOME}/scripts/utils.sh"
=======
source "${LG3_HOME:?}/scripts/utils.sh"
>>>>>>> develop
source_lg3_conf

PROGRAM=${BASH_SOURCE[0]}
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
modified: Mutect2_TvsN.pbs
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ Mutect2_TvsN.pbs:13 @

# shellcheck source=scripts/utils.sh
source "${LG3_HOME:?}/scripts/utils.sh"
<<<<<<< HEAD
=======
source_lg3_conf
>>>>>>> develop
source_lg3_conf

PROGRAM=${BASH_SOURCE[0]}
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
...

So, you need to manually edit/fix all those. This is why I said "and then fix up any conflicts before anything else is done" in https://github.com/UCSF-Costello-Lab/LG3_Pipeline/issues/133#issuecomment-584796084. Previously, there were not that many conflicts.

Aborting the above ...

$ git merge --abort

You could try with -X theirs but you really have to make sure you don't overwrite what you want to keep;

$ git merge -X theirs develop
$ git diff origin/feature/lg3.conf   # manually inspect this is what you want

[ivan108]

OK, so step-by-step sequence could look like this, let me know if I am getting it right:

1. Create some new directory and go there

2. Create fresh copy of the pipeline

   $ git clone git@github.com:UCSF-Costello-Lab/LG3_Pipeline.git
   $ cd LG3_Pipeline

3. Download develop branch $ git checkout develop

4. Download feature/lg3.conf branch $ git checkout feature/lg3.conf

5. Try to merge branches and capture conflicts

   $ git merge develop | tee conflicts
   $ grep CONFLICT conflicts| head -n 4

   CONFLICT (content): Merge conflict in scripts/trim_galore.sh CONFLICT (content): Merge conflict in scripts/runLOH_ivan.sh CONFLICT (content): Merge conflict in scripts/pindel_setup.sh CONFLICT (content): Merge conflict in scripts/pindel_filter.sh

Total 30 conflicts detected

6. Manually resolve conflicts by editing current local code ......

7. When no more conflicts detected run pipeline tests

8. If all tests passed

   $ git commit ??? 
   $ git push ???

Questions:

• Will this process update only origin/feature/lg3.conf branch, or also origin/develop?
• How easy is to roll back the code changes at the origin, if something goes wrong?

Ultimately we want both legacy code and new code available to the users.... We will continue fixing errors in legacy code, but won't develop it. Only the new code from origin/feature/lg3.conf branch will be further developed.

What do you think?

Thanks as always!