[Discussion] Move back from the submodule approach to a monorepo

[EinarElen]

I'm making this issue so we can have the discussions regarding whether or not to move away from submodules in a persistent place (as opposed to slack 😄).

Summarizing the discussion on Slack briefly

From @tomeichlersmith

• Proposed timeline
  • T: the time when the submodule->monorepo conversion is done
  • T - 1 week: announce last call for pre-conversion contributions.
  • T - 3 days: write docs page about conversion (e.g. how to update your branch) and post reminder announcement
  • T - 1 day: last merges and final releases of submodules pre-conversion, make pre-conversion ldmx-sw release
  • T - 2 hours: convert submodules into subdirectories and post PR
  • T: merge conversion PR
  • T + 1 hour: new ldmx-sw release marking change after double checking functionality locally
  • T + 1 day: transfer issues from submodules to ldmx-sw repo, tag with relevant subdirectory
  • T + 1 week: add comment to submodule README, archive submodules
  • T + 2 weeks: test run past ldmx-sw version to make sure archived submodules are still accessible as promised

Questions:

• What should T be?
• How to deal with stale branches (Tom favours removing them)
• Should this be a major release?
• What parts can/should remain submodules?
  • G4DarkBreM (separate paper)
  • MagFieldMap (Large and stable, needs to be checked if it can be in a monorepo)
  • Framework (Up for discussion, but has utility in being separate + quite stable)
  • Projects owned by others (ACTS, ruckus, doxygen-awesome-css, HLS_arbitrary_Precision_Types)

From @EinarElen:

• T should be before we start getting summer students

From @tvami

• Keeping G4DarkBreM separate makes sense. Does MagFieldMap and Framework need to be?

[tvami]

My main point is that maybe the measure should be "how ldmx specific the package is" instead of stability. And given that both Framework and MagFieldMap are very ldmx specific, I'd argue that they should also be merged into the mono-repo.

[tomeichlersmith]

Okay, after doing some extremely simple estimates, I am not worried about pulling MagFieldMap into ldmx-sw anymore.

tom@nixos:~/code/ldmx/monorepo-ification/ldmx-sw$ du -sh \
  --exclude 'Tracking/acts*' \
  --exclude '*G4DarkBreM*' \
  --exclude '*doxygen-awesome-css*' \
  --exclude 'Trigger/ruckus*' \
  --exclude 'Trigger/HLS_arbitrary_Precision_Types*'
425M    .

GitHub doesn't have a hard limit on the total size of a repo, but they generally suggest a soft limit of ~1GB. This means our code base could double in size and we'd still would have space for 10 more copies of the magnetic field map (15MB). If we have that many copies, we should probably look at a better solution like a calibrations database entry for the mag-field anyways.

[tvami]

What should T be?

Why dont we go ahead with T = May 1st? There is a sw-dev meeting next week and a SWAN meeting 2 weeks from now, and this would give yet another week for people to act.

How to deal with stale branches (Tom favours removing them)

+1 from me

Should this be a major release?

I'd say so. Even tho it doesnt change the physics it's a big change structurally.

[tvami]

What should T be?

Why dont we go ahead with T = May 1st? There is a sw-dev meeting next week and a SWAN meeting 2 weeks from now, and this would give yet another week for people to act.

@bryngemark @vdutta what do you think about this? (I see Tom and Einar already "thumbs-up"-ed it)

[tomeichlersmith]

Notes from SW-Dev Meeting 4/17

• T is tentatively May 1st

  • ask for feedback over the next week, willing to push T back if requested by a contributor

• Notification Avenues

  • Announce at General Weekly (@bryngemark or @vdutta during report on past meetings)
  • Announce at SWAN on Monday (@therwig )
  • Email over mme (@tomeichlersmith )
  • Slack at software-dev channel (@tomeichlersmith )

• Documentation on how to update branch

  • Partition cases (changes in ldmx-sw only, changes in submodule only, changes in both)
  • Post on doc website and link in Email/Slack messages

[tomeichlersmith]

trunk now contains the results of running the following script. I am using a simple cleanup change #1293 to double check that this change doesn't alter the code results compared to v3.4.0. I ran this script with git 2.34.1.

#!/bin/sh

set -o errexit
set -o nounset
set -o xtrace

# submodules without submodules of their own are easier
for sm in Conditions Ecal Framework Hcal MagFieldMap TrigScint cmake
do
  sm_main="$(git -C ${sm} remote show origin | sed -n '/HEAD branch/s/.*: //p')"
  git remote add ${sm}_origin git@github.com:LDMX-Software/${sm}.git
  git fetch ${sm}_origin ${sm_main}
  git merge -s ours --no-commit --allow-unrelated-histories ${sm}_origin/${sm_main}
  git rm --cached ${sm}
  sed -i "/${sm}/d" .gitmodules
  rm -rf ${sm}/.git
  git add ${sm} .gitmodules
  git commit -m "transition ${sm} from submodule to subdirectory"
  git remote rm ${sm}_origin
done

# this method of copying the commit reference of the submodules of the submodules
# disregards the commit history of that submodule :(
for sm in SimCore Tracking Trigger
do
  git -C ${sm} submodule status > ${sm}.submods.list
  git -C ${sm} submodule --quiet foreach git remote get-url origin > ${sm}.submods.remote
  sm_main="$(git -C ${sm} remote show origin | sed -n '/HEAD branch/s/.*: //p')"
  git remote add ${sm}_origin git@github.com:LDMX-Software/${sm}.git
  git fetch ${sm}_origin ${sm_main}
  git merge -s ours --no-commit --allow-unrelated-histories ${sm}_origin/${sm_main}
  git rm --cached ${sm}
  sed -i "/${sm}/d" .gitmodules
  rm -rf ${sm}/.git .git/modules/${sm}
  paste ${sm}.submods.list ${sm}.submods.remote | while read commit relpath describe url
    do
      rm -rf "${sm}/${relpath}"
      git submodule add "${url}" "${sm}/${relpath}"
      git -C "${sm}/${relpath}" checkout "${commit}"
      git add "${sm}/${relpath}"
    done
  git add ${sm} .gitmodules
  git commit -m "transition ${sm} from submodule to subdirectory"
  git remote rm ${sm}_origin
  rm ${sm}.submods.list ${sm}.submods.remote
done