migrate away from trac and svn

[totaam]

They have served us well for over 10 years, but without any meaningful progress, especially with trac still not supporting python3 properly.. It's time to move on.

Options (in no particular order):

• migrate to gitlab using tracboat (itself hosted on github!?) - either hosted at gitlab.com or self-hosted on the current hardware, or even elsewhere
• migrate to github using trac-to-github

There are pros and cons for each option:

• amount of work for migrating
• amount of maintenance work
• cost
• accessibility
• making it easier for others to contribute
• maintaining existing documentation links and resources (could keep trac in read-only mode for a while)
• preserving code authors in git history
• ci / cd
• use oauth if self-hosted?

[totaam]

As an aside, if going for the hosted solution, it might be worth hosting the mailing list too. (there are many mailman hosting services out there, all relatively cheap)

See also: move to new signing keys (#2968)

It would be nice if some of the release script could preserve the incremental revision number feature of svn:

• Simulating a global revision number with git

[totaam]

2020-12-08 17:08:31: stdedos commented

I would much more prefer Github than Gitlab, but that may come down purely on personal preference.

There are tidbits here and there that make everything feel more palpable (e.g. the URLs are very clearly read and manipulated to move from here to there in Github), and there are no "weird behaviors (e.g. the annoying @-reference automatically assigning the issue to the referencee as to-do that exists in Gitlab)

For git-specific stuff, I guess you have already seen https://git-scm.com/book/en/v2/Git-and-Other-Systems-Migrating-to-Git (and might be a good idea to update it if/after you have improved something).

[totaam]

2021-01-01 23:14:13: brief commented

If considered that this step will be necessary again in the future, data should be available for migration.

Self hosted solutions (e.g. gitlab) guarantee to be able to access this data.

[totaam]

• generate new ssh and gpg keys for antoine@xpra.org and xpra@xpra.org (see #2968)
• enable 2factor yubikey with github
• create a user mapping authors.txt from known svn users
• convert using svn2git:

  mkdir Xpra
  cd Xpra
  # use svn tags as git branches:
  svn2git https://xpra.org/svn/Xpra \
  --trunk trunk \
  --branches tags --notags \
  --metadata \
  --exclude web \
  --exclude trunk/fakexinerama \
  --exclude trunk/scripts/fakeXinerama-snapshot.sh \
  --authors ~/authors.txt
  # then remove EOL branches (too many):
  for x in v0.0.7.x v0.1.x v0.10.x v0.11.x v0.12.x v0.13.x v0.14.x v0.15.x v0.16.x v0.17.x v0.2.x v0.3.x v0.4.x v0.5.x v0.6.x v0.7.x v0.8.x v0.9.x v2.0.x v2.1.x v2.2.x v2.3.x v2.4.x v2.5.x; do git branch -D $x; done
  # create tags for specific versions (the last one from each supported branch only):
  for V in v1.0.14 v3.0.13 v4.0.6; do
  echo $V
  MV=`echo $V | cut -c2-4`
  git checkout "v"$MV".x"
  NOV=`echo $V | cut -c2-10`
  #get the xpra source snapshot:
  XPRAV="xpra-$NOV"
  wget "https://xpra.org/src/$XPRAV.tar.xz"
  tar -Jxvf $XPRAV.tar.xz $XPRAV/svn-version
  #get the svnversion from it:
  SVNVERSION=`cat $XPRAV/svn-version | xargs echo -n`
  rm -fr $XPRAV*
  echo "r$SVNVERSION"
  #find the git commit for this svn version number:
  COMMIT=`git log --all --grep="$SVNVERSION" | grep "^commit" | awk '{print $2}'`
  echo "git-commit: $COMMIT"
  #preserve the date:
  GIT_COMMITTER_DATE=`git show --format=%aD $COMMIT | head -n 1 | xargs echo -n`
  #tag it:
  git tag $V $COMMIT
  unset GIT_COMMITTER_DATE
  done
  # configure email and keys:
  git config user.email "antoine@xpra.org"
  git config user.signingkey 0xDCDCA2B6F88896E4
  # "upload" to github:
  git remote add origin git@github.com:Xpra-org/xpra.git
  git push --set-upstream origin master
  git push --all origin
  git push --tags origin

• move the debian, rpmbuild, osx and win32 directories to one sub-directory? (win32 will need more work as it contains scripts used by setup.py)
• move the code one level up: remove redundant src dir
• split the HTML5 client - from the html5 folder directly because I couldn't make svn2git honour --exclude (see -exclude parameter doesn't seem to skip folders as directed):

  $ svn2git https://xpra.org/svn/Xpra \
  --trunk trunk/src/html5 \
  --nobranches \
  --notags \
  --metadata \
  --authors ~/authors.txt \
  --revision 5070
  # configure email and keys:
  git config user.email "antoine@xpra.org"
  git config user.signingkey 0xDCDCA2B6F88896E4
  # "upload" to github:
  git remote add origin git@github.com:Xpra-org/xpra-html5.git
  git push --set-upstream origin master
  git push --all origin
  git push --tags origin

• convert trac tickets to github issues:

  # generate map from revision to commit:
  git log  | egrep "git-svn-id:|^commit" | awk '/    git-svn-id:/{print$0p}{p=$0}' | sed 's+.*trunk@++g' | awk '{print $1" "$3}' > rev-map-file
  # edit config: add token, etc - then:
  bundle exec trac-hub -r ./rev-map-file

Still in progress:

• git2svn generates URLs like https://xpra.org/svn/Xpra/trunk@28364, apache will need to rewrite those to valid URLs
• configure github to never generate pull requests from stable branches to master? can it be done or do we have to just ignore it?
• make it possible to generate the releases directly on github?
• add README, LICENSE, etc
• make the repository public
• migrate the wiki pages (requires a public repository)
• edit trac and www to point to github
• lock svn and trac
• make svn and trac backups public
• send announcements
• transfer relevant issues to the xpra-html5 repository etc

[totaam]

2021-01-20 16:34:19: antoine uploaded file TracWiki2MediaWiki.pl (36.6 KiB)

perl script found here: https://www.mediawiki.org/w/index.php?oldid=2103135#Code

[totaam]

2021-01-21 15:00:17: antoine uploaded file extract-trac-attachments.sh (1.4 KiB)

needed to keep links to the attachments working

[totaam]

2021-01-22 13:32:45: antoine uploaded file trac-hub.patch (3.0 KiB)

remove email addresses from tickets, remove dependency on unused mysql backend, increase delay to prevent hitting the github API rate limits, don't post empty comments, fix wiki links

[stdedos]

The repositories and tickets are now available here: ​https://github.com/Xpra-org

🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

Welcome to the other side 😀 You could've added tags labels for these ones too!

component: external | priority: major

[totaam]

Adding a redirect in each ticket - this will take time to run:

for i in `seq 4 3008`; do
  python2 -c "from trac.env import Environment;from trac.ticket import Ticket;\
  env = Environment('/var/lib/trac/Xpra/', create=0);\
  tkt = Ticket(env, $i);tkt.save_changes('migration script', comment='this ticket has been moved to: [https://github.com/Xpra-org/xpra/issues/%s]' % \"$i\")";
done

github seems to do the right thing for tickets that have been moved to https://github.com/Xpra-org/xpra-html5

[stdedos]

I don't know why you splitted the html repo; but yeah - if it works, you should do that.

At least https://xpra.org/trac/ticket/2967 -> https://github.com/Xpra-org/xpra/issues/2967 works

[totaam]

LGTM.

Public announcement on the mailing list: migration to github

[stdedos]

After so many years (and more than 2 years of experience in Gitlab) ... I sort-of regret my decision :sweat_smile:

Github has "less features" than Gitlab - which work better in Github ... but Github CI is so far behind with comparison to Gitlab's.

After learning to ignore a lot of Gitlab's stuff that you don't use, then the CI is many times over worth it

I know it's "a little bit too late", but I did want to just leave this here for posterity