Mirror all JDownloader repositories

[darkdragon-001]

Please also mirror dependency repositories:

• AppWork Utilities: svn://svn.appwork.org/utils
• JDBrowser: svn://svn.jdownloader.org/jdownloader/browser
• MyJDownloader: svn://svn.jdownloader.org/jdownloader/MyJDownloaderClient

[T3rm1]

Why do they even still use SVN?

[milahu]

https://github.com/svn2github/jdownloader https://github.com/svn2github/appwork-utils https://github.com/svn2github/jdownloader-browser https://github.com/svn2github/jdownloader-MyJDownloaderClient

the bad news:

This is a clone of an SVN repository at svn://svn.jdownloader.org/jdownloader/trunk. It had been cloned by http://svn2github.com/ , but the service was since closed. Please read a closing note on my blog post: http://piotr.gabryjeluk.pl/blog:closing-svn2github . If you want to continue synchronizing this repo, look at https://github.com/gabrys/svn2github

20 January 2019: Disable automatically updating the repos

[milahu]

• svn://svn.jdownloader.org/jdownloader
  • https://github.com/mycodedoesnotcompile2/jdownloader_mirror
  • https://github.com/mirror/jdownloader - 2023, outdated
  • https://sourceforge.net/p/jdownloader-svn-mirror-milahu/code/
  • pull: rsync --recursive --info=progress2 svn.code.sf.net::p/jdownloader-svn-mirror-milahu/code/ jdownloader/
    • fast: 90 rev/sec. rsync is 100x faster than svn → fast initial sync, then slow update from upstream svn repo
  • push: svnsync sync svn+ssh://milahu@svn.code.sf.net/p/jdownloader-svn-mirror-milahu/code/ file://$PWD/jdownloader/
    • pushing requires hooks/pre-revprop-change - see SVN Import
    • slow: 0.4 rev/sec
• svn://svn.appwork.org/utils
  • https://github.com/milahu/appwork-utils
• svn://svn.jdownloader.org/jdownloader/browser
  • part of the main jdownloader repo
• svn://svn.jdownloader.org/jdownloader/MyJDownloaderClient
  • part of the main jdownloader repo

sync tool https://github.com/milahu/svn2github - requires github ssh public key

#!/bin/bash
githubUser=milahu
set -e # fail on error
for repoName in appwork-utils jdownloader-browser jdownloader-client
do
  svn2github update $githubUser/$repoName
done

[milahu]

Please also mirror dependency repositories

this also makes sense, because svn://svn.jdownloader.org/jdownloader is a monorepo with 45K revisions and if you do a partial mirror, the revisions are not continuous (aka "sparse")

the jdownloader documentation says we should add the folders separately but most commits are in trunk/ = jdownloader sources so there is only little overhead in using the monorepo

svn2github

i tried to mirror the full repo to github, but github says

[GitHub] Git LFS disabled for milahu

Git LFS has been disabled on your personal account milahu because you’ve exceeded your data plan by at least 150%. Please purchase additional data packs to cover your bandwidth and storage usage:

https://github.com/account/billing/data/upgrade

Current usage as of 12 May 2022 02:50AM UTC:

Bandwidth: 0.0 GB / 1 GB (0%)

Storage: 1.6 GB / 1 GB (160%)

i guess 1.6GB means the bare git repo size (du -sh .git/)

anyway, i gave up on mirroring this to github

but i still have a mirror on sourceforge: https://sourceforge.net/p/jdownloader-svn-mirror-milahu/code/HEAD/tree/ this can be cloned with rsync in about 8 minutes

rsync -r rsync://svn.code.sf.net/p/jdownloader-svn-mirror-milahu/code/ ./jdownloader --verbose

problem is, you cannot verify the integrity of my mirror so to make sure, you still need to fetch the original svn repo

svnsync of the original repo takes 6 hours (plot) svnrdump should be much faster

svnrdump dump --quiet --incremental --revision 0000:0999 --file svn-dump.0 svn://svn.jdownloader.org/jdownloader
svnrdump dump --quiet --incremental --revision 1000:1999 --file svn-dump.1 svn://svn.jdownloader.org/jdownloader
svnrdump dump --quiet --incremental --revision 2000:2999 --file svn-dump.2 svn://svn.jdownloader.org/jdownloader
# ...

to convert svn to git, i use svn-all-fast-export which is much faster than the old svn2github implementation via git svn fetch

i wrapped everything into a python script svn2github which is not perfect, but it's a start

happy mirroring ; )