Mirror github repositories internally at gitlab.nsls2.bnl.gov

[cowanml]

On Tue, 31 May 2016, Eric Dill wrote:

?

Is this to address John Hill's desire to have a backup of stuff on github?

Unless there's some specific benefit to mirring it on gitlab, it seems like it would just cause confusion?

"There should be one-- and preferably only one --obvious way to do it" https://www.python.org/dev/peps/pep-0020/

If we're trying to address the backup issue we just need to get it to a disk that's backed up to tape (or maybe bacula has plugins or something that could do it directly?).

"A simple script to backup an organization's GitHub repositories, wikis and issues. " https://gist.github.com/rodw/3073987

Something like that once a day should be fine, I don't think we need to worry about hooks to mirror everything as hit happens; that would be cool if it's easy to do, but I think we'd be able to easily recover last day changes from individual repos.

Note: doesn't look like it captures PR discussions? And several comments reference bugs/improvements that may or may not have been fixed.

-matt

[ericdill]

Is this to address John Hill's desire to have a backup of stuff on github?

Yes.

Unless there's some specific benefit to mirring it on gitlab, it seems like it would just cause confusion?

We have an internal gitlab server. What sort of confusion are you expecting? github is where the development happens. gitlab would be a local mirror. I am not sure that anyone would be confused by what a local mirror is.

"There should be one-- and preferably only one --obvious way to do it" https://www.python.org/dev/peps/pep-0020/

That feels a little pedantic

If we're trying to address the backup issue

If we're just trying to address the backup issue, then everyone who has the git repo on their laptop serves as a backup. Last count that is at least 5 people, not to mention most of our home directories on the nfs also have these git repos. We have lots of places where these repos are "backed up"

Something like that once a day should be fine,

Agreed.

I don't think we need to worry about hooks to mirror everything as hit happens;

Agreed.

but I think we'd be able to easily recover last day changes from individual repos.

I'm not sure what this means

"A simple script to backup an organization's GitHub repositories, wikis and issues. " https://gist.github.com/rodw/3073987

Would you have the bandwidth to get this running so we can backup the nsls-ii, scikit-beam and nikea organizations that we have on github.com?

[cowanml]

Some people will have trouble understanding the distinction between github/gitlab, especially when they both seem to do the same thing. We can easily avoid that, making it clear where development is supposed to happen by not having 2 places it can happen.

pedantic? maybe, but there's a reason that was chosen as a core ideal of python.

yes, the repos are already "backed-up" with all our individual clones, that's why I wouldn't worry about using hooks to capture everything. It will be nice to backup all the PR and issue discussions and any wiki stuff, etc. And I like that that script works at the org level, so it should automatically cover new repos.

just ran a slightly tweaked version of that script wrapped in another script to

get all our orgs (am I missing any?) and...

[cowan@abbix10 ~]$ du -shx github-backups/* | sort -hr 351M github-backups/scikit-beam 140M github-backups/nikea 70M github-backups/nsls-ii 42M github-backups/nsls-ii-chx 34M github-backups/nsls-ii-lix 5.3M github-backups/nsls-ii-srx 684K github-backups/nsls-ii-hxn 196K github-backups/nsls-ii-xpd 40K github-backups/nsls-ii-fmx

40K github-backups/nsls-ii-amx

still need to put this somewhere it will get to tape and get it in a cron job, but almost done.

-matt

[ericdill]

That's awesome, thanks!

nsls-ii-ixs is another one

[stuartcampbell]

Hi @cowanml This is done and running isn't it ?

[cowanml]

On Fri, 25 Aug 2017, Stuart Campbell wrote:

Hi @cowanml This is done and running isn't it ?

I don't know? Wouldn't think so, but I can't currently see any projects on gitlab I'm not a member of?

I know we have a nightly backup of github orgs/repos (that we've been told about), but I don't think it's been tested very much. Should backup the repo and make a dump of associated issues, wiki, etc. Puts everything in rums:/var/GITHUB/github-backups/

I'm not sure I understand the purpose of mirroring to gitlab? Seems like that would just cause confusion?

-matt

rumsfeld:/var/GITHUB$ cat github-backup.git/backup-github-allorgs.sh

!/bin/bash -e

simple wrapper script to loop through a list of orgs, running backup-github.sh on each.

GHBU_BACKUP_BASE=${GHBU_BACKUP_BASE-"dirname $0"} . $GHBU_BACKUP_BASE/orgs.txt

for bl in $BEAMLINES; do ORGLIST="${ORGLIST} nsls-ii-${bl}" done

for org in $ORGLIST; do ${GHBU_BACKUP_BASE}/backup-github.sh $org done

rumsfeld:/var/GITHUB$ cat github-backup.git/orgs.txt ORGLIST="nsls-ii nsls-ii-mx nsls-ii-sysadmin nikea scikit-beam nsls2ap channelfinder epicsdeb synchbot"

these are each prepended with "nsls-ii-" and added to the ORGLIST

BEAMLINES="amx chx cms esm fmx hxn isr ixs lix smi srx tes xfp xpd"