search

christianspecht / scm-backup

Makes offline backups of your cloud hosted source code repositories
https://scm-backup.org/
GNU General Public License v3.0
61 stars 20 forks source link

SCM Backup: Bitbucket Repos of more than a 100 #33

Closed helpdesktdl closed 4 years ago

helpdesktdl commented 5 years ago

Hi guys,

I'm running into a issue when backing up my bitbucket repos. As I have over 100 repos the backup fails.

IS there a way I can adjust script so that a "sleep" instruction is put in the script and then keeps looping through the remaining repos in blocks of 99.

Thanks.

J-P

christianspecht commented 5 years ago

From your description alone, it's hard to tell where exactly the backup fails.

SCM Backup first calls Bitbucket's API to get a list of your repos, and then it loops that list and clones/pulls each repo with Git/Hg.

There's an integration test which checks that the API call works for a user with lots of repos, but so far there's no test for cloning/pulling.

Can you show me a logfile? This would help to find the exact place in the code where it fails.

helpdesktdl commented 5 years ago

Hi Christian,

Find attached log file.

[scm-backup-2019-03-26-13-03-03.log](https://github.com/christianspecht/scm-backup/files/3040881/scm-backup-2019-03-26-13-03-03.log)

helpdesktdl commented 5 years ago

Any update @christianspecht

christianspecht commented 5 years ago

Sorry, not enough spare time at the moment.

According to the log, 249 repos are successfully backed up...and then it blows up while trying to backup the 250th repo (bcd-web-air). Given it's exactly the 250th repo, the reason is probably the number of repos and nothing with this specific repo.

But I'd like you to check this, to make sure it's really not about this repo: Could you please ignore bcd-web-air in the config and execute SCM Backup again?

    ignoreRepos:
        - bcd-web-air

If it's really because of the number of repos, the same error should occur while backing up the next repo (alphebetically) after this one.

If that's really the case, I'll add some "wait X seconds after backing up Y repos" option to SCM Backup like you suggested. (if not, then it's some problem with this specific repo and we need to try something else)

But honestly, I have no idea for the reason behind this if it turns out to fail after the 249th repo. SCM Backup just gets a list of your repos from the Bitbucket API, loops over that list and makes Git/Mercurial command line calls for each repo. So far I have no clue why this would work 249 times and fail the 250th time.

christianspecht commented 5 years ago

By the way, is mttnow a user or a team?

If it's a user, this probably applies to you: https://scm-backup.org/2019/04/16/release-1-2-1-bitbucket-api-breaking-changes/

helpdesktdl commented 5 years ago

Hi, it ignores that repo but fails again later on another repo. MTTNOW is the name of our team domain

christianspecht commented 5 years ago

Okay, I'm still not sure whether this specific repo might be the culprit (and the other one, which fails later, as well) or whether it's a problem in SCM Backup related to the number of repos.

Here are the Git commands SCM Backup uses internally to backup a repo. Can you execute them on your machine?

git init --bare "c:\scm-backup\mttnow\bcd-web-air"

git -C "c:\scm-backup\mttnow\bcd-web-air" fetch --force --prune https://mttnow:YOURTOKEN@bitbucket.org:443/mttnow/bcd-web-air.git  refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*

Does that work or do you get the same error?

christianspecht commented 4 years ago

Closed for inactivity