feature, bulk importing of repositories #2569

Open bradder555 opened 6 years ago

Recently i hacked about in the back-end and database to migrate all of our stash repositories and jam these into gitea.

It's not an operation that i'll need to repeat, as far as i can tell in the documentation there's no procedure for doing this sort of thing.

I was thinking, as a feature, it might be handy to have an "upload zip" for migration to facilitate easy migration from other git servers.

I'm sure there're higher priorities, but figured it would make it easier for people to migrate and adopt gitea.

Thanks

You can do migrations via API. Write some scripts to do that.

Such migrations should be done in scripts or external tools but should not be part of main gitea repo

Any pointers to valid Github-to-Gitea scripts? I come up empty.

@tacotexmex currently @JonasFranzDEV has a really good one: https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator

Thank you very much @tecknowlogick. quietly thinking how I could’ve discovered this selfhosted gem without your help, dreaming of a federation of Gitea instances, Mastodon-style

@tacotexmex you are very welcome. Currently there is a working group that is dedicated to solving the issue of federated services. I actually just spent the morning catching up on discussions from the group. If you want to follow along you can see: https://github.com/git-federation/gitpub/issues/5 Unrelated to federation we have a discord channel where the above tool was posted to.

Such migrations should be done in scripts or external tools but should not be part of main gitea repo

@lafriks why? i think migration should be as simple as possible.

imagine a workflow like this:

login to gitea with github
click "import from github"
select repos you want to import (all selected by default)
wait some seconds (or minutes) ... migration done

if load on the gitea server could be a problem, a queue could be implemented.

@davidak while I agree that it should be as simple as possible but if we implement imports from github/bitbucket/gitlab etc it will be hard keep up with their api changes. There is example that it can be easily done outside of Gitea main code base - https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator

Of course there are few things that we need to implement in Gitea API to have possibility to import more data but that's other thing

Anyone have a mirror or copy of https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator ? The site is throwing a 500 error when trying to either clone the repo locally or even download the source code.

@JonasFranzDev your server has problems ;)

Edit, use at your own risk and don't randomly download zips from the internet.

Cheers guys, thanks 😄

https://git.jonasfranz.software/JonasFranzDEV/gitea-github-migrator is back up and states:

This repository got moved to the offical Gitea server and will be developed there. https://gitea.com/gitea/migrator/

I just tested it, it worked flawlessly (Debian 9/10)

apt update
apt install make golang go-dep git make
go get gitea.com/gitea/migrator
cd go/src/gitea.com/gitea/migrator
dep ensure -v
make build

sudo -u gitea ./gitea-github-migrator migrate --gh-repo my/project --gh-token $GITHUB_PERSONAL_TOKEN --url https://example.com/gitea --token $GITEA_API_TOKEN

Edit: limitation: doesn't import pull requests/release notes. Didn't create the project as "mirror" (no automatic pulls in the future).

Repository, issues, tags, wiki were imported correctly. Operation took 11 minutes on a github project with 1300 issues.

For others coming to this issue, the migration functionality is now built into gitea itself.

@techknowlogick you mean using the + New migration action?

This doesn't import issues/wiki for me, only the git repository.

@nodiscc @techknowlogick is right. You can migrate almost all things from github.

When you input a github URL on migration UI, and put a username/password or token/, then you will find some migrations options displayed, you can check issues/wiki/pull requests/milestones/labels .

Of course you have to have v1.9 or above.

@lunny my bad, I'm on 1.8.0, still have to upgrade. I will definitely try this (my gripe was it didn't set up the repository as mirror and there was no way to change it after the fact).

By the way, thank you and all contributors for the hard work on Gitea. I'm in the process of moving all my projects to a self-hosted Gitea instance (with automatic Github/Gitlab mirrors through hooks), it has been great so far. You can find my ansible role to install gitea here (inspired from [1], [2]).

I think this issue can be closed as bulk import/migration is perfectly doable using https://gitea.com/gitea/migrator/ (which is linked at https://docs.gitea.io/en-us/third-party-tools/) - or one by one from the web interface.

@lunny ping

@nodiscc Currently we can import almost all things of one repository from github, but maybe this issue is importing many repositories on one operation.

I would say this is a feature for tea link: https://gitea.com/gitea/tea/issues/22

PS: when tea reaches v1.0 we can add it to gitea docs as tool to automate/do stuff via CLI ...

Currently the suggested "https://gitea.com/gitea/migrator/" shows:

This tool is no longer maintained and got replaced by the migration functionality in Gitea. 
Please use the "New migration" menu to migrate your GitHub repositories including issues, pullrequests etc.

But "new migration" inside gitea does only migrate a single repository, so this does not help much if one has to migrate hundreds of repos.

@csarn and others. this is true I wanted to bulk migrate repo's but migratory is 1 repo at a time???

what to do?

@ameeno best way at the moment: use API via go-sdk or curl ... script

The other option is to use adopt repository if the repositories are in the right place with the correct names.

anyone have a script ready for batch migration? I found a Chinese one on awesome-gitea but I am having problems with it it only working for orgs.

I need to bulk import 80+ repositories from GitLab to Gitea - any pointers for how this could be done? Thanks a lot!

I need to bulk import 80+ repositories from GitLab to Gitea - any pointers for how this could be done? Thanks a lot!

You can do it via some scripts with migration API.

sure, I know this is possible. Has anybody written any such script? I couldn't find one... :/

Hi any update on it?

I have like 300 repos I want to migrate and doing this one by one isnt an issue. the only script I found was at dev.to which doesnt support migration of public vs not or multiple repos. does someone have a github to gitea script for that? :/

Try something like this (assumes you have Github cli installed - on Mac you can brew install gh)

#! /usr/bin/env bash

GITEA_DOMAIN=http://myserver
GITHUB_TOKEN=
GITEA_TOKEN=
GITHUB_USERNAME=
GITEA_USERNAME=

REPOS=$(gh repo list -L 1000 | awk -F '\t' '{print $1}')

for REPO_NAME in $REPOS; do
    echo $REPO_NAME
    curl -X POST "$GITEA_DOMAIN/api/v1/repos/migrate" -u $GITEA_USERNAME:$GITEA_TOKEN -H  "accept: application/json" -H  "Content-Type: application/json" -d "{  \
        \"auth_username\": \"$GITHUB_USERNAME\", \
        \"auth_password\": \"$GITHUB_TOKEN\", \
        \"clone_addr\": \"https://github.com/$GITHUB_USERNAME/$REPO_NAME\", \
        \"description\": \"$(gh repo view ${REPO_NAME} --json description --jq '.description')\", \
        \"mirror\": false, \
        \"private\": true, \
        \"repo_name\": \"$REPO_NAME\", \
        \"repo_owner\": \"$GITEA_USERNAME\", \
        \"service\": \"git\", \
        \"uid\": 0, \
        \"wiki\": true}"
    sleep 2
done

What it doesn't do which I wish it did is set up push forwarding of each repo back to Github so I can use Gitea as my primary and maintain a backup on Github.

I stumbled across this thread too and didn't find a suitable solution for my usecase. So, I created this gist: https://gist.github.com/CodeShakingSheep/5dc2cf6ac3b6d265218a7214f8f1210b

No access to Github CLI needed and it works for organization and user repos. If you need user repos, just leave GITHUB_ORGANISATION empty.

I also created a script for removing the repos from Gitea in case something went wrong. https://gist.github.com/CodeShakingSheep/65a7f46ab6067a72835e13b8459c0f7d

+1 on this one, in my case it's gitLab not github that I need to migrate from and needing to study a new scripting language or curl just to get it "the right way" it's a bigger nuissance than migrating 20 repos manually, so I'll be doing that for the next half an hour.

I stumbled across this thread too and didn't find a suitable solution for my usecase. So, I created this gist: https://gist.github.com/CodeShakingSheep/5dc2cf6ac3b6d265218a7214f8f1210b

No access to Github CLI needed and it works for organization and user repos. If you need user repos, just leave GITHUB_ORGANISATION empty.

I also created a script for removing the repos from Gitea in case something went wrong. https://gist.github.com/CodeShakingSheep/65a7f46ab6067a72835e13b8459c0f7d

Currently, only the first 100 repos are imported. This script will import all repos. https://gist.github.com/boydaihungst/809861ff696deceefbb89a8a075f6b02

I have made some slight tweaks to @boydaihungst 's gist and now it includes setting the org fullname,avatar,link and description (if set). This will also set them to be a mirror by default, behaviour can be changed at the bottom of the script.

how-to ./import-org go-gitea

#!/bin/bash
set -x

GITHUB_USERNAME=aneurinprice
GITHUB_TOKEN=<redacted>
GITHUB_ORGANISATION=$1
GITHUB_API_CALL=

GITEA_USERNAME=archive
GITEA_TOKEN=<redacted>
GITEA_DOMAIN=http://<redacted>.local:3000
GITEA_REPO_OWNER=$GITHUB_ORGANISATION

if [ -z "$GITHUB_ORGANISATION" ]
then
      echo "\$GITHUB_ORGANISATION is empty"
      GITHUB_API_CALL="https://api.github.com/user/repos?per_page=100"
else
      echo "\$GITHUB_ORGANISATION is NOT empty"
      GITHUB_API_CALL="https://api.github.com/orgs/$GITHUB_ORGANISATION/repos?per_page=200&type=all" 
fi 

GET_REPOS=$(curl -H 'Accept: application/vnd.github.v3+json' -H "Authorization: token $GITHUB_TOKEN" $GITHUB_API_CALL | jq -r '.[].html_url')

ORG_DATA=$(curl -H 'Accept: application/vnd.github.v3+json' -H "X-GitHub-Api-Version: 2022-11-28" -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/orgs/$GITHUB_ORGANISATION) 

curl -X POST "$GITEA_DOMAIN/api/v1/orgs" -H  "accept: application/json" -H "Authorization: token $GITEA_TOKEN" -H  "Content-Type: application/json" -d "{  \
  \"username\": \"$(jq -r '.login' <<< $ORG_DATA)\",
  \"description\": \"$(jq -r '.description' <<< $ORG_DATA)\",
  \"full_name\": \"$(jq -r '.name' <<< $ORG_DATA)\",
  \"website\": \"$(jq -r '.html_url' <<< $ORG_DATA)\",
  \"visibility\": \"public\"}"

curl "$(jq -r '.avatar_url' <<< $ORG_DATA)" --output logo
ORG_AVATAR_B64=$(base64 -i logo)

curl -X POST "$GITEA_DOMAIN/api/v1/orgs/${GITHUB_ORGANISATION}/avatar" -H  "accept: application/json" -H "Authorization: token $GITEA_TOKEN" -H  "Content-Type: application/json" -d "{  \
\"image\": \"$ORG_AVATAR_B64\"}"    

rm logo

for URL in $GET_REPOS; do
    REPO_NAME=$(echo $URL | sed 's#.*/##')
    echo "Found $REPO_NAME, importing..."

    curl -X POST "$GITEA_DOMAIN/api/v1/repos/migrate" -H  "accept: application/json" -H "Authorization: token $GITEA_TOKEN" -H  "Content-Type: application/json" -d "{  \
    \"auth_username\": \"$GITHUB_USERNAME\", \
    \"auth_password\": \"$GITHUB_TOKEN\", \
    \"clone_addr\": \"$URL\", \
    \"mirror\": true, \
    \"private\": false, \
    \"repo_name\": \"$REPO_NAME\", \
    \"repo_owner\": \"$GITEA_REPO_OWNER\", \
    \"service\": \"git\", \
    \"uid\": 0, \
    \"wiki\": true}"

done

If I get the time, I might create a dedicated tool for stuff like this

If anyone cares, I have a basic prototype up and running. Import orgs or users along with their github avatars. Work in progress, but it works.

https://gitlab.nyeprice.space/aneurinprice/gitea-bulk-exporter

Edit: Add screenshot

go-gitea / gitea

feature, bulk importing of repositories #2569