Single part upload using GitHub storage

begonaguereca commented 1 month ago

Summary:

This PR introduces the ability for the GEI CLI to upload archives to GitHub-owned storage using single part uploads. To enable this, a new CLI option, --use-github-storage, has been added. This option allows users to explicitly specify that their archives should be uploaded to GitHub's managed storage instead of their own.

Key Features: Single Upload Support: Leveragingsingle upload for file uploads. This ensures the robustness of the uploads, minimizing the chances of failure during transmission.

New Option: --use-github-storage:

This option is required to upload archives to GitHub-owned storage and will be hidden until GitHub-owned storage reaches general availability (GA). By default, archives are still uploaded to user-specified storage locations unless the --use-github-storage flag is explicitly set.

Motivation:

As discussed during our EDR, we want to allow users to utilize GitHub-owned storage as an option without making it the default. This approach lets us roll out GitHub-managed storage in stages and gather feedback while providing an explicit way to opt-in to this feature.

Check in BBS MigrateRepoCommandArgs removed, relating to changes in this PR: https://github.com/github/gh-gei/pull/1057

Take it out for a 🚗 :

Remember to export env variables

export GH_SOURCE_PAT=<<YOUR_PAT>>
export GH_PAT=<<YOUR_PAT>>

Migrating from GitHub Enterprise Server:

dotnet run --project src/gei/gei.csproj -- migrate-repo --github-source-org valet-testing --source-repo integration-tests --github-target-org octoshift-staging --target-repo YOU_ADD_NEW_NAME --use-github-storage true --ghes-api-url https://octoshift-ghe.westus2.cloudapp.azure.com/api/v3 --verbose

Successful migration_id: RM_kgHaACRlZjNiMmExMy02ZTRkLTQ0NWYtOTc4Yy05YTk1MTQ5NzE2OTk

Migrating from BBS:

Source repo: https://test-bbs-o.githubapp.com/projects/IM/repos/codeowners-test/browse You'll need to add yourself to the pinhole firewall at https://portal.azure.com/?wa=wsignin1.0#@githubazure.onmicrosoft.com/resource/subscriptions/7c45ec63-636b-445b-8185-54accbc0190d/resourceGroups/octoshift-test-bbs/providers/Microsoft.Compute/virtualMachines/bbs-test/networkSettings Here's the SSH key (and other creds) for our bbs-test-o instance: https://start.1password.com/open/i?a=LKXPAKKGYNBF7IOHIU6VPDFEU4&v=akrfryl3erhcqdcvgiqqtvh254&i=23dceluq4vdvpptr7qen3wcbiu&h=github.1password.com

dotnet run --project src/bbs2gh/bbs2gh.csproj -- migrate-repo --bbs-project IM --bbs-repo codeowners-test --ssh-user bbs --ssh-private-key /Users/begonaguereca/.ssh/id_rsa_bbs_octoshift --use-github-storage true --bbs-server-url https://test-bbs-o.githubapp.com --github-org octoshift-staging --github-repo ADD_NEW_NAME --verbose

Successful migration_id: RM_kgHaACRhODI4NWRmZS0yNWM3LTQ1YjEtYTJhOC1jYmZjNTkxZDE0Mjk

Closes: https://github.ghe.com/github/octoshift/issues/8969 Closes https://github.ghe.com/github/octoshift/issues/9535

[ ] Did you write/update appropriate tests
[ ] Release notes updated (if appropriate)
[ ] Appropriate logging output
[ ] Issue linked
[ ] Docs updated (or issue created)
[ ] New package licenses are added to ThirdPartyNotices.txt (if applicable)

github-actions[bot] commented 1 month ago

Unit Test Results

822 tests 822 :white_check_mark: 22s :stopwatch: 1 suites 0 :zzz: 1 files 0 :x:

Results for commit 60348006.

github-actions[bot] commented 1 month ago

Package	Line Rate	Branch Rate	Complexity	Health
gei	80%	71%	540	✔
ado2gh	84%	78%	627	✔
Octoshift	87%	77%	1286	✔
bbs2gh	79%	74%	666	✔
Summary	84% (6937 / 8302)	75% (1568 / 2084)	3119	✔

begonaguereca commented 3 weeks ago

Closing down bc we are addiing both single part and multi part logic together

github / gh-gei