Import projects from GitLab & BitBucket

[fmsouza]

It would be great if I could import my projects from Gitlab too.

[harbin1053020115]

What's the plan to support import projects from gitlab?

[DenysVuika]

Are there any technical difficulties, or we just need to give you guys some time to do the integration? Thanks in advance.

[heavybeard]

News about this?

[mjyc]

Would love to see this happen as well!

[SeriousM]

Any updates on this? I would love to see the integration happen!

[purplem1lk]

Thank you for your patience - We are launching new core technology into beta next month that should resolve this issue so this will be closed accordingly. If this issue persists, feel free to tag me or ask to reopen this issue with screenshots of the error. Thank you! :)

[bschelling]

Am I too impatient? Is it next month already?

[snpranav]

This issue hasn't been solved on the hosted version of StackBlitz on stackblitz.com. Any updates on the GitLab integration @purplem1lk?

Please reopen this issue, so the integrations can be implemented.

[k33g]

@snpranav at the end I gave it a try to GitPod, since then, it's my day to day IDE with a perfect GitLab integration

[snpranav]

@k33g Same here, I just moved to GitPod recently, and I'm loving the prebuilds, and the ability to run multiple workspaces without seeing my computer struggle for resources to run dev servers/programs simultaneously :sweat_smile:

[janstadt]

Any update on this? Would really love to be able to keep track of modifications directly in our own self hosted instance of bitbucket datacenter.

[mpcref]

Just in case anyone wants to bring their interest for this issue to GitLab's attention: https://gitlab.com/gitlab-org/gitlab/-/issues/22414#note_818442359 https://gitlab.com/gitlab-org/gitlab/-/issues/197206#note_818438893

[mpcref]

@purplem1lk please reopen this issue. We're all eagerly awaiting GitLab support.

[FossPrime]

I heard 11 days ago @HeyGarrison will reopen this, and unleash a PR to end Github as we know it, and support open source git hosts.

[HeyGarrison]

Back by popular demand, I present this issue. ❤️❤️❤️

For clarity, this isn't currently on the roadmap. I did see an amazing demo around git today, but it'll launch with GitHub support initially and there is no timeline on launching.

However, since I have this opportunity... Does everyone here use both gitlab/bitbucket and GitHub? What are you using for what (I.e. is one for work and one for open source)?

[FossPrime]

We mainly use Gitlab for it's better CI, K8s and Wiki tools... the whole team is on it and we have zero need for Github otherwise... there is nothing I prefer using Github for. Migrating all the projects and tooling to Github would be a big ordeal. At a previous employer we had to use gitorious for some corporate clients, with the new provenance requirements after solarwinds that's a more common situation now. Finally we have a couple customers that manage content in their own git then webhook us when they change things.

In CodeSandbox that's a non issue as git is preinstalled and we can use /tmp for cloning/pushing or any other temporary git integration.

As knative and flux mature, I might considering moving Google Cloud Repositories, which Similar to Azure Repos, can simplify ACL, reduce upstream feature deprecation uncertainty, lets us better control performance.

I've confirmed the following workaround works... though it requires a clunky cors proxy server to run:

npx isomorphic-git clone --corsProxy="http://cors.localhost:9999" --url=https://gitlab+deploy-token-420:pasuwaado@gitlab.com/mygroup/my-repo.git

Btw, you really should not advertise "Advanced Git operations (beta)" when you actually mean Github... that was disappointing... I still don't know what that means. I see Github like I do my ISP, an interchangeable pipe with nothing special. The conversation of why we don't use one Git vendor specifically seems backwards when git is one of the most standard protocols out there. I understand why Chrome is required, it has something special that you really need.

Github isn't even open source, which is plain ridiculous and does matter when using features that aren't documented with spec level detail... if I have an issue with the Gitlab CI charts, I can pull them up and file PR with them, if I need the exact regex they are using to parse auto closing of issues, I can do that too. If the IME for editing markdown has an odd bug that we are running into, we can file a bug, see the code and check the commit history.

Here's the vscode.dev UI for "Connect repository", from the owner of Github:

[FossPrime]

After trying to live with the isomorphic-git workaround, I encountered a show stopping push issue with web containers. I tried to debug it by importing the isomorphic-git repo to the Stackblitz IDE, but it's 1000+ commits seem to bog down the IDE to the point that it's unusable. The only way to debug it seems to modify node_modules directly.

Here's my new workaround, use GH as a proxy for Gitlab, powered by a GH action. MY_GH_KEY is just an SSH private key I've loaded into Gitlab as a repository Deploy Key with write access.

name: CI
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]
jobs:
  mirror:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      # Runs a set of commands using the runners shell
      - name: Push to Gitlab
        env:
          MY_GH_KEY: ${{ secrets.MY_GH_KEY }}
        run: |
          mkdir -p ~/.ssh
          echo "$MY_GH_KEY" > ~/.ssh/id_ed25519
          ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts
          chmod 700 ~/.ssh
          chmod 600 ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/known_hosts
          git push git@gitlab.com:steam/deck-os.git

[FossPrime]

HUGE UPDATE: isogit can now push in stackblitz... Confirmed working in Gitlab, and surely in Github too (should you not like the default git flow). Note, Gitlab Deploy Tokens won't work as they can't be configured with write access, Access Tokens for your account however CAN push to repos you have access to.

Example, Bring Your Own Secrets...

Related issues:
https://github.com/stackblitz/webcontainer-core/issues/654 https://github.com/isomorphic-git/isomorphic-git/issues/1523

[FossPrime]

https://stackblitz.com/edit/koa-starter-firdac?file=index.js Proxy server: https://runkit.com/hesygolu/isogit-cors-proxy

This is the most seamless integration of isomorphic-git i've managed yet... It's using an open source FREE Runkit worker as a proxy to pull code from git, without using closed-source APIs from third parties... The proxy code can run anywhere... I've run it on Runkit, localhost and Kubernetes, but CFW and Netlify should work as well.

Unfortunately this is implementation is clone only, bidirectional stuff is possible. I have too many nightmares from CSB's sandbox sync'ing to muster up the courage to properly try it on the package root of a StackBlitz project, but I've tried it in /tmp/myrepo and it worked. I made g4c a while ago for CSB... that worked well enough, but CSB is sooooo slow. I also think I made it work on Fastly's Glitch, which is somehow way way slower still. I should really try to port that to StackBlitz, but it was designed to work around CSB's awful glitchy filesystem sync, so I should probably fork and strip all the dumb things I had to do to make it work on CSB.

You may use my proxy server, but I don't see why you would want to... it takes 3 seconds to click the fork button on runkit, at which point it's ready for use. If you use my cors-proxy runkit endpoint I will be able to see what repo's you're cloning like so:

sidenote... CSB claims it has 300K weekly downloads of Gitlab specific sandboxes...

[FossPrime]

New example: https://stackblitz.com/edit/feathers-vite-chat?file=.force-update.mjs

This one just clones a repo and overwrites a subfolder on top of the stackblitz project folder. Could easily be modified to do other things.... to run use node .force-update.mjs

Update: How to push all changes using a similar method:

This works best if have a premium account where you can keep a SECRETS.json file with a project access token or a personal access token that has push permissions. Otherwise you should add SECRETS.json to your .gitignore which will make StackBlitz not save it; You'll have to remake it every time you push, this can be made easy with a StackBlitz user snippet. This code will never commit SECRETS.json to git, regardless of .gitignore.

Code for the git.mjs push script

```js // git.mjs -- push all changes to Git // Usage: `node git.mjs` import { fetch, clone, fastForward, currentBranch, add, commit, push, statusMatrix } from 'isomorphic-git' import { default as isomorphicGitFsClient } from 'node:fs' import path from 'node:path' import { readFile } from 'node:fs/promises' import { URL } from 'node:url' // HARD CONFIG const PUSH = true const CHECKOUT = false // DANGER: true WILL overrite your SB code const FETCH = true // BUG: Overwrites any unpushed commits when true const NS = 'git-' + Date.now() const SECRETS_FILE_PATH = 'SECRETS.json' const GIT_COMMIT_MESSAGE = 'StackBlitz Commit.' const { GIT_SOURCE_PROXY, GIT_SOURCE_URL, GIT_SOURCE_USERNAME, GIT_SOURCE_PASSWORD, GIT_SOURCE_AUTHOR, GIT_SOURCE_EMAIL } = JSON.parse( await readFile(SECRETS_FILE_PATH, { encoding: 'utf-8' }) ) const isomorphicGitHttpClient = await import( './node_modules/isomorphic-git/http/node/index.js' ) const isomorphicGitWorkingTreeDir = './' const PROJ = path.basename(path.resolve('.')) if ( path.resolve('/home/projects/' + PROJ) !== path.resolve(isomorphicGitWorkingTreeDir) ) { throw new Error( `${NS}: Current working directory is not a stackblitz project!` ) } const gitUrl = new URL(GIT_SOURCE_URL) if (GIT_SOURCE_USERNAME) { gitUrl.username = GIT_SOURCE_USERNAME gitUrl.password = GIT_SOURCE_PASSWORD } const isomorphicGitUrl = gitUrl.toString() // const workdir = tmpdir() + '/' + NS const gitConfig = { fs: isomorphicGitFsClient, dir: isomorphicGitWorkingTreeDir, cache: {} } const gitRemoteConfig = { http: isomorphicGitHttpClient, corsProxy: GIT_SOURCE_PROXY, url: isomorphicGitUrl } const gitClone = async () => { await clone({ ...gitConfig, ...gitRemoteConfig, singleBranch: true, noCheckout: !CHECKOUT, depth: 1 }) } const gitFastForward = async () => { const params = { ...gitConfig, ...gitRemoteConfig, singleBranch: true // corsProxy: GIT_SOURCE_PROXY, we don't need this as it's saved in repo config. } if (CHECKOUT) { await fastForward(params) } else if (FETCH) { await fetch(params) } else { throw new Error( `${NS}: FastForward failed. Nither fetch, nor checkout are enabled.` ) } } const gitCurrentBranch = async () => { try { const branch = await currentBranch({ ...gitConfig, fullname: false }) return branch } catch (e) { if (e.code === 'NotFoundError') { return '' } else { throw e } } } const makeMapReversible = (map) => map.forEach((v, k, m) => m.set(v, k)) const HEAD_STATUS = new Map([ ['absent', 0], ['present', 1] ]) const WORKDIR_STATUS = new Map([ ['absent', 0], ['identical_to_head', 1], ['different_from_head', 2] ]) const STAGE_STATUS = new Map([ ['absent', 0], ['identical_to_head', 1], ['identical_to_workdir', 2], ['different_from_workdir', 3] ]) const STATUS_MAPS = [HEAD_STATUS, WORKDIR_STATUS, STAGE_STATUS] STATUS_MAPS.forEach(makeMapReversible) const gitStatus = async () => { const matrix = await statusMatrix({ ...gitConfig, filter: (f) => f !== SECRETS_FILE_PATH }) return matrix } const gitAdd = async (matrix) => { const result = { unchanged: 0, added: [], removed: [], other: [] } for (const status of matrix) { const [filePath, headStatus, workdirStatus, stageStatus] = status if (workdirStatus === WORKDIR_STATUS.get('different_from_head')) { result.added.push(filePath) add({ ...gitConfig, filepath: filePath }) } else if ( headStatus === HEAD_STATUS.get('present') && workdirStatus === WORKDIR_STATUS.get('identical_to_head') ) { result.unchanged++ add({ // isomorphic-git's commit function will delete anything not in stage ...gitConfig, filepath: filePath }) } else if ( headStatus === HEAD_STATUS.get('present') && workdirStatus === WORKDIR_STATUS.get('absent') ) { // remove not neccessary due to isomorphic-git quirk result.removed.push(filePath) } else { result.other.push(status) } } if (result.other.length > 0) { const prettyOther = result.other.map( ([filePath, headStatus, workdirStatus, stageStatus]) => { console.log({ filePath, headStatus: HEAD_STATUS.get(headStatus), workdirStatus: WORKDIR_STATUS.get(workdirStatus), stageStatus: STAGE_STATUS.get(stageStatus) }) } ) console.warn( `${NS}: addMatrix: stage is in an unexpected state. Please stash your changes to the stage.`, JSON.stringify(prettyOther, null, 2) ) } return result } const gitCommit = async (message) => { const sha = await commit({ ...gitConfig, author: { name: GIT_SOURCE_AUTHOR, email: GIT_SOURCE_EMAIL }, message }) return sha } const gitPush = async () => { const pushResult = await push({ ...gitConfig, ...gitRemoteConfig, remote: 'origin' // probably unnecesary }) return pushResult } const main = async () => { const currentBranch = await gitCurrentBranch() if (currentBranch === '') { await gitClone() } else { await gitFastForward() } const matrix = await gitStatus() const addResult = await gitAdd(matrix) if (addResult.added.length > 0 || addResult.removed.length > 0) { const message = GIT_COMMIT_MESSAGE + '\n' + JSON.stringify(addResult, null, 2) console.log(message) await gitCommit(message) } else { console.log('Nothing to commit.') } if (PUSH) { await gitPush() console.log('Push successful.') } } // Bootstrap if (import.meta?.url?.endsWith(process.argv[1])) { await main() } ```

Update 2: updating g4c

I'm working on an overhaul to g4c to make it work with StackBlitz, drop support for CodeSandbox, and drop the binary git dependency. It will handle about a dozen of the most common git commands. The api will be npx g4c pull etc. Secrets will load from SECRETS.g4c.json. Any files that start with SECRETS. will be ignored. Config will be merged with package.json -> g4c, so repoUrl could public.

Making cloning from a monorepo tree sub directory cloning work with a generic interface is a challenge, but I'll try to make it work. It's probably going to involve a subDir config next to repoUrl, and it may be noticeably slower than not using subDirs. That's due to the need to copy the whole project around the virtual filesystem to keep the IDE happy. In practice a combo command could make the performance hit negligible, such as npx g4c clone+add+commit -m 'fix emoji support.'.

[FossPrime]

It lives!

https://www.npmjs.com/package/g4c https://gitlab.com/vblip/g4c

It supports the following commands in StackBlitz:

git checkout --force HEAD
git pull --ff-only
git status
git add --all
git commit -m "My commit message"
git push

Something like this should obviously be part of SB.... but it's better than nothing.

[bergamin]

Back by popular demand, I present this issue. ❤️❤️❤️

For clarity, this isn't currently on the roadmap. I did see an amazing demo around git today, but it'll launch with GitHub support initially and there is no timeline on launching.

However, since I have this opportunity... Does everyone here use both gitlab/bitbucket and GitHub? What are you using for what (I.e. is one for work and one for open source)?

I have both for personal projects. I started using GitHub earlier and it's kind of a mess right now. I've been using it since the beginning of my career and git studies.

GitLab is where I'm putting more of my "professional grade" personal projects. It had private repos for free before GitHub had as well. Also, I think it's better overall than GitHub. The only real downside of GitLab is that it doesn't have as much spotlight as GitHub and that it is more expensive than GitHub (I'm still milking the free content of both though).

[FossPrime]

Regarding Codeflow:

It's painfully close. Any git server works... once you're running Codeflow on top of a GitHub repo.

$ git clone https://FossPrime:glpat-123456@gitlab.com/FossPrime/codeflow-test.git
$ rm -rf .git
$ mv codeflow-test/.git .

From there you can go to Source Control -> Discard all changes, that has some bugs right now with sub-directories. git stash, and reset --hard, merge and checkout --force are all not implemented. Instead, you could try hoperyy/sync-directory instead. pr.new/mygitserver.com/repo.git will not work, even if it's a public repo that doesn't require credentials. There is no word on secrets support, for storing access tokens to git servers securely.

SB CodeFlow therefore takes the following two steps back in terms of Git support:

1. You can no longer save changes without git. A crash will mean you'll lose all your changes. (ETA Q2 2022)
2. You can no longer save secrets to access Git servers in SB storage with a gitignore'd file. The only way to manage secrets right now is to commit secrets to git, a security nightmare.

---

The best way forward right now seems to be making a stub GitHub repository with a .stackblitzrc file that deletes .git/ and clones your git repo into the current directory. Alternatively, you can mirror the repo to GH and then use a postinstall script to git remote set-url origin ... to the real Git repo.

[FossPrime]

The git CLI approach above worked well for a single user. Unfortunately collaborators can't use private Github repos with Codeflow unless the repo is in a GH organization. A couple other things have changed as well, including a suspicious env var that hints at git CLI being available in StackBlitz classic, as well as git checkout origin/main working correctly.

More importantly, we now have decent git reset --hard origin/main support. Watch out for any git CLI commands that may change the .git/HEAD file to a git commit hash, rather than a branch head pointer like ref: refs/heads/main, SB's CLI doesn't support the hash format.

This means you can use stackblitz.com/~ combined with a global user Snippet for shell scripts, and clone/commit from any git server that supports login via access tokens, like Gitlab. The main downside right now is it's not fully automated and your secret access token will get printed in plain text for everyone to see, every time you commit.

Better credential management would be nice... there is a WASM SSH client that would be a god send if integrated into CodeFlow.

Example shellscript.json User snippet:

{
    "Print to console": {
        "prefix": "gitlab clone",
        "body": [
            "rm main.sh",
            "git init",
            "git remote add origin https://you:glpat-abc123@gitlab.com/myorg/${1:myrepo}.git",
            "git fetch origin main",
            "git reset --hard origin/main",
            "git push --set-upstream origin main",
            "npm i",
            "npm run dev"
        ],
        "description": "Call with `bash main.sh`. Clone $2 from gitlab using my access token."
}

This could be improved by engraving ~/.jshrc with project secrets from a temporary secrets.json file, or a dedicated secrets/jshrc repo.

[AmazingGinger]

I gave up and just used this... the Git UI is way better than VS Code's

https://replit.com/@fossprime/Node-18