Integrate more with github

[viankakrisna]

@CompuIves this is a continuation of our discussion in twitter DMs :D Right now when you import a repo and edit the files, you create a forked version of that repo in CodeSandbox. It would be nice if we have commit and PR button.

So basically we follow the GitHub PR flow.

1. Import from GitHub
2. Edit
3. Save => create a branch in the user GH account if not already, warns if the user leaves without committed changes
4. Commit => input the commit message
5. PR => Open a PR in the parent repo.
6. PR Merged => continue working from the parent repo (back to step 1).

Maybe the flow needs more discussion and planning, but that's what I have in mind right now :D

[CompuIves]

Yaaah, this is a very wanted feature and also a very interesting one!

I like your flow, I'd suggest some adjustments though.

I think it would be very beneficial if CodeSandbox would get a branches or snapshots feature in general. Where you can have these of any sandbox. This way you can create a branch of a git sandbox, and still switch back to master/other branches. Then the branch could have a PR button, so you create a PR on the main repo.

Looks a lot like your suggestions. But with the branch feature built in CodeSandbox. The architecture is already built for it, we save the source code in separate entities so you could swap these 'sources' out for a sandbox.

Now questions are:

1. How do we allow multiple users to create branches on one git repo. Do they kind of 'clone' the repo so they are working on their GitHub version?
2. How do we make it easy? When users edit a sandbox, do we give them the option to fork or clone? Or should we do this implicitly?

[viankakrisna]

I think it would be very beneficial if CodeSandbox would get a branches or snapshots feature in general. Where you can have these of any sandbox. This way you can create a branch of a git sandbox, and still switch back to master/other branches. Then the branch could have a PR button, so you create a PR on the main repo.

Nice! It can act as a 'local' git branch. I can't say anything more because I'm not familiar with CSB backend etc.

For simplicity, I think we can just follow the GitHub flow at first, so this feature is only for the sandbox that is imported from GitHub.

1. How do we allow multiple users to create branches on one git repo. Do they kind of 'clone' the repo so they are working on their GitHub version?

Follow the GitHub flow. On commit, we need to see if they have a write access to the repo. If they have it, ask to create a branch or commit directly. If not, fork the repo on GitHub and create a branch (codesandbox-patch-n?).

note: I'm not too familiar with GitHub API so I don't know if this functionality even exists / possible :D

2. How do we make it easy? When users edit a sandbox, do we give them the option to fork or clone? Or should we do this implicitly?

Just create a forked sandbox (like what's happening right now), and only follow this flow when they commit. The commit button would only be shown if the current sandbox is forked from the ones that imported from GitHub and the current user is associated with a GitHub account.

So the flow would be like this:

1. Open an imported from GitHub sandbox
2. Edit
3. Save => create a forked sandbox like it is right now.
4. Commit on GitHub => input the commit message, check for write access and gave options like stated above. After committing, all changes are saved to GitHub and current forked sandbox can be converted to be based on the created GitHub branch to avoid having many interstitial sandboxes.
5. PR => Open a PR in the parent repo. (this is nice to have, it can be delegated to GH interface)

Btw, I dream a full online workflow where I could be productive on any device without installing anything locally. So thank you for building the client in React and put it in here :D

[CompuIves]

Wow, this is funny. I think I misread your reply last week because I just wanted to reply with a suggestion of the exact flow you described!

Very eager to build this, I'm already experimenting with the GitHub API for a bit. So to go into the details:

1. Open import GitHub Sandbox
2. Edit
3. Save => create forked sandbox
4. Show "Commit on GitHub" button, but we always have one flow(!) a. If the user is not the owner => we implicitly fork (I think this is important, don't show confusing choices when the user almost always wants to do this). b. If the user is the owner => we use existing repo.
5. User can select files (only changed files), set a title, description and input branch (with autocomplete for existing branches, defaults to current branch. If branch is unknown we create it.)
6. After commit we either redirect if there's a new branch, or don't if the branch stays the same.

Would be great to also show a button to PR. Like you said, we can use the GitHub view for this.

[CompuIves]

Oh, just thought of this: we cannot allow the user to select an existing branch besides the branch the user has worked on. So the UI changes a bit: we show either existing branch, or the option to enter a new name (creation).

[CompuIves]

Would be good to make this step by step process, don't overflow the user with choices. Ooooh I'm really eager to build this, will start some mockups today!

[viankakrisna]

Wow, this is funny. I think I misread your reply last week because I just wanted to reply with a suggestion of the exact flow you described!

Hahaha, maybe the way I write it makes you misread it... English is haaaard.

User can select files (only changed files), set a title, description and input branch (with autocomplete for existing branches, defaults to current branch. If branch is unknown we create it.)

I think selecting files to commit is too much, maybe follow the behavior of git add --all && git commit -am "the commit message" ?

Oh, just thought of this: we cannot allow the user to select an existing branch besides the branch the user has worked on. So the UI changes a bit: we show either existing branch, or the option to enter a new name (creation).

Yup, so the option is only to commit in current branch (if in branch) or to create the commit in a new branch (give a text input to name it, maybe generate a name as default?).

Would be good to make this step by step process, don't overflow the user with choices. Ooooh I'm really eager to build this, will start some mockups today!

Exactly! That's why I think following GH flow is a nice first step :D

I'm so happy that you are interested in building this! 😄 I imagine that in the future people can use codesandbox for their work in a private repo. (would this be possible? private GH, GitLab, BitBucket?)

[billiegoose]

:grinning: This is perfect, exactly what I was thinking too!

[viankakrisna]

recently came across https://github.com/wmhilton/isomorphic-git, could it be possible to implement git terminal in CodeSandbox? :grinning:

[Hum4n01d]

Looking forward to this :) It would really help the github + react workflow

[CompuIves]

@viankakrisna isomorphic-git is great! Did some talking with @wmhilton, but the logic we implement (eg. GitHub PR) is pretty specific. That's why we thought we could better use the API for this.

Small update! I started working on this, I can now see the file differences between a sandbox and a repo. Now on to making the commit, then UI.

[billiegoose]

@viankakrisna It would! I talked with @CompuIves about possible integrations. Where isomorphic-git really shines right now is in offline support and creating commits. (It's probably the only JS library for that can validate and create PGP-signed commits.) It has a decent "git log" mechanism. However it's pretty heavy in terms of memory and IndexedDB usage. (IDB is really slow at writes for some reason so I have to mirror the virtual filesystem in memory, which doubles the speed.)

It also excels at being agnostic - you can clone and push to Github, Bitbucket, Gitlab... any server that supports the git "smart" HTTP protocol. However, it lacks Github-specific capabilities that rely on the Github API, such as pull requests.

I suspect that 'isomorphic-git' will eventually be integrated into CodeSandbox, but it might have to be combined with a refactoring of the virtual file system used by CodeSandbox to make it more compatible with the node "fs" (and hence BrowserFS) library. But I think that should be put off until I get a Cache based backend written, because right now the performance with the IndexedDB backend (speed & memory) usage is probably unacceptable.

[viankakrisna]

@wmhilton I see. So actually, it can be implemented regardless 😁

Small update! I started working on this, I can now see the file differences between a sandbox and a repo. Now on to making the commit, then UI.

hurray!

[billiegoose]

I started on a CacheStorage backend today. I hate being the bottleneck in the way of progress, and once I said out loud "wait until I make BrowserFS faster" I realized, "whoops I better actually do that then". Currently all the tests are failing but I'll post an update when I've got the new backend working with isomorphic-git.

[CompuIves]

This is great @wmhilton! Keep us posted 😄 .

[CompuIves]

It's here! We now have GitHub Commit, PR and Repo creation support. I'll keep this open, because I still want to move the UI around a bit + add a diff view.

[ecwyne]

This is all seriously so cool. The one last killer feature for me I think is the ability to work with private repositories.

Have you had any thoughts about supporting private repos @CompuIves?

And thank you SO much for codesandbox, it makes working with remote coworkers awesome!

[jimthedev]

Codesandbox hosts a github key

It seems like you would need the ability for codesandbox to generate an ssh key pair for a given github account in the settings. Then when a given codesandbox users is in a given repo the commits would need to be performed on a codesandbox server using the "safe" key. This could be dangerous if you don't trust codesandbox to keep your keys.

Local git strategy

Another way to do it would be for private repo commits to communicate with your local machine git via some sort of server. My guess is codesandbox's web-ui would then communicate commits to be performed over the postMessage api.

Regardless of which way you go you would also want to consider 2FA.

Thoughts?

[ryanpcmcquen]

I see that using 'Open PR' creates a branch. Is there a plan to allow naming this branch? Also, is there a plan to allow importing from branches and committing directly to certain branches?

[CompuIves]

Also, is there a plan to allow importing from branches and committing directly to certain branches?

This part is already done, just open the branch in GitHub and open that URL in CodeSandbox again.

Is there a plan to allow naming this branch?

Yes, that's a good one to add. Would you be interested in opening a separate issue for this?

[CompuIves]

Codesandbox hosts a github key

Agreed, we'd need to extend the rights to also have control over private repos. Which is absolutely possible, but imposes some extra checks we need to make (like th 2FA check).

Local git strategy

That would work, but we'd require the user to install something locally and it wouldn't work anymore for eg. iPads. So I'm not 100% sure how much the feature then would be used. Still worth exploring more though.

I'll close this issue for now, if there's something new needed we can open a new issue for it!

[kristianmandrup]

@CompuIves Would be nice with a smoother branching experience than having to go through import from github each time you want to checkout a branch. Why not just select the available branches from a drop down, then have it do the import behind the scene.