Feature: Add non-local repository

larsnystrom commented 4 years ago

I work a lot on code that lives on a different machine than the one I'm sitting at. I'm accessing the code over SSH.

At the moment, Fork can only open repositories that live on your current computer. I could possibly use sshfs to mount the remote host locally and then open it in fork, but this is pretty complicated on MacOS because it requires me to install osxfuse (which is closed source and tends to break for every new MacOS release) and sshfs and make sure that the mount point keeps working.

The work-around I'm using right now is to clone the code to my local computer and then add the remote host as a remote in my local repository. Then I can cmd+shift+f to fetch the latest changes in Fork.

But it would be so much easier if I could just add a new repository in fork using a normal url like me@some-machine:projects/my-project. Sometimes that remote repository may be a bare repository, but it will always be something that git remote add would accept.

Fork is such an amazing application and this feature would let me explore repositories on various hosts without having to clone them first.

Thank you!

DanPristupov commented 4 years ago

The work-around I'm using right now is to clone the code to my local computer and then add the remote host as a remote in my local repository. Then I can cmd+shift+f to fetch the latest changes in Fork.

I think this is the correct solution there.

larsnystrom commented 4 years ago

Yes, but there are two things that could be improved in that workflow:

It would be handy if I could give Fork a URL and it would clone the repository for me (to some random location somewhere on my computer, the point here is that I should only have to provide a URL and then the repository would open in Fork).
When looking at the the repo in Fork, "Branches" should list the branches of origin and "Remotes" and "Stashes" should be hidden. Basically, I want to see the repository as if I was on the origin machine (the one I cloned the repo from).

DanPristupov commented 4 years ago

When looking at the the repo in Fork, "Branches" should list the branches of origin and "Remotes" and "Stashes" should be hidden.

The point is that this workflow is very rare. We don't have much resources for implementing and, more importantly, maintain the code for this feature in the future. I'd rather keep the Fork source code simple.

larsnystrom commented 4 years ago

That's totally understandable. It was an idea I had to simplify the workflow I'm using every day so I thought I'd share it. But I understand it's not possible to implement all the things.

Feel free to close this issue if you decide not to pursue this particular feature. And thank you for Fork! I'll go buy a license now, because I'm using Fork every day for my work.

masonmark commented 4 years ago

I think there is a sea change in development going on right now that might merit taking a second look at this feature. Namely: almost everybody I know seems to be switching from local development to developing remotely using something like this:

https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh

The idea of "cloud IDE in the browser" hasn't really panned out for most workloads, but this VS Code feature (and similar ones coming out for other editor/IDEs) let you keep the source code in the cloud, you build and run in the cloud, but do your programming locally on your Mac with a reasonable editor and good performance. This is becoming popular, I think, because you can set up the dev env once, and then just sit down at any machine with VS Code installed and do the programming part. You don't need to set up your client machine with all the dependencies, and you can have the cloud machine be as beefy and powerful as you need it to be.

What I am seeing my team — and me too — do in this environment is to stop using our favorite Git clients (mine is Fork, some co-workers use Tower or whatever), and just go back to relying on command-line git, and and maybe the VS Code git UI.

We all use GUI git apps to make our lives more convenient. There's a free, functional command line available in which we can do everything we need to do, it is just nicer and faster to do it using a nice GUI. So while I do indeed use Fork any time I am developing something locally, I find myself doing that less and less — and so I am using Fork less and less.

Cloning a remote repo to my Mac, waiting for that to finish, then opening it and navigating to the file I am interested in and then finally viewing the diff or whatever in Fork — it is nicer in the end result than looking at it on the command line or in VS Code's slightly weird git thing, but it takes way longer and so I don't do it.

I think eventually the ability to open a remote git working copy via SSH will probably become a table-stakes feature for GUI git clients. I guess today Fork does do a lot of direct local filesystem I/O, but theoretically it could all be done via SSH instead, for "SSH remote working copies".

Will probably be a required feature in the future and indeed would be immediately useful to many of us today.

larsnystrom commented 4 years ago

@masonmark Well put. I’m also using VSCode SSH remote extension. It works like magic. Right now I’m keeping local copies of the repos only to look at them with Fork.

Noleli commented 3 years ago

I have mounted the remote using sshfs to use Fork, but it’s painfully slow. (My use case is with a VM hosted locally, so I also tried SMB; it’s similarly slow.)

DanPristupov commented 3 years ago

@Noleli I myself use Parallels VM and the performance between parent (macOS) and child (Windows) is almost the same as the native performance.

My sharing settings are pretty standard:

Noleli commented 3 years ago

Thanks, @DanPristupov. VirtualBox only lets you share a folder on the host to the guest. The mounted volume on the Linux guest doesn’t seem to support symlinks, named pipes, or other special file types and permissions. So I’m stuck with trying to find a way to make the guest filesystem available to the host.

brechtm commented 2 years ago

I have mounted the remote using sshfs to use Fork, but it’s painfully slow. (My use case is with a VM hosted locally, so I also tried SMB; it’s similarly slow.)

I had the same experience (although I was using rclone instead of sshfs for the mount). Looking for a solution, I discovered sshfsexec, which I thought should be able to provide a solution for SSH similar to wslgit for Fork and WSL (fork-dev/TrackerWin#305) which I have been using succesfully for months. This did require some additional work to for Fork to be able to open a remote SSH repository, but nothing too outrageous. It turns out it's not even necessary to mount the remote directory as Fork seems to rely mostly on git calls to interact with the repository. It suffices to create an empty directory on your machine and provide a corresponding mapping in sshfsexec. I did notice that the list of stashes is missing, so I suspect Fork does access files in the .git directory directly for this. Another quirk is that Fork seems to loose track of which commit is currently checked out under certain conditions.

While not as fast as accessing a local repository, browsing the commit history seems to be fast enough to be practically useable. Committing is not working yet as the commit message file needs to forwarded to the remote, but that shouldn't be difficult to manage.

Interactive rebase is however a more complicated story, because of the interaction with fork_ri. I embarked on a small adventure trying to forward the editor calls on the remote server back to fork_ri on the host. This works well for the git-rebase-todo file, but for some reason fork_ri does not write changes to the _COMMITEDITMSG file when rewording a commit. @DanPristupov Perhaps you have an idea what I'm overlooking here? Any idea why this works for git-rebase-todo but not for _COMMITEDITMSG. Thanks!

DanPristupov commented 2 years ago

@brechtm fork_ri reports errors to the IR output. Do you see any error dialog when the problem occurs?

Regarding what's going on in fork_ri. The tool is called with a path argument which is normally /.git/COMMIT_EDITMSG. Here's is the simplified source code:

            var outputTodoPath = args[0];
            var messageArchivePath = outputTodoPath; // /.git/COMMIT_EDITMSG
            messageArchivePath = Path.GetDirectoryName(messageArchivePath); // /.git/
            messageArchivePath = Path.Combine(messageArchivePath, "rebase-merge"); // /.git/rebase-merge
            messageArchivePath = Path.Combine(messageArchivePath, "fork-message-archive"); // /.git/rebase-merge/fork-message-archive
...
            var message = ReadMessage(messageArchivePath);
...
            File.WriteAllText(outputTodoPath, message);

brechtm commented 2 years ago

fork_ri reports errors to the IR output. Do you see any error dialog when the problem occurs?

No, I didn't see any error. However, I was not aware of the role of fork-message-archive. git-rebase-todo and _COMMITEDITMSG were copied over from the remote to a tmp directory on the host (so, there was no rebase-merge directory).

I now recreated the .git structure (just the .git/rebase-merge/git-rebase-todo and _.git/COMMITEDITMSG) in the temporary directory and I see that the fork-message-archive (containing the new commit message) file is being created in the rebase-merge directory. Alas, the new message is still not being written to _COMMITEDITMSG.

Is there any other data being passed between Fork and fork_ri that might be getting lost? Perhaps through environment variables or interprocess messaging? Does fork_ri read any other files besides the ones mentioned here?

Somewhere along the debugging path I saw an error mentioning .git/rebase-merge/done. Is that file also processed by fork_ri, and at what point?

Many thanks for the pointers!

DanPristupov commented 2 years ago

I guess, the reason is simple. File.WriteAllText(outputTodoPath, message); accesses the file system. Most likely it's the same problem as the one with stashes and HEAD which you mentioned before. How does the absolute path looks in your case?

brechtm commented 2 years ago

I guess, the reason is simple. File.WriteAllText(outputTodoPath, message); accesses the file system. Most likely it's the same problem as the one with stashes and HEAD which you mentioned before. How does the absolute path looks in your case?

That shouldn't be the issue here. I start a second process on the host that downloads (scp) git-rebase-todo and _COMMITEDITMSG (after git creates them), calls fork_ri on them and uploads them. But fork_ri doesn't write the new commit message to _COMMITEDITMSG, even though that works fine for git-rebase-todo.

Some other things I tried, without success:

downloading the files to the usual location (in the checkout dir's .git folder)
with the remote files mounted locally (not using sshfs but rclone which I happen to have installed)

In the latter case, Fork has access to all files, so it should work if no data is being passed in another way. But perhaps there is some caching going on that breaks things.

Actually, the stashes don't show up even when the remote files are mounted locally, so I'm not sure what the reason is.

DanPristupov commented 2 years ago

Actually, the stashes don't show up even when the remote files are mounted locally, so I'm not sure what the reason is.

Did you check the log file %localappdata%\fork\logs\fork.log? It may contain some info.

brechtm commented 2 years ago

Did you check the log file %localappdata%\fork\logs\fork.log? It may contain some info.

I'm on Mac, so I am checking ~/Library/Logs/Fork.log, yes.

In any case, thanks for your help! I'll try to have another look at what could be the problem soon. I'll post updates here.

GregoryR13 commented 2 years ago

I'm interested in the topic because I'm using sshfsexec myself and only the "git rebase --interactve" (all git rebase finally) on remote doesn't work because client ide expects an editor return from my remote server. "Error rebasing root intranet: Terminal is dumb, but EDITOR unset"

brechtm commented 2 years ago

I haven't looked into this further for now and I don't know when I will. If that happens, I'll post an update here.

brechtm commented 2 years ago

My attempt at getting this to work is now available here: https://github.com/brechtm/sshgit

I haven't yet revisited getting interactive rebase working. IIRC, _forkri was also used for plain commits in the past, but that no longer seems to be the case (using git commit --file=/tmp/.../commit_msg_file instead). Perhaps interactive rebases are now also simpler to handle 🤞

DanPristupov commented 2 years ago

@brechtm Fork reads many files directly from the repository folder. Does it handle that?

brechtm commented 2 years ago

Not without an SSH mount. I haven't really noticed any major issues except the fact that it's not possible to delete untracked files. I have only tested things briefly though. Is there anything in particular that I can test?

DanPristupov commented 2 years ago

@brechtm

This should be a concern.

I haven't really noticed any major issues except the fact that it's not possible to delete untracked files.

:(

That's only the beginning.

As for now Fork reads many files, including:

.git/COMMIT_EDITMSG .git/HEAD .git/hooks/ .gitmodules .git/fork-settings .git/fork/user-colors

And we are going to read even more files directly.

Generally we use String(contentsOf:URL, encoding:String.Encoding) https://developer.apple.com/documentation/swift/string/init(contentsof:encoding:)

Is there any way we can make it more accessible for you?

brechtm commented 2 years ago

So far, the sshgit setup seems to work well enough to be practical. Using an SSH mount should fix issues with Fork reading files directly, but I think FUSE (needed for SSH mounts) has stability issues on macOS. Alternatively, sshgit could perhaps copy over a (small) set of remote files after each git call? But that wouldn't allow Fork to reflect changes to the checkout that are made independently on the command line.

Is there any way we can make it more accessible for you?

Having a list of files that are accessed directly, and for which purpose would be very useful information. Of course, avoiding to access files directly would make things much easier, but I'm sure there are good reasons to do that.

Of course, it would be better if Fork could offer native remote-SSH support, but I understand that this would be far from trivial to implement and support, especially if you are planning on accessing more files directly. Are you considering supporting this is some way? I don't suppose many users are interested in this.

brechtm commented 1 year ago

I've tested this setup for a couple of days and I'm happy to say that it works pretty well. I've documented some of the issues I've experienced in the sshgit README. While not at all a show-stopped, this is the one thing that annoys me most at the moment:

the current branch and HEAD commit are not indicated properly (commits on the current branch are normally displayed in black and the HEAD commit should be displayed in bold text)

New commits are shown in grey and the same commit stays highlighted (bold). The current branch is marked in bold in the sidebar, however.

@DanPristupov How does Fork determine the current branch and HEAD commit? I tried copying over .git/HEAD and .git/refs after each git call, but besides slowing things down a lot it didn't fix that.

metafeather commented 1 year ago

Using Fork with sshgit is brilliant!

My usecase is that I am at a (famous) company with does not allow source code on laptops (including SSHFS/Syncthing/etc).

Being able to use a native app to efficiently inspect diffs and make commits is already a massive increase in productivity for me.

Some suggestions for minimal changes that may help adoption and reduce risk:

Allow the selection of a git binary on a per repo basis so that customisations like these can be isolated.
Change all other direct filesystem calls pass through a cli that can also be mapped to a separate binary by sshfsexec (even if necessary it is within the Fork.app directory).
Gate the advanced features of Fork if operating on a remote repo.

One mechanism to do this may be to read a Fork specific config from the repo itself, e.g. .fork.json to allow advanced users to opt-in to experimental non-UI exposed features.

Thanks for your consideration.

DanPristupov commented 1 year ago

@brechtm

How does Fork determine the current branch and HEAD commit?

.git/refs/HEAD.

I tried copying over .git/HEAD and .git/refs after each git call, but besides slowing things down a lot it didn't fix that.

You must also need to replicate the original file modification dates.

@metafeather

Some suggestions for minimal changes that may help adoption and reduce risk:

The real problem there is that Fork must read many files directly and I don't know how to do that over SSH.

brechtm commented 1 year ago

@DanPristupov The sshgit setup breaks fetching for local repositories. I was hoping you would have a clue why this happens:

$ git fetch --prune --all --verbose

git: 'remote-https' is not a git command. See 'git --help'.
error: could not fetch origin

Calling .../sshgit/bin/git fetch --prune --all --verbose from a terminal works, however.

PS I'm still happily using Fork + sshgit on a daily basis 😄

DanPristupov commented 1 year ago

git: 'remote-https' is not a git command. See 'git --help'.

It looks like an ENV problem.

Try to start Fork from terminal using open -a Fork . command to make it inherit ENV of the terminal.

brechtm commented 1 year ago

Try to start Fork from terminal using open -a Fork . command to make it inherit ENV of the terminal.

Same error :-(

You are talking about the environment variables, right? Any clue as to which could be related?

DanPristupov commented 1 year ago

I guess, git can't find git-core directory which must contain the git-remote-https file. Usually git-core is in PATH.

You can find out the location of git-core by running git --exec-path on the remote computer. https://stackoverflow.com/questions/42008055/how-can-you-determine-the-location-of-the-git-core-folder-dynamically#42008526

However, I don't know how to pass it to sshgit.

I've got an idea how to debug that. Create a pre-commit hook which prints PATH to the output echo $PATH and make a dummy commit from CLI and from Fork. You will see the difference in the output.

brechtm commented 1 year ago

Great idea! Thanks for the tip.

The PATH contents when committing from the command line:

/usr/local/Cellar/git/2.36.1/libexec/git-core
... (the entries common to both)

The PATH contents when committing from Fork:

/Users/brechtm/Code/sshgit/libexec/git-core
/Applications/Fork.app/Contents/Resources/git-instance/git-lfs
/Applications/Fork.app/Contents/Resources/gitflow-avh
/opt/homebrew/bin (doesn't exist on my system)
/usr/local/bin
... (the entries common to both)

/Users/brechtm/Code/sshgit/libexec/git-core is empty, but exists to trick Fork into accepting sshgit as a valid git instance. But that doesn't stop sshgit from working from the command line. From a local checkout, .../sshgit/bin/git --exec-path returns /usr/local/Cellar/git/2.36.1/libexec/git-core, which is what we want. From a remote checkout, it returns the git-core path on the remote. Fetching from a remote checkout works fine however.

I don't understand why, when running from Fork, the PATH includes /Users/brechtm/Code/sshgit/libexec/git-core instead of /usr/local/Cellar/git/2.36.1/libexec/git-core.

brechtm commented 1 year ago

Calling git --exec-path from the pre-commit hook corresponds to the PATH contents in both contexts:

command line: /usr/local/Cellar/git/2.36.1/libexec/git-core
Fork: /Users/brechtm/Code/sshgit/libexec/git-core

I think Fork sets GIT_EXEC_PATH, which could explain this behavior. Is it necessary for Fork to set it, since git sets it automatically?

Soft-linking sshgit's git-core to /usr/local/Cellar/git/2.36.1/libexec/git-core seems to fix the issue.

brechtm commented 1 year ago

I've updated ~sshgit~ forkgit to address the GIT_EXEC_PATH issue. I've also made changes to how remote checkouts are configured and added an option to force setting GIT_PATH. For details, see the README at https://github.com/brechtm/forkgit.

brechtm commented 1 year ago

the current branch and HEAD commit are not indicated properly (commits on the current branch are normally displayed in black and the HEAD commit should be displayed in bold text)

I've had another look into this and I discovered Fork checks the timestamp of .git/logs/HEAD (logs/HEAD for submodules, for some reason) as part of determining the current HEAD commit. Just touching it (the contents do not matter) on every forkgit call seems to be enough to finally fix this issue!

@DanPristupov Is this really necessary? Can't the same be achieved by means of a git command, e.g. git rev-parse HEAD? I have the impression having to touch these HEAD files slows things down noticably...

samskiter commented 1 year ago

Has anyone worked out how to use Fork Git with Codespaces? The SSH access is non trivial, but might be exposable through a VSCode extension?

DanPristupov commented 1 year ago

@samskiter Fork needs a direct access to the repository files. I don't think it's possible with Codespaces.

brechtm commented 1 year ago

@samskiter I haven't used Codespaces myself, but if you can add ~/.ssh/config entries to enable password-less login to the remote, you should be able to get it to work with forkgit.

Alternatively, you could hack forkgit to call gh codespace ssh instead of ssh.

samskiter commented 1 year ago

I think when using vscode it has SSH access to the files (using something like gh codespace ssh)

On Wed, 16 Aug 2023, 10:58 Dan, @.***> wrote:

@samskiter https://github.com/samskiter Fork needs a direct access to the repository files. I don't think it's possible with Codespaces.

— Reply to this email directly, view it on GitHub https://github.com/fork-dev/Tracker/issues/929#issuecomment-1680312971, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJWOW7PCG7HLDRE7TIBHSDXVSKSXANCNFSM4LCEMGVA . You are receiving this because you were mentioned.Message ID: @.***>

brechtm commented 1 year ago

I just noticed the example in the gh codespaces ssh man page that looks like it makes setting up ~/.ssh/config trivial:

$ gh codespace ssh --config > ~/.ssh/codespaces
$ printf 'Match all\nInclude ~/.ssh/codespaces\n' >> ~/.ssh/config

brechtm commented 1 year ago

@DanPristupov Did recent versions of Fork change how stashes are retrieved? They are no longer listed for 'remote' forkgit checkouts. I don't see anything relevant mentioned in the release notes.

DanPristupov commented 1 year ago

Did recent versions of Fork change how stashes are retrieved?

Yes, there are internal changes. You can download 1.31.2 here: https://cdn.fork.dev/mac/Fork-2.31.2.dmg

Returning to the old question, why Fork reads .git/logs/HEAD. It's just a small optimization. Fork checks if the HEAD file modification date is different, if it is, then it performs git rev-parse HEAD and if it's changed, refreshes the commit list (which is the heaviest operation in Fork).

brechtm commented 1 year ago

Yes, there are internal changes. You can download 1.31.2 here: https://cdn.fork.dev/mac/Fork-2.31.2.dmg

Thanks. Can you give a hint as to what changed? That can be a real time-saver 😄

Returning to the old question, why Fork reads .git/logs/HEAD. It's just a small optimization. Fork checks if the HEAD file modification date is different, if it is, then it performs git rev-parse HEAD and if it's changed, refreshes the commit list (which is the heaviest operation in Fork).

👍

DanPristupov commented 1 year ago

Thanks. Can you give a hint as to what changed? That can be a real time-saver 😄

Now Fork reads stashes directly from the repo. I think it will be difficult to handle in your case.

samskiter commented 1 year ago

I just noticed the example in the gh codespaces ssh man page that looks like it makes setting up ~/.ssh/config trivial:
$ gh codespace ssh --config > ~/.ssh/codespaces
$ printf 'Match all\nInclude ~/.ssh/codespaces\n' >> ~/.ssh/config

I think that the SSH config changes over time - I might be wrong.

brechtm commented 1 year ago

Thanks. Can you give a hint as to what changed? That can be a real time-saver 😄

Now Fork reads stashes directly from the repo. I think it will be difficult to handle in your case.

For stash operations, does Fork only access .git/logs/refs/stash or also other files? I see Fork is also accessing objects (pack files) after reading the former. If Fork only reads .git/logs/refs/stash, implementing support for it in forkgit might still be feasible.

DanPristupov commented 1 year ago

@brechtm Fork also reads (using mmap) the objects (both .pack and .idx files) in the .git/object folder.

brechtm commented 1 year ago

@brechtm Fork also reads (using mmap) the objects (both .pack and .idx files) in the .git/object folder.

I'm sure there are good reasons to do this, but I'm wondering what they are. Do you plan removing the git dependency completely eventually?

You can imagine I'm a bit sad that these kinds of changes will eventually make it impossible to use Fork remotely. Losing the ability to inspect/apply stashes is already enough of a handicap for me to roll back to 2.31.2. Of course, I realize that using Fork on remote checkouts is a very niche use case, so I understand it's not a priority to support it in any way.

jonasflint commented 1 year ago

I love fork, but my development environment is a separate linux box that I shell into via vs code. Would love to be able to use fork on my all of my repo's remotely....

samskiter commented 10 months ago

Thanks. Can you give a hint as to what changed? That can be a real time-saver 😄

Now Fork reads stashes directly from the repo. I think it will be difficult to handle in your case.

For stash operations, does Fork only access .git/logs/refs/stash or also other files? I see Fork is also accessing objects (pack files) after reading the former. If Fork only reads .git/logs/refs/stash, implementing support for it in forkgit might still be feasible.

Just wanted to say a big thanks to brechtm for creating forkgit - it's really made a bunch of situations easier for me....

I still think there's a place for trying to get forkgit or Fork itself to with VSCode repos so forkgit can piggy back of the connection VSCode makes...

I have found the limitations of forkgit quite quickly though with my Raspberry Pi development environmnent - e.g. I don't seem to be able to browse changes in the tree. I presume this is related to the conversation between @brechtm and @DanPristupov above. If it helps, I'm happy to contribute (a little) time or sponsor the project to get the updates in place to get forkgit working seamlessly again with remote repos. @DanPristupov VSCode has made remote dev so much easier these days but I still have a strong preference for good GUI-based tools for coding and managing SCM...

samskiter commented 10 months ago

Do we know which files fork needs access to to mmap over and why yet?

I don't really know too much about mmap but could sshfsexec be tweaked to allow mmap to occur over a remote. If so then all that would be needed is for fork to allow us to configure the 'mmap' binary (as we can the git binary)....

fork-dev / Tracker

Feature: Add non-local repository #929