Feature Request - Git Site Source

[aormsby]

Okay, hear me out. I think your project has a ton of potential as a Hugo CMS, and I *love that it's a desktop app. (It's the only one I've seen!) I got so excited about it that I thought about a lot of improvements to be made, but there's one huge feature that I think could take Hokus from being a pure developer tool and turn it into something that people might consider giving to non-technical users.

Automatic Git Repo Clone!~

Here's how it could work:

1. A user installs Hokus.
2. Hokus asks for a git repo URL. The user enters it.
3. Hokus automatically clones the repo locally and updates some project settings based on a config file in the repo.
4. Nothing else. Hokus can already start a live preview, edit, and publish as needed!

(By the way, it's amazing to me that no one else has taken advantage of Hugo's local live previews with a CMS like you have.)

I know, this kind of feature comes with baggage - repo permissions, perhaps user accounts, a login system of some kind. It's a lot. But right now with the Hokus setup using local files, a developer is required for configuration and use. Imagine if I could give Hokus to a non-tech client of mine purely for content editing and have them set it up all by themselves - Hokus could be the dream desktop CMS we've always wanted!

Let me know what you think! I'd be thrilled to brainstorm with you about it. 😄

[julianoappelklein]

Hello @aormsby - your feature request makes total sense to me.

What we need is a Git Source. Now we have the Folder Source - you point to a folder and you are ready to go (not quit, because you still need to configure some things). What you want is to point to a git repository.

And we would also need a way to push/pull data. We already have the concept of multiple workspace (one site, multiple versions) - in the Git Source case, each branch can be a workspace.

This makes sense to you? What do you think?

[aormsby]

That sounds right, yes!

Forgive me, I'm not super familiar with how Hokus is built. If I get extra time, I may learn more about it. What kind of work would a feature like this involve?

[julianoappelklein]

@aormsby - Hokus is built using Electron and React. This is not a simple feature. It's necessary to have experience in both frameworks. And for this particular case, you must have experience with Git.

Let me know if that is the case.

If this is not the case, don't worry because this feature is in our roadmap. Once it is ready you can help us to validate it.

[aormsby]

I have plenty of git experience, but it's the other two frameworks that I've never touched. 😅 I may try to spend time learning at some point. At the very least, I'm happy to help with testing and validation.

[julianoappelklein]

If you have the time, you can do more! We can use your git experience, for sure :)

I'm not sure yet of how would be the workflow - you can help me with this. The idea is to abstract git operations, but to do this we need some planning. For example, when we should commit, push, pull?

Maybe we can make a commit every "Save" button and create a way to tell the editor if there is content to sync (to send to remote or to fetch from). We would need to know what are the git commands we can use to achieve that.

Maybe you have experience with Netlify and we can use your experience of how they do things. But keep in mind that for now, I'm seeking a very simple workflow and we can think of something more elaborated later.

[aormsby]

Actually, I'd love to help plan that out! After playing around with forestry.io a bit, I found that it was making an insanely high number of git commits whenever any small thing was changed. It really seemed to me that you could have a much cleaner, simpler git workflow.

As you mentioned, I'd say fetch/pull, commit, and push are definitely the core commands to consider, but it grows quickly from there. If someone is working with a team to write site content, a push may have to happen with every commit in order to avoid complicating the workflow for users. You could separate the push by having some kind of 'sync' button, but I'm not sure of the benefits of doing that. Anyway yeah, things get hairy when you pull and have to start doing merges and resolving conflicts.

I guess I could start thinking with a 'single user site' in mind for simplicity's sake, but we'd definitely want to expand the workflow to be good for teams. When I find some extra time I will definitely draw up some ideas for you to take a look at. Sounds fun!

[julianoappelklein]

Actually, I'd love to help plan that out! After playing around with forestry.io a bit, I found that it was making an insanely high number of git commits whenever any small thing was changed. It really seemed to me that you could have a much cleaner, simpler git workflow.

Is this kind of experience that we need.

I agree with you. We don't need one workflow that handle all cases. We can have different approaches. It's just a matter of setup.

I image to have friendly workflows names, like "Single Editor Workflow" and "Team Workflow".

The Team Workflow probably would not need a sync button, because we would need to sync everything automatically as soon as possible. But the sync button would make sense for the Single Editor Workflow.

I see we are very well aligned. For now, the Single Editor Workflow is what we need. I hope you can find some time to come up with with a nice workflow!

[julianoappelklein]

@aormsby - any update on this? I'm looking forward to implement this feature. Any new insight would be helpful. Thank you!

[aormsby]

Hi! Sorry, I've been dealing with visa and travel issues, which is not the easiest thing right now. I haven't forgotten. I just haven't had a chance quite yet.

[julianoappelklein]

Hi! Sorry, I've been dealing with visa and travel issues, which is not the easiest thing right now.

No worries! These are tough times. I hope you can solve it soon!

[aormsby]

A quick update - I'm about halfway through writing up a proposal. Unfortunately, I hit a rather nasty issue that I have to work out. Hopefully, I can show you my work soon. 😃

The underlying question here is if we should pull/push the entire repo or if we should focus on specific directories. For push, we actually have to push everything in the repo, even if Hokus doesn't manage it (like config, source code, themes, etc.). That's just how push works. Basically, I'm trying to work out if that's acceptable behavior and then figure out how best to manage it.

[aormsby]

Crap. No, I have to spend more time considering a few scenarios. I've been trying to avoid any git merge situations, but it seems they're very easy to run into. Perhaps I can work out a method for dealing with them in a very opinionated way. I'll think about it more. 🤯

[aormsby]

Proposed Git Flow

Okay! I've split up the git command sets we might need into 4 main processes

1. Commit local content changes
2. Fetch / Compare local repo with remote repo
3. Pull new commits from remote repo
4. Push local commits to remote repo

These can be initiated on their own or with the Git Sync action I'm proposing below.

An additional state we would need to add to Hokus would check if there are any unmerged files. I'll call this the Merging State.

Some assumptions I've made:

1. All actions are performed on a repo's master branch.
2. There is only one remote connection for any repo.
3. Users are not using git submodules in their repos. (This might not matter.)

Please remeber that this is all for a single-user workflow. It's a start!

Overview

Before I get in to the deeper details of each process, here's a general overview of when and how each one would occur during the Hokus application lifecycle.

On Hokus Application Start

• Hokus automatically runs a Fetch on all git-connected repos and then Compares the local and remote revision hashes to see if either are out of date or if they have divergent commits.
• Based on the comparison, Hokus automatically Pulls data from the remote repo and updates the user's content locally. If all the steps are successful, user data is automatically updated on Hokus start. :smile:
  • The pull fails when there are uncommitted local changes in the repo that cause conflicts.
  • The pull also fails when local commits and remote commits have confilcts. Unfortunately, git will automatically try to merge things. If this case happens, we just run a git command to abort the merge.
  • All failures from the auto-pull need to be fixed by running a Git Sync. Dealing with conflicts is an involved process, and a Git Sync is more detailed than an auto-pull on start. There should be some kind of visual notification to users that they should run a Git Sync before continuing. (Or perhaps block any further work until this happens.)
• There should probably be an option to disable this action if users don't want it, but I really think it's helpful overall.

On File Save / Exit

• Git Commits occur when a user clicks the :white_check_mark: button to save their changes in a post. (This action already saves their work, so its great time to also make a commit.)
  • The commit message includes the filename and a simple timestamp. These commits could possibly be used to show post revision history later on.
• A Commit also happens automatically when the user closes an edited file in Hokus. This prevents untracked changes from sticking around and causing problems.
  • Possibly, the user may see a dialog asking to save or discard their changes.

Note: These Commits will only track the edited file in question. We don't want to commit anything else in the repo that isn't being edited. However, if other files are staged, they will get committed along with the post. I will need to try and find a way around that. :sweat:

On Hokus Application Quit

• Hokus automatically Pushes any local commits to the remote repo on application quit as a final action. This is to make sure that users don't make a bunch of changes, forget to sync them manually, and then start work on another computer without access to that work. It also helps to avoid git conflicts later on.
  • There should probably be an option to disable this action if users don't want it, but I really think it's helpful overall.

On Git Sync Action (new)

• If users click the Git Sync button, a combination process begins that pull remote data and then sends local changes to the remote. This button lives on the Site Management page next to the Publish button.
• A simple, successful Git Sync looks like this:
  • Fetch/Compare
  • Pull
  • Push
  • :tada:
• A lot can go wrong during the Pull portion of a Git Sync, but the issues we'll deal with can be boiled down to these:
  • Uncommitted local changes prevent a pull - If the user doesn't cancel the process, we deal with this by stashing the changes, pulling, and then re-applying the changes on top of the pulled data. We abort and re-apply the changes at any point of failure. The user may also decide to discard the local working changes and pull anyway.
  • Divergent commits on both local and remote cause merge conflicts - We deal with these either by aborting the merge entirely (up to users to fix outside Hokus) OR we allow the conflicts, put Hokus into a Merging State, and have users manually merge their files before doing another Git Sync. (explained below)
• If a Git Sync is not successful at any time, we either abort the whole process and reset or put Hokus into the Merging State. We decide all this based on user options in a dialog box.

The Merging State

• During the Pull segment of a Git Sync, users might decide to go ahead with merging conflicts in their files. To complete the merge, all files with conflicts must be manually edited and then staged for commit. Otherwise, we'll have 'unmerged' files sitting around, and the merge will be considered ongoing, which prevents the git actions we want to take.
• Upon entering this state, we mark all conflict files with some sort of visual cue to let the user know they need to make edits.
• When they open the file, they'll see the git conflict data inside their post text. It'll look just like a normal file diff displayed as text. (Additionally, we could mark the lines with some highlight color.)
• As users open the files, make their edits, and close them, we can automatically stage the files (not commit them!) with their merges.
  • It's basically the normal workflow, but there's one key difference: Instead of committing the files on save / exit, we should simply stage them for a commit so that users can choose to abort the merge at any time. (We give them another button for this.) This would unstage the files and reset to before the merge. So... during the merging state, the commit files action should stage files instead. The action timing is the same, though - on file save / exit.
• While there are still unstaged merge files, all git actions should be blocked. (Perhaps other Hokus actions should also be blocked.) When all merge files have been staged with their fixes, then users are prompted to run a Git Sync again to Commit the merged files, finish the process, and Push.

Note: While this all sounds complicated, it actually shouldn't happen to very many content files at once for single users. I don't expect them to be simultaneously editing a ton of files that all conflict with each other. Anyway, we'll handle it in the simplest way possible.

Some Git Gotchas

• We cannot limit a git Push to only Hokus-managed content. Therefore, all commits of any files will be pushed to the remote when we push. This isn't terrible, but it's maybe good to let users know. (We could potentially deal with this through branching in a future update. I'm unsure.)
• We could limit the Pull action to Hokus content, but we shouldn't. IF we did that, users could have some up-to-date content and some out-of-date content in their local repo, which is very confusing. Pulling the whole repo means that some merge conflicts may be in non-Holus files, but that's preferable to mismatched git revisions.
  • We can notify users which files will be in conflict in a dialog message, but they would still have to take care of the conflicts themselves, We will still need to keep track of them to some extent during the merging state.
• I'm currently not offering the user a merge where they can choose to overwrite with 'ours' or 'theirs'. This can be added later if I can figure out a decent way to handle it without losing commit history. I'll have to run some tests.
• I have only considered text files in this proposal, so dealing with conflicts in media files (images, audio, etc.) may need further work.

More in Next Post

This post is actually getting rather long, so before I write out each git process step-by-step with the full commands from my notes, maybe it's better to see if all of this makes sense first. I'm hoping it's clear enough to understand, but there's a lot going on here. At this point, let's just have a chat about what's here and see where we are. I'm happy to answer any questions if things don't make sense. This is so exciting! :slightly_smiling_face:

[julianoappelklein]

Wow! Great work. I'll read it carefully! :)

[Apollinaire]

I love the feature, and I think it can propell Hokus to the top in terms of CMS. However, for this to really be a killer feature, we should have the possibility to create a packaged app that is "locked" on a git repository. Basically, what I want is an app that I can just send to my client for him to edit the site. All he has to do is open the app, and he sees his site. No git url or authentication needed. That is the advantage we can have because this is an electron app, not a browser site like forestry. On the plus side, I think you could monetize the app building, distribution and upgrading. As a client of your service, I would setup the app on your site, including an access to a git service like Github, and it would build it all for me. Then I get a download link to send to my clients. When I change something in my setup, every other app automatically updates.