es-git: js-git 2.0

[mariusGundersen]

js-git is a fantastic project! Having played with the source for a little bit I think it's really well done: it's clearly laid out and it's easy to understand the general architecture. But, it's grown a bit quiet around here, and in the mean time the world of JavaScript has moved forward, leaving the source code of js-git behind.

We now have ES2015, ES2016, ES2017, promises, fetch and many other great features. We also have a lot of new tools, like webpack and typescript, which help with developing and using libraries.

I want to use js-git, but I also want to write it in a modern dialect of JavaScript using the tools available, so I have started a general rewrite of js-git (into es-git, as in EcmaScript-git). This is based on discussion earlier about using promises and async/await, but when I started that I saw that there were several other small things to improve.

I'm writing es-git in TypeScript, since it supports pretty much all the features of ES2017 (and some proposed features), but I will compile it both to ES2017 and to ES5, so that consumers of the package can use it based on their preference. I am removing some legacy support though, for example WebSQL and the fallback deferral code, and replacing them with more modern (and well supported) features, like IndexedDB and promises.

While I'm rewriting this to get it up to a modern standard, it's not just about new and shiny all the things. Async/await lets us write the same code in much fewer lines, making it clearer what is going on. TypeScript is great for libraries, as IDEs can now give very good feedback, and it's great for maintainers as refactoring becomes easier.

Luckily for me the code in js-git is pretty straight forward to convert. I'm not sure when I will have a working version, or how much I will have to leave behind, but when it's gotten to a usable point i will probably publish it as a separate nuget package, as a fork of js-git, since there will be a lot of breaking changes. That is, unless you are interested in merging it back into js-git.

Anyways, I mention this because I really like this project, and I'm sad that it's so quiet around here. I really need git in JavaScript in the browser, and the small experiments I've done show that it is working well enough for me. So I want to help out 😃

[CMCDragonkai]

Perhaps there can be some improvements lifted from: https://github.com/SamyPesse/gitkit-js

[mariusGundersen]

neat, didn't know of that one. I will have a look and compare them :)

[CMCDragonkai]

Just having read through the issues here, and also played around with the js-git code, there's alot of functionality missing from js-git (like init not creating the refs and objects directory, and git log not working because refs directory is missing heads/master file). I'm going to be exploring gitkit and see how it performs. (It's also more recent, with most commits 1 year old).

[strangesast]

Nice work @mariusGundersen! I'd like to point out that I'm working on a similar project here

Much of our code is very similar, but differences I'm aware of include my addition of karma testing and a removal of creationix's bodec, culvert, "run", etc in favor of now-available TextDecoder/Encoder and typed array operations.

I'd be willing to concert our efforts if that's something you're interested in.

[CMCDragonkai]

I've also decided to start a fork from gitkit instead of js-git. Gitkit is also missing some things. In pursuit of this I hard forked webpack's memory-fs to create virtualfs which supports symlinks simulated stat data and soon hardlinks.

On 15 Jun 2017 23:17, "Sam Zagrobelny" notifications@github.com wrote:

Nice work @mariusGundersen https://github.com/mariusgundersen! I'd like to point out that I'm working on a similar project here http:///strangesast/js-git/tree/working

Much of our code is very similar, but differences I'm aware of include my addition of karma testing and a removal of dependencies on creationix's bodec, culvert, "run", etc with now-available TextDecoder/Encoder and typed array operations.

I'd be willing to concert our efforts if that's something you're interested in.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/creationix/js-git/issues/132#issuecomment-308726483, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnHHW1IwLgMHI3GvqediwehUtWvX4Zvks5sES8EgaJpZM4NwLwG .

[CMCDragonkai]

https://github.com/MatrixAI/VirtualFS

On 16 Jun 2017 02:59, "Roger Qiu" roger.qiu@polycademy.com wrote:

I've also decided to start a fork from gitkit instead of js-git. Gitkit is also missing some things. In pursuit of this I hard forked webpack's memory-fs to create virtualfs which supports symlinks simulated stat data and soon hardlinks.

On 15 Jun 2017 23:17, "Sam Zagrobelny" notifications@github.com wrote:

Nice work @mariusGundersen https://github.com/mariusgundersen! I'd like to point out that I'm working on a similar project here http:///strangesast/js-git/tree/working

Much of our code is very similar, but differences I'm aware of include my addition of karma testing and a removal of dependencies on creationix's bodec, culvert, "run", etc with now-available TextDecoder/Encoder and typed array operations.

I'd be willing to concert our efforts if that's something you're interested in.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/creationix/js-git/issues/132#issuecomment-308726483, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnHHW1IwLgMHI3GvqediwehUtWvX4Zvks5sES8EgaJpZM4NwLwG .

[mariusGundersen]

I'm guessing we have slightly different needs. I'm more interested in the low level storage, while you seem to be more interested in the high-level features and using it with a file-system. The best would be if gitkit could use the lowlevel stuff from js-git, instead of having to reimplement it all by itself.

[mariusGundersen]

@strangesast cool, I haven't looked into those yet. I've just tried to get it to compile as a TypeScript project. I'm going to focus on a subset of it for now, since getting everything to work is a bit too much for me and the time I have available at the moment. Probably I will publish it as several npm packages that can be combined, instead of one large package.

[creationix]

Good luck fellows!

[creationix]

And sorry I never had the time to finish this project. Gotta pay the bills and I couldn't get enough funding to finish this.

[mariusGundersen]

Small progress update: I've started working on this more now, and have gotten quite far. I have most of the mixins converted already, and will start to look at the network and protocol code next. I've decided to make it as multiple packages that will be published independently instead of having one package with everything. Nothing is published yet, as I'm still working out the most difficult part: naming.

The forked repo is here: https://github.com/mariusGundersen/es-git/tree/master/packages

[mariusGundersen]

Ok, so I've got most of the features done, including basic fetch and push using https. This means I can now start using the project for something useful. I've created both a github and npm organization, to make it easier for others to have write-access to it, if you want to.

The repository is now here: https://github.com/es-git/es-git

It's currently quite limited in documentation, but I hope to add at least a basic readme file to each package.

[billiegoose]

Gosh darnit @mariusGundersen. I guess I'll need a new name for my pile of files... https://github.com/wmhilton/esgit 😆

Where were all you forkers a year ago when I started on this? ^ the above repo being my 2nd or 3rd rewrite (see predecesor from Feb) Thank god this idea is taking off! It's a freaking brilliant idea. Think we can make "git in the browser" a standard feature of every webapp by 2020?

@CMCDragonkai I am using BrowserFS to handle git in the browser. It is a very nice, feature complete implementation of the Node fs api. I believe it has support for simulated symlinks and fs.stat calls, depending on the backend. The only thing it doesn't have is fs.watch and I'm already working on an implementation of that.

[CMCDragonkai]

@wmhilton Cool.

I'm actually working on js-virtualgit https://github.com/MatrixAI/js-virtualgit which is a port of libgit2 to JS using emscripten and monkeypatching its FS to point to our recently developed js-virtualfs: https://github.com/MatrixAI/js-virtualfs

Our virtualfs also completely supports symlinks, hardlinks, file descriptors, streams (tested that the stream has similar if not the same asynchronous behaviour as the node fs streams), and slated to add character devices (/dev/null, /dev/zero), fs singleton, copyfile, lseek, mknode, URL and Uint8Array, emscripten library support.. this is all in a separate branch emscripten.

We didn't see BrowserFS before until we started on js-virtualfs. But I think we have a slightly cleaner implementation. One downside, no browser support yet because of rollup issues.

I think there's room for a JS implementation of Git, and an official port of libgit2 to JS.

[billiegoose]

We didn't see BrowserFS before until we started on js-virtualfs. But I think we have a slightly cleaner implementation. One downside, no browser support yet because of rollup issues.

Story of my life! Something should make a service that scans Github and autodetects similar projects. FYI BrowserFS has fairly good emscripten support. I don't think it has virtual character devices, but it is fairly extensible. Once you get past the TypeScript (ugh) it's not hard to work in. I recently upgraded the XHR backend to use window. fetch, and am planning to add a window.caches backend that should support much larger files than local storage or indexed db backends. There's a key/value store interface that makes it possible to implement a new fs backend on top of any key value store interface, but since window.caches supports streaming, the author may want to take advantage of that and do something more ambitious than treating it as a kv store.

Good luck getting libgit2 to run in emscripten! I absolutely agree, there is room for many many implementations. I'm focusing on a file system based implementation rather than using an in memory or indexed db approach, and focusing on replicating the file system behavior of the git CLI so I can use it interchangeably to manipulate repos on my desktop.

[CMCDragonkai]

@wmhilton The browserfs emscripten implementation doesn't replace the rootfs of emscripten. Because emscripten doesn't have proper rootfs remounting capability. I found a way to do this though, and have to refactor the virtualfs a bit a support some extra stuff (that I mentioned above) to make this monkeypatching work.

I'm also focused on a filesystem focus of git, because I need the concept of a "working" directory and staging changes, thus the virtualfs requirement and needing the monkeypatch the emscripten rootfs mounting. It's also because libgit2 isn't going to be the only module interacting with the fs, I intend to get the other libraries like node tar and monkeypatch their fs reference such that I can archive the in-memory fs implementation.

Other than that, the actual translation of libgit2 to JS is not difficult and is already working. https://github.com/kripken/emscripten/issues/4902

[mariusGundersen]

So es-git is written to be a low-level git, for when you need the most basic git functionality. But it's designed to be easily extended, so you are free to build other stuff on top of it.

I have put a few examples online: https://es-git-examples.mariusgundersen.net/

Currently I'm using es-git in my side-project, which is also the main driver for implementing new features: https://github.com/mariusGundersen/Ekkiog

[billiegoose]

For anyone following this thread, I'm nearing a 1.0 release for my module! I named it isomorphic-git. I've tried to distinguish it from js-git and es-git (and gitkit, and js-virtual-git) by creating a high-level API that's more like the git CLI, and focusing on interoperability with the .git directory structure so it works with repos on the desktop too. The commands still left to implement before 1.0 are git log and git merge, but the rest (clone, init, fetch, push, add, remove, commit, status, checkout) are all there: https://github.com/wmhilton/isomorphic-git