monorepo / concurrent foreach / repo processing pipelines

[roscopecoltran]

Hi guys,

Hope you are all well !

I just have 3 quick questions:

1. Monorepo, scalable vcs filesystem What are the missing components, in your views, to make , from go-git.v4 and go-billy.v4, a mono repository management system ? a distributed filesystem (eg seaweedfs) ? a full text-index of the source code (eg. zoekt) ?

Ref:

• https://github.com/Microsoft/VFSForGit
• https://blogs.msdn.microsoft.com/vsappcenter/how-mono-repo-and-one-infra-help-us-deliver-a-better-developer-experience/

2. Concurrent/Thread-safe iterator over worktree

So my 2 other questions are connected to the first one, as I would like to pre/post process source code of all my repos (>300 repos) with hookable plugins. It would include queues, workers and plugins triggered by filename, path or content matching (eg. gitleaks). It sounds pretty much like all your other awesome projects like borges or hercules. :-)

The pre-requisite is to have a faster and concurrent filepath Walker with a callback registry.

Also, I noticed that memfs from go-billy.v4 is not thread-safe. I will push a code example but just try to run your go script with the -race ... arg and you will see lots of data race if you use memfs. As I want to gain performances, and only index/process the content of some starred or owned repositories, that's why I focus on memory based filesystem.

So my question is, where would it be better to add this concurrent dir walker in go-git.v4 or go-billy.v4 ? In a new go-billy's plugin or the commit_worktree.go files ? The same as if I would like to add a fsnotify/filesystem watcher plugin on repo opened with go-git.v4 ?

Can you share any recommendations if I would like to do it right in a forked version of packages ?

Examples:

• https://github.com/sniperkit/snk.golang.dirwalk/blob/sniperkit/_examples/cfiles-processor/main.go#L15
• https://github.com/sniperkit/snk.golang.dirwalk/blob/sniperkit/_examples/cwalk-traversaltime/main.go#L18
• https://github.com/sniperkit/snk.golang.dirwalk/blob/sniperkit/_examples/cfiles-processor/main.go#L15
• https://github.com/catmullet/Easy-GoLang-Parallel-ForEach/blob/master/Parallel/Parallel/Parallel.go

3. Node-based processing pipeline I would like to create a system with builtin-plugins or shared plugins and to trigger a pipeline of tasks based on some text based rules while iterating files from a repository opened/cloned/forked with go-git.v4.

An example of UI:

Nb: it could be managed by an fbp based system like noflow or nodered. The video above is really for visualizing the task flow management interface for each specific scenarios.

Examples of post/pre processing tools:

• Fast and furious :-) language detection, https://github.com/src-d/enry
• Fast and curious :-) trigram code indexing, https://github.com/google/zoekt
• Fast and powerful :-) alternative to grep, https://github.com/svent/sift
• Fast event triggered pipelines, some related examples could be:
  • Package manager detection and build manifest parsing:
    • Filetype registry: https://github.com/codexray/cxdig/blob/master/config/filetypes.go#L20-L431
    • Pip: https://github.com/mikejewell/diligent/blob/python/pip/pip.go
    • Govendor/Glide/Dep: https://github.com/mikejewell/diligent/blob/python/govendor/govendor.go
    • Npm/Yarn: https://github.com/mikejewell/diligent/blob/python/npm/npm.go
• Pipeline example:
  • find and parse all projects//hunter.cmake files, after after updating the fork from the original repo, github.com/ruslo/hunter**, to keep up with their latest c++ project and versions and index all new version attributes (version, sha, url, ...) into a bleve search index.
    • https://github.com/ruslo/hunter/blob/master/cmake/projects/caffe/hunter.cmake#L12-L58

So my last questions are:

• Have you already planned to create such pipeline/dataflow manager as you are processing lots of repositories ?
• Should I created a plugin/built-ins registry in go-billy ?

Thanks for any insights about the questions above ^^.

Thanks for your time guys.

Cheers, Rosco

[smola]

Hi @roscopecoltran!

You are welcome to join #go-git channel on our Slack to discuss any of these issues.

I'll try to answer some:

What are the missing components, in your views, to make , from go-git.v4 and go-billy.v4, a mono repository management system? a distributed filesystem (eg seaweedfs)? a full text-index of the source code (eg. zoekt)?

I have no idea. We didn't think about building a mono repository management system. I have no good understanding of problems to be solved in that space. But if you have more concrete questions, maybe we had similar use cases.

The pre-requisite is to have a faster and concurrent filepath Walker with a callback registry.

Could you elaborate a bit more?

I would like to create a system with builtin-plugins or shared plugins and to trigger a pipeline of tasks based on some text based rules while iterating files from a repository opened/cloned/forked with go-git.v4.

Sounds great. Although that falls completely outside of the scope of go-git. We are happy to support you with whatever can be done in go-git to support this use case.

Have you already planned to create such pipeline/dataflow manager as you are processing lots of repositories?

We have currently no plans to do such thing.

Should I created a plugin/built-ins registry in go-billy?

I don't know what plugins would do in go-billy, since it is not a framework. Implementations are already pluggable. But feel free to share more details about the use case and we can consider it.