search

markuskont / go-sigma-rule-engine

Golang library that implements a sigma log rule parser and match engine.
Apache License 2.0
92 stars 17 forks source link

Reorganize project layout #7

Closed markuskont closed 2 years ago

markuskont commented 2 years ago

See: https://github.com/markuskont/go-sigma-rule-engine/issues/6#issuecomment-1078766502

Currently sigma engine project is organized like a CLI application. This was mostly just a ad'hoc choice, as I was more focused on researching the topic at the time, I needed some command line helpers for benchmarking and testing, and the tool which actually used this library was built with cobra commands anyway. So it was a quick and convenient choice. Initial codebase was also fragmented between different pkg packages which I ended up merging into a single package.

However, this project was always intended to be a importable library, not a tool. Thus, go-project-layout does not really make sense and is simply a artifact of original exploratory coding research phase.

I'd propose reorganizing pkg/sigma/v2 into root of this project, with simple testing tools and example apps in exaples or tools sub folder. I don't think v1 serves any point whatsoever any more and github history already serves as archive for that.

However, this would be a breaking change for anyone that already imports the project, and update is not really critical. Opening this issue to distill thoughts on the topic.

newodahs commented 2 years ago

I agree with the proposed restructure and I think if there is a time to do this, it's probably now (as painful as a breaking change may be today, it'll be worse the longer it goes on).

Or maybe we spin /pkg/sigma/v2 out into its own repo and use the //Deprecated functionality in go.mod (https://go.dev/ref/mod#go-mod-file-module-deprecation) and hope the people importing this library are paying attention?

Basically, commit a deprecation notice now, something like: // Deprecated: pkg/sigma/v2 library has moved to markuskont/go-sigma-rule-engine-v2/

Cut the pkg/sigma/v2 library over to the new repo in the deprecation message; leave what is here in place for now but don't make any more updates to it (all new updates go to the new repo) and at some point in the future kill/archive whats here and call it done?

I'm mostly spitballing here?

markuskont commented 2 years ago

To be honest, I think I'd rather break the import than migrate the project. I think this library has gained some traction, but I guess the actual userbase is still small. I kinda consider project deprecation as worse breakage, as it's silent. I mean, that's why we love Go, right. When API-s are changed, it's pretty easy to fix up your imports as compiler will tell you what happened and what needs to be fixed.

So as long as the last version with old structure has a major version tag, I don't think it's such a big deal. That's why we have versioning. Especially if the change is clearly documented in README as well (which I should update anyway).

I should have some time to work on it the week after the next one. All of my time has currently gone into other projects.

newodahs commented 2 years ago

So the plan here will be to:

  1. Tag where we are (0.2.4?)
  2. Restructure cmd to examples (or tools, I like examples I think)
    • main.go moves to a where the current cmd ends up
  3. Move the root of pkg/sigma/v2 to the root of the project (that is /pkg/sigma/v2 => root of the repo)
  4. Dump everything else that isn't documentation (basically, v1)

Should we consider any further restructuring of what is currently pkg/sigma/v2? Should things like helpers.go end up in a util?

markuskont commented 2 years ago
  1. yep
  2. I also prefer examples, doubt there would be need for main.go at all
  3. yep
  4. yes, I kept v1 as reference while coding v2 but I dont see any need for it any longer. Git has a history anyway
  5. perhaps, maybe stuff like reference data models can also go to utils, or maybe even do something like models. But not at first I think, would rather avoid needlessly splitting the package
markuskont commented 2 years ago

I pushed the 0.2.4 tag and started work on reorganization in https://github.com/markuskont/go-sigma-rule-engine/tree/next-0.3-reorg-2022-04

Still WIP, cmd section is still there and I want to rewrite the commands to just use Go flag package, so I could ditch the need for cobra and viper.

markuskont commented 2 years ago

In fact, regarding the code in helpers.go, I'm considering making a separate project for that. Since I kinda need that logic all the time and ended up just implementing a way more advanced (and faster) version of that in another project (https://github.com/markuskont/pikksilm/blob/staging/pkg/models/models.go#L16).

markuskont commented 2 years ago

Done, closing this thread.