Pkg Monorepo potential for misuse

daemonl commented 4 years ago

A few reasons as to why we would use 'pkg', to make sure they are captured:

Approval for working on open source projects may be easier it it's a smaller number of repos
go imports use the last part of the path as the import name as best practice, so either we would need a package called 'log' in the top level of the repo, which is greedy, or 'go-log', which misses the best practice.

A common method to address the second one is to use an alias server, e.g., gopkg.anz.com/log is a static website with a go get redirect to github.com/anz-bank/log.go or something similar.

The potential downside for using a monorepo for packages is that we end up creating a standard set of libraries which are interdependent and need to be versioned together (as two separate issues possibly?). E.g., if we wrote a 'httpLib' here which logged, it would probably use the pkg/log library. We might add a 'trace' package which the logger and httpLib know to check for certain IDs, then an errors package which httpLib knows to inspect for 'http specific' errors, etc etc. That pattern allows you to create a whole 'world' in which we work, a whole new basis for writing go code, and effectively creates our own highly opinionated ecosystem of go, independent from the wider go community.

I see this as a 'bad thing', but maybe it's not, and maybe that's exactly what we want to create, so raising it here as a ticket so that it's explicitly discussed and decided, rather than accidentally done to solve unrelated problems with go packaging and ANZ policy.

juliaogris commented 4 years ago

@daemonl , discussed with @camh-anz - we don't really understand your issue:

import "github.com/anz-bank/pkg/log"

works just fine imo and you are not likely to use several different log packages together i'd say?

wrt versioning that's also not a real concern imo as the go.mod is the repo-root. when you upgrade you might jump a few versions (if we get around to auto-tagging), but that's no concern imo?

camscale commented 4 years ago

I see this as a 'bad thing'

Please refrain from such judgements without any supporting arguments. There is no way to respond to this except to just dismiss it. If you have specific issues, please describe them so we can consider ways to address them.

daemonl commented 4 years ago

My description deliberately uses open language to trigger a discussion, including (but not limited to) directly asserting that my opinion may be incorrect.

I can't prove a mono-repo for libraries is a bad thing, It's an opinion.

camscale commented 4 years ago

Discussion is fine - proof is not necessary. So let's discuss. Complete this sentence: I see this as a 'bad thing' because ...

daemonl commented 4 years ago

I see mono-library-repo as a bad thing because it leads us down the path of 'anz standard packages'.

The language ecosystem our engineers work in I see as multiple layers of libraries and functionality:

The Go language itself
The standard library
X Libraries (practically standard)
Open source community
ANZ libraries
Fabric libraries
Squad libraries
Service itself

As you get further down

The effort given to maintain quality decreases (i.e., the libraries aren't as good)
The likelihood of a new developer knowing the library decreases
Rigidity increases (for better or worse)
Individuality / snow flake services is reduced
It becomes quicker to ramp up a new service

To avoid and work around the downsides, I suggest that packages written to solve generic tech problems, but without an ANZ specific business problem in mind should be pushed up the layers as far as possible. Most will end up in the Open Source Community layer, as that has the lowest barrier to entry. Packages which solve a specific business problem should be pushed down as far as possible, to the service itself.

The logging library has been built as open source, so it's pushed up the stack, which is awesome.

Calling the repository 'pkg', however, implies that there will be other packages other than 'log' in here in the future. What that does is simply move the ANZ libraries eco-system up to be open source, it doesn't remove the layer.

The issue comes when you end up with, say, a pkg/log, pkg/trace and pkg/http. The http package will rely on log and trace to do header inspection, write logs etc etc. This is really handy to use, but means that using one of these brings all of them, it's no longer optional. Once it's not optional, you are basically creating a framework which developers have to work inside (and work around). (This may be desired for the log/trace/http triplet if the point of the http library is tracing and logging, but then the repo should be called visibility or something more specific)

This is not a pure black and white technical decision, it's more about the mindset when developing. I believe that promoting the log library the top level to make it its own repo will aid having the mindset of 'independent' libraries, and independent versioning.

Another approach which may make this easy is to create an anz-go 'org', and put 'log' in the root of that.

juliaogris commented 4 years ago

@daemonl - the problem here is that we don't understand what you are trying to communicate.

Approval for working on open source projects may be easier it it's a smaller number of repos.

This repo is already approved for independent, generic go tooling packages - how would getting approval for a new repo be easier? I have worked on the ANZ open sourcing process for over a year - getting approval to start a new repo is not an easy thing. Please let me know if I have misunderstood.

go imports use the last part of the path as the import name as best practice, so either we would need a package called 'log' in the top level of the repo, which is greedy, or 'go-log', which misses the best practice.

This repo is, as the README.md says, a places fo go packages so there is no need for a go- prefix.

log as top level directory is perfectly fine for that purpose imo. Can you please state your concerns here? Do you want to start a competing go logger for ANZ rather than modify the exisiting one or using one of the many exisiting ones?

we end up creating a standard set of libraries which are interdependent and need to be versioned together

There is one go package version per go.mod file (that's how go mod works). There is one go.mod file in this repo, so there is one version for all its libs. What you are saying does not make sense to me - please help me if I have missed something.

Aside: There is now auto-tagging in place (#41) with every merge to master getting a patch bump unless #minor or #major is part of the commit message.

effectively creates our own highly opinionated ecosystem of go, independent from the wider go community.

This concern is independent of repo layout:

mono-repo with multiple go modules
mono-repo with single go module
many independent repos each having a single go module

I don't see how the structure makes a difference to your argument: we could have lots of small repos that are still tightly coupled.

If you prefer to get approval for open source repos independently, that is good too.

What we have here is a pragmatic approach to get packages into the open source world.

daemonl commented 4 years ago

Ah, ok, I think I see where I have created confusion. There are arguments both for and against the pkg approach in my issue description.

I am more concerned with the development mindset than the actual technical implementation.

pkg makes sense IFF it's a work-around to the difficulty in creating new open source repositories at ANZ. But that would need to be well understood to maintain the mental separation of concerns.

Would it be possible to get a whole github org marked for OSS at ANZ?

juliaogris commented 4 years ago

Would it be possible to get a whole github org marked for OSS at ANZ ?

You can always try. I have run out of energy to push any such agenda.

https://github.com/anz-bank is this in some sense.

juliaogris commented 4 years ago

and yes from my perspective pkg is a workaround, but making this to clear might bring other difficulties.

from a usage perspective there isn't really a big difference imo opinion re wether the import path is:

"github.com/anz-bank/pkg/log" of "github.com/anz-go/log"

I personally would also prefer the second one - but again this comes with procedural challenges.

anzdaddy commented 4 years ago

I see mono-library-repo as a bad thing because it leads us down the path of 'anz standard packages'.

I see mono-library-repo as a good thing because it leads us down the path of 'anz standard packages'.

I do not think it's a good thing generally that every team gets to pick their own logging approach, error handling approach, config management approach, etc. I understand that there are cons to saying, "This is the way ANZ does config," but I think the pros vastly outweigh the cons, especially when contrasted with the each-to-their-own strategy. And, reading the majority of the post, @daemonl, I get the sense that this is your view as well:

The logging library has been built as open source, so it's pushed up the stack, which is awesome.

If this is mostly about repo structure, I am less concerned about that, and there are non-technical issues associated with refactoring the current structure. We can face into that later if need be, but I highly doubt it will come to that. A mono-repo has its challenges, but they are manageable.

daemonl commented 4 years ago

I think the only technical change to make, which may aid the mindset challenge, is to move the go.mod into log

daemonl commented 4 years ago

I think @anzdaddy that's probably closer to the point I was trying to raise for discussion. Not how do we do it technically (we have many good techs making many good decisions there), but what is the direction we want to head.

To be really clear, my opinion on this is that each squad should be totally free to implement their solutions using whatever technology and libraries they see fit with absolutely no limitations. In reality that doesn't fly at a big company with multiple squads (apparently... but I'm not arguing that here). I will concede many points, but I will always be arguing in the direction of developer freedom and individual choice.

juliaogris commented 4 years ago

Moving go.mod into log makes auto-tagging a lot harder - I'd prefer to wait for this until we actually need it.

juliaogris commented 4 years ago

@daemonl - if you wish to add a package with its own go.mod file and tagging strategy you can whilst leaving a top-level go.mod file (see docs)

From the same docs:

Should I have multiple modules in a single repository? Adding modules, removing modules, and versioning modules in such a configuration require considerable care and deliberation, so it is almost always easier and simpler to manage a single-module repository rather than multiple modules in an existing repository.

juliaogris commented 4 years ago

Are there any outstanding concerns now? Otherwise - especially if there isn't anything actionable here - I'd like to close this issue.

daemonl commented 4 years ago

I would like to see an issue created to track moving the go.mod to the log directory, and it should be completed before a second library is added to this repository.

The context for these issues is to track the discussion from a ticket (which I can no longer find) which was something along the lines of "Make fabric use the logging library" - We decided to create issues to track the things we would like changed before we were made to use something.

This captures a blocker to adoption: We need something (code, docs, split, whatever) to ensure that this is 'just a logging library', and is not step one of a 'anzx standard library' which we must use.

juliaogris commented 4 years ago

Sorry @daemonl I don't get your point again:

Docs are separate by the package structure. The log package has got its docs here: https://pkg.go.dev/github.com/anz-bank/pkg/log

As mentioned earlier - multi go.mod files per repo are tricker (and officially not recommended). They need subpath to be part of tags which cannot be easily automated and doing this manually is error-prone and cumbersome.

daemonl commented 4 years ago

I understand they are tricky, that's one of the many reasons I don't think we should use a mono-repo for go packages.

By docs I mean explicitly document how the separation of these packages works.

I want the separation to be at the repository level. I would be somewhat satisfied by separating them at the module level. If we can't separate them, then this is presented as a single go module which implements - in the future - many different and unrelated functionality. You would have to call that out explicitly in the root readme, make it very clear that this is NOT a base standard ANZx standard library, in a place which is clear and obvious and peer reviewed - i.e., make sure everyone agrees, using github.com/anz-bank/pkg is a purely tactical solution to the problem that ANZ doesn't make open source easy.

camscale commented 4 years ago

We need something (code, docs, split, whatever) to ensure that this is 'just a logging library', and is not step one of a 'anzx standard library' which we must use.

I have seen no evidence that this module is to be an "'anzx standard library' which we must use". The github.com/anz-bank org is not even ANZx.

I understand your concerns - you do not want the packages in this module to become tightly coupled such that using one requires the use of another. This is just good software engineering (low coupling, high cohesion). We do not need to have that called out as in issue - we just cover it in code reviews and ensure unnecessary coupling does not creep in. Splitting this repo into separate repos or making a multi-module repo does not automatically reduce coupling - the same engineering discipline needs to be applied. I could easily add a dependency between modules just as much as I can between packages in a module.

It has always been my understanding within NWOW that each squad has its own autonomy - it can choose to use whatever tools suits it. I have always pushed very hard for this. I expect that over time some commonality will be extracted and squads starting out after this will have the choice to use this common functionality if it suits them.

As for the name pkg - I assume it came from github.com/pkg, but I could be wrong. It doesn't seem to be a very useful name to me. Was it selected just for "policy" reasons?

daemonl commented 4 years ago

pkg == github.com/anz-bank/pkg - Updated the comment to reflect.

As for what this structure implies or does not imply:

I see mono-library-repo as a bad thing because it leads us down the path of 'anz standard packages'.

I see mono-library-repo as a good thing because it leads us down the path of 'anz standard packages'.

It's pretty clear to me that there is a disagreement on the direction. I think that clarifying and removing that disagreement will help us work together on what the code which implements that decision looks like. It needs discussion, If not here, then where?

juliaogris commented 4 years ago

I've spent some time trying out the one go.mod file per package approach. It breaks CI and Makefile with no obvious quick fix, this is another disadvantage aside from no automated versioning.

Separate go modules should go in separate repos IMO.

It would be lovely if we could do this here and have each package in a separate repo. Seeing as there are some procedural hurdles attached to this IMO what we have is the second best option.

anz-bank / pkg

Pkg Monorepo potential for misuse #20