On PRs for new providers and parsers

[knadh]

koanf currently bundles the most used providers and parsers. While only the providers that are actually used are compiled, new providers and parsers keep expanding the dependency tree that needs to be pulled and cached.

Thus, it is not ideal to keep bundling more providers and parsers. Anyone who writes a provider or parser, we can list them in the docs for others to easily go get.

[hoshsadiq]

FWIW, I believe some of these providers should be moved to separate repos. Part of the attraction of koanf is that it's extensible. For sure, adding additional packages into here, or keeping them here, doesn't include it in the final binary, but it still affects the performance of say go mod and other things. Separating all of this out minimizes go.mod/go.sum entries, and doesn't confuse people on what they're actually including in their application.

[knadh]

I agree and I've been thinking about this also. Only thing is, moving packages in v1 will break existing implementations. We could do a /v2 and move them out.

[hoshsadiq]

Yes! It might be worth creating a koanf org on github as a placeholder. Such packages could move there when v2 is released. Though, I think the main koanf repo might have to stay under your user.

[gsingh-ds]

Let me know if we can help in any way @knadh

[knadh]

Picking this up now given that https://github.com/knadh/koanf/issues/196 has hit a roadblock.

• Move providers / appconfig, consul, etcd, s3, vault to their own separate repos under the new org.
• Move parsers / hcl, hjson, nestedtext to their own separate repos under the new org.
• Remove these from the koanf repo and make a breaking release /v2.

Makes sense? Any other providers or parsers that should be moved out? @hoshsadiq @rhnvrm ?

[hoshsadiq]

In my opinion, any parser that isn't provided by the go std library should be moved to a separate repo. That includes yaml, dotenv, and toml in addition to the ones you listed.

Same with providers, so in addition to the ones you listed, this should include posflag.

I might have missed things, so worth doing a sanity check before making a breaking change.

[knadh]

In my opinion, any parser that isn't provided by the go std library should be moved to a separate repo. That includes yaml, dotenv, and toml in addition to the ones you listed.

The significant majority of koanf users will be reading a YAML, JSON, or TOML config. I think it's a fair tradeoff to have these bundled in the core. ~90% (making that number up) shouldn't have to pull an external package just to start using koanf.

[hoshsadiq]

Seems reasonable. I guess if we're separating out packaging, it shouldn't matter whether it's pulled from this package or separate package. In the end, the result will be the same, just gives more flexibility to those who don't use those packages.

By the way, as part of this, I wonder if it's worth coming up with a plan to move this package to the new org as well. Assuming it's something that you'd be comfortable doing.

[knadh]

By the way, as part of this, I wonder if it's worth coming up with a plan to move this package to the new org as well.

I've no issues personally, but changing the package name + URL will leave all the existing users in a lurch and probably create a lot of chaos. No way to avoid this, right?

[jxsl13]

I do also use .env parsing heavily for cli apps. It works in conjunction with flag parsing and environment variables, tho externalizing pflag might be ok as well. It would be great if .env file parsing was left in the core. It's like the most basic config format next to environment variables without any nesting. (dotenv)

[knadh]

Yep. env, and even pflag are used in most setups.

[knadh]

I spent several hours (huh) to try out the structure https://cs.opensource.google/go/x/tools follows. Ref: https://go.dev/ref/mod#vcs-dir

Each external package, provider gets its own go.mod. This removes all "sub" references from go.mod from the main package and when koanf is pulled, none of the sub packages are pulled. They've to be installed separately.

koanf
  - go.mod
  - parsers
    - yaml
      - go.mod (github.com/knadh/koanf/v2/parsers/yaml)
    - toml
      - go.mod (github.com/knadh/koanf/v2/parsers/toml)

For v2, these tags should be present. The "sub" packages need their own tags referring to specific versions as per https://go.dev/ref/mod#vcs-dir

• v2.0.0
• v2/parsers/toml/v2.0.0
• v2/parsers/yaml/v2.0.0

go get -u github.com/knadh/koanf-test/v2@v2.0.0 (works)

go get -u github.com/knadh/koanf-test/v2/parsers/toml@v2.0.0

go: downloading github.com/knadh/koanf-test/v2 v2.0.0
go: module github.com/knadh/koanf-test/v2@v2.0.0 found, but does not contain package github.com/knadh/koanf-test/v2/parsers/toml

Tried tagging v2/parsers/toml/v2.0.0, parsers/toml/v2.0.0 etc. This works well with v1, but breaks with v2. Unsure where this is going wrong.

[hoshsadiq]

go get -u github.com/knadh/koanf-test/v2/parsers/toml@v2.0.0

Wouldn't this be?:

go get -u github.com/knadh/koanf-test/parsers/toml@v2

[knadh]

Indeed. Figured this discrepancy a couple of hours ago. https://github.com/knadh/modtest/blob/master/parsers/toml/go.mod

The module name in go.mod in the "sub" module should also not have the v2 path prefix as the "root" module. This was the confusing bit.

This approach works. One bit of headache is that each of these modules need to have tags maintained separately on the main repo, eg: parsers/toml/v2.7.0. This seems like a fair tradeoff because providers and parsers change very rarely.

I'll set up a full fork of koanf with these changes applied and share it here. We can give it a shot and see how it feels to the end user. Should be as simple as go get $koanf and go get $desired-module.

[knadh]

https://github.com/knadh/koanf/pull/199

Ready to merge. Please test.