search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
24.02k stars 3.46k forks source link

Apache licensed packages depend on AGPL packages #13019

Open lionello opened 6 months ago

lionello commented 6 months ago

Describe the bug Most of Loki is licensed with AGPL, but some packages in this repo use the Apache (2) license. These either have a file LICENSE_APACHE2 and/or are listed in https://github.com/grafana/loki/blob/c0113db4e8c4647188db6477d2ab265eda8dbb6c/LICENSING.md

Importing an AGPL licensed package in an Apache licensed package makes it effectively AGPL licensed. It's near-impossible to detect whether the imported Apache licensed package ended up pulling in, compiling, linking, AGPL licensed code.

To Reproduce Steps to reproduce the results below:

  1. git clone git@github.com:lionello/GoLicenseGuard.git
  2. go build to build the GoLicenseGuard tool
  3. Change directory to a Go package using Loki
  4. Run GoLicenseGuard

Expected behavior I expected to see no output, ie. Apache licensed packages only depend on other Apache (or non-AGPL) licensed package. I expected an Apache licensed package to be self contained and have all type and function declarations that it needs and not depend on internal Loki AGPL licensed code.

Screenshots, Promtail config, or terminal output Using https://github.com/lionello/GoLicenseGuard:

$ GoLicenseGuard
Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/loghttp using packages:
  imports github.com/grafana/loki/v3/pkg/logql (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/logql/log (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/logql/syntax (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/querier/plan (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/storage/stores/index/seriesvolume (AGPL-3.0)
Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/logproto using packages:
  imports github.com/grafana/loki/v3/pkg/logql/syntax (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/querier/plan (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase/definitions (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/storage/chunk/cache/resultscache (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/storage/stores/shipper/indexshipper/tsdb/index (AGPL-3.0)
Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/logqlmodel using packages:
  imports github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase/definitions (AGPL-3.0)
Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/logqlmodel/metadata using packages:
  imports github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase/definitions (AGPL-3.0)
Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/util/server using packages:
  imports github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/storage/errors (AGPL-3.0)
paul1r commented 6 months ago

Those packages are explicitly called out in this licensing doc: https://github.com/grafana/loki/blob/main/LICENSING.md

You are correct, the default license for Loki is AGPL, but there are a handful of directories that are Apache licensed.

lionello commented 6 months ago

Those packages are explicitly called out in this licensing doc: https://github.com/grafana/loki/blob/main/LICENSING.md

You are correct, the default license for Loki is AGPL, but there are a handful of directories that are Apache licensed.

That’s true, I linked to the same LICENSING document. The problem is that those packages depend on AGPL packages. That’s a problem, and it doesn’t matter that those are also from Loki. Would be a problem to depend on 3rd party AGPL packages too.

I think the only way around it is to move any AGPL code used in the Apache licensed packages into those packages.

paul1r commented 6 months ago

I am certainly not saying you are incorrect, but I do need to get a bit more understanding here.

First off, I have not been able to get any output from that tool showing an Apache->AGPL error as you have pasted above. Can you give me more information on how you ran the tool? (I have added some debugging information and the like to ensure it is running, in some form, on my system. I've also tried creating a test package where I import a supposedly invalid item into an Apache 2 licensed project.)

Using part of the snippet above:

Apache-2.0 licensed package github.com/grafana/loki/v3/pkg/util/server using packages:
  imports github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase (AGPL-3.0)
  imports github.com/grafana/loki/v3/pkg/storage/errors (AGPL-3.0)

I believe that the server package only references queryrangebase within recovery.go (and its corresponding test file). Looking at those directories: -github.com/grafana/loki/v3/pkg/util/ contains an Apache 2 license -github.com/grafana/loki/v3/pkg/util/server does not contain a license file, and is part of the Apache 2 license from the directory above

So, with that particular example, I see something that looks potentially valid, and something that doesn't. Can you explain your license tool a bit more, notably how to run it to recreate your output, and whether or not individual files are examined?

lionello commented 6 months ago

First off, I have not been able to get any output from that tool showing an Apache->AGPL error as you have pasted above. Can you give me more information on how you ran the tool? (I have added some debugging information and the like to ensure it is running, in some form, on my system. I've also tried creating a test package where I import a supposedly invalid item into an Apache 2 licensed project.)

The tool uses go list -deps -json under the covers, as such it will need a project using Loki for it to pick it up as a dependency. There might be a way to use go list to enumerate everything in the GOHOME folder, but that seems bit too arbitrary to me.

I believe that the server package only references queryrangebase within recovery.go (and its corresponding test file). Looking at those directories: -github.com/grafana/loki/v3/pkg/util/ contains an Apache 2 license -github.com/grafana/loki/v3/pkg/util/server does not contain a license file, and is part of the Apache 2 license from the directory above

That's what the tool does: checks for a license* file in the package directory or its parent etc.

  • recovery.go only references roundtrip.go in the github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase package, which itself has an Apache 2 license embedded in the header of the file. As such, I'm not convinced this is an issue.

Ah, the tool only checks for separate license files. In fact, the go list tool only gives me package (directory) names, so that trick can't be used to find file-level dependencies and check those files for license headers :( So there will be false positives in the output.

So, with that particular example, I see something that looks potentially valid, and something that doesn't. Can you explain your license tool a bit more, notably how to run it to recreate your output, and whether or not individual files are examined?

Here's a simple example: https://gist.github.com/lionello/96784ad698d05d281c122417aef8cc9c This example is using the HTTP API, all types here are from the Apache licensed loghttp package, yet it pulls in AGPL packages (check the go.mod!) Put the two files in a new folder and run go mod tidy, then GoLicenseGuard from that folder. It should result in the same output as the one from my post above (perhaps different order.)

To be honest, just the size of that go.mod shows the loghttp package's transitive dependencies are out of control. It's also worth noting that the v1 version of loghttp is clean! But using v3 results in these excessive (and AGPL) dependencies.

lionello commented 6 months ago

FWIW, I've updated the GoLicenceGuard to check the headers of the individual source files in a package, in lieu of an absent LICENSE* file, but it didn't affect the results of this particular example, possibly because go list returns the Go files in a package, but not which ones are actually being used.

paul1r commented 6 months ago

Here's a simple example: https://gist.github.com/lionello/96784ad698d05d281c122417aef8cc9c This example is using the HTTP API, all types here are from the Apache licensed loghttp package, yet it pulls in AGPL packages (check the go.mod!) Put the two files in a new folder and run go mod tidy, then GoLicenseGuard from that folder. It should result in the same output as the one from my post above (perhaps different order.)

I was able to recreate the output you sent, thank you.

To be honest, just the size of that go.mod shows the loghttp package's transitive dependencies are out of control. It's also worth noting that the v1 version of loghttp is clean! But using v3 results in these excessive (and AGPL) dependencies.

While many files were touched for the v3 versioning, I truly suspect there is a tooling issue related to the go list command in the v1 modules, which were not fully compliant with go module version support. The actual change for the errors import was minimal. I've not checked other imports, but there was not a huge amount of structural change for v3.

FWIW, I've updated the GoLicenceGuard to check the headers of the individual source files in a package, in lieu of an absent LICENSE* file, but it didn't affect the results of this particular example, possibly because go list returns the Go files in a package, but not which ones are actually being used.

Thank you. So, at this point, it is a manual exercise of going through these results to validate them and then take action on the items if applicable. It would be fantastic to find a way to see what files were actually being used to weed out the false positives, so that we could work on building a CI step that does this routinely.

paul1r commented 6 months ago

removed_files.txt required_files.txt

I did some manual work using the gist example you sent. Within the vendor directory for Loki, there are files pulled in that aren't needed by this particular example (removed_files.txt).

The required_files.txt has a list of all files that are needed in order to get the gist to run, and there is indeed a mix of Apache 2.0 and AGPL licenses.