search

m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

add empty go.mod to avoid go mod problems #1038

Closed gfr10598 closed 2 years ago

gfr10598 commented 2 years ago

When building etl-gardener on darwin, go mod fails because of colons in paths in etl/parser/testdata. This allows go mod to ignore that directory.

This change is Reviewable

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 7031

Files with Coverage Reduction New Missed Lines %
active/active.go 4 88.54%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 7029: -0.07%
Covered Lines: 3840
Relevant Lines: 5986
💛 - Coveralls
gfr10598 commented 2 years ago

Ok - my local problem was triggered by upgrading (some months ago) to go116. This version defaults GO111MODULE to "on" instead of "auto", so that triggered complaints that there was no main module, and one should run go mod init etc.

There is a single dependency in etl-gardener on etl/etl/globals.go, so go mod tidy includes etl.

So, this might be the fun bit. The tests include these lines: ./parser/parser_test.go: pipe.Exec("tar", "-C", "testdata", "-xvf", "testdata/pt-files.tar.gz"), ./parser/parser_test.go: pipe.Exec("tar", "-C", "testdata", "-xvf", "testdata/web100-files.tar.gz"), ./parser/parser_test.go: pipe.Exec("tar", "-C", "testdata", "-xvf", "testdata/sidestream-files.tar.gz"), Perhaps go mod tidy is actually looking at these files because they are referenced in the go code.

UPSHOT:

  1. I can solve my local problem with GO111MODULE=auto.
  2. I think it is time to break the etl-gardener -> etl dependency, by moving common code into another repository that contains only the minimum code that is referenced by multiple repositories.
  3. We should probably pay more attention to any warning associated with go modules. I don't believe we should remove colons from filenames, just because some old operating systems made them reserved. If empty go.mod files in test directories avoid go mod problems, that seems like a reasonable solution. Avoiding or hiding code dependencies on archive files seems like the wrong solution.
stephen-soltesz commented 2 years ago

Closing to clean up old PRs. For point 2 - https://github.com/m-lab/etl/issues/1083