Pants cannot handle `go.mod` with replace

[asherf]

details: 18:20:36.69 [ERROR] 1 Exception encountered:  Engine traceback: in select in 
pants.backend.project_info.peek.peek in 
pants.engine.internals.graph.resolve_unexpanded_targets in 
pants.engine.internals.graph.resolve_addresses_from_specs in
 pants.engine.internals.build_files.addresses_from_address_specs in pants.engine.internals.graph.resolve_target_parametrizations (plugin/builtin/prefixtransformer:prefixtransformer) in pants.backend.go.target_type_rules.generate_targets_from_go_mod in 
pants.backend.go.util_rules.third_party_pkg.download_and_analyze_third_party_packages in pants.backend.go.util_rules.third_party_pkg.analyze_module_dependencies Traceback (most recent call last): 
File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 705, in 
native_engine_generator_send res = func.send(arg) File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0rc0_py38/lib/python3.8/site-packages/pants/backend/go/util_rules/third_party_pkg.py", line 200, in 

analyze_module_dependencies version = mod_json["Replace"]["Version"] KeyError: 'Version'  FATAL: exception not rethrown

to repo: clone https://github.com/kubernetes-sigs/kustomize Add the pants script and a minimal pants config:

[GLOBAL]
pants_version = "2.11.0rc0"
backend_packages = [
  "pants.backend.experimental.go",
]

run ./pants tailor followed by .pants peek :: crash happens on ./pants peek

[harnash]

It seems I have the same issue when replacing import via go mod's replace directive:

replace gitbub.com/example/package => ../../lib

[asherf]

in pants.backend.go.util_rules.third_party_pkg.download_and_analyze_third_party_packages
in pants.backend.go.util_rules.third_party_pkg.analyze_module_dependencies
Traceback (most recent call last):
File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0a0_py39/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
res = func.send(arg)
File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0a0_py39/lib/python3.9/site-packages/pants/backend/go/util_rules/third_party_pkg.py", line 212, in analyze_module_dependencies
version = mod_json["Replace"]["Version"]
KeyError: 'Version'

still happening in pants 2.14.0 rc0

[rdeusser]

15:55:07.97 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
    ..
  in pants.backend.project_info.list_targets.list_targets
    `list` goal
  in pants.engine.internals.specs_rules.resolve_addresses_from_specs
    Find targets from input specs

Traceback (most recent call last):
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 257, in resolve_addresses_from_specs
    includes, ignores = await MultiGet(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 509, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 245, in resolve_addresses_from_raw_specs
    without_file_owners, with_file_owners = await MultiGet(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 509, in MultiGet
    return await _MultiGet((__arg0, __arg1))
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/specs_rules.py", line 175, in addresses_from_raw_specs_without_file_owners
    target_parametrizations_list = await MultiGet(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/graph.py", line 283, in resolve_target_parametrizations
    all_generated = await MultiGet(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 358, in MultiGet
    return await _MultiGet(tuple(__arg0))
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/backend/go/target_type_rules.py", line 369, in generate_targets_from_go_mod
    all_packages = await Get(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/backend/go/util_rules/third_party_pkg.py", line 577, in download_and_analyze_third_party_packages
    module_analysis = await Get(
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "/Users/me/Library/Caches/nce/c75e6f54c031ca755a5f693d7d6f558fdd6c194598741388cf87aea4895f78ec/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/backend/go/util_rules/third_party_pkg.py", line 235, in analyze_module_dependencies
    version = mod_json["Replace"]["Version"]
KeyError: 'Version'

Looks like this is still a problem in 2.16.0.

[edvardm]

Confirming it still exists in 2.19.1, in my case just running pants tailor :: was sufficient. Project I tried with it contains go.mod with replace pointing to local file without version, but the package compiles fine without Pants.

As per https://go.dev/doc/modules/gomod-ref#replace, local replacements without versions are fine in Go.

🟡 Update: out of curiosity I executed rm **/BUILD and then reran pants tailor ::, it succeeds. But if I then rerun pants tailor :: it will fail with

Engine traceback:
  in `tailor` goal

ProcessExecutionFailure: Process 'Download Go module ../<repo>@latest.' failed with exit code 1.
stdout:

stderr:
go: cannot use relative path ../<repo>@latest to specify module

I wonder if it's due to some cached result, because at some point I tried to fix Pants locally by using pseudo version "latest" in the function analyze_module_dependencies() if the mod_json hash does not have Version key

[benjyw]

So IIUC an omitted Version means the replacement must be a local path? So really this is about supporting local clones of third-party packages?

[edvardm]

I'm quite a newbie in Go still, but that is my understanding. In a monorepo, that local, relative directory would always refer to a directory in the same monorepo and as such it would automatically have the same version

I didn't try yet what would happen if in Pants I'd simply skip such entry. Quite sure it would not be ideal though, as one of the great features of pants is the ability to infer affected files by traversing AST (assuming this applies to Go as well)

[benjyw]

That all makes sense, but I'm trying to figure out what if any are the obstacles to supporting local paths here. @tdyas, is this difficult to do for some reason, or did we just not do it because we didn't get to it?

[wobeng]

Hi guys, I'm having the same issue. Does this issue still persist?

[rdeknijf]

It still exists in 2.22.0 and 2.23.0rc0

Here's a mini repo reproducing it: https://github.com/rdeknijf/pants-repro-go-replace