inps/go-secp256k1 dep causing `duplicate link` issues with go-ethereum in the same go module as lotus

Schwartz10 commented 1 year ago

Checklist

[X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
[X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
[X] This is not a new feature request. If it is, please file a feature request instead.
[X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
[X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
[X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
[X] I did not make any code changes to lotus.

Lotus component

[ ] lotus daemon - chain sync
[ ] lotus miner - mining and block production
[ ] lotus miner/worker - sealing
[ ] lotus miner - proving(WindowPoSt)
[ ] lotus miner/market - storage deal
[ ] lotus miner/market - retrieval deal
[ ] lotus miner/market - data transfer
[ ] lotus client
[ ] lotus JSON-RPC API
[ ] lotus message management (mpool)
[ ] Other

Lotus Version

lotus `v1.18.1-0.20221201215704-9bf6052069de`

Describe the Bug

Thanks @Stebalien for the help on articulating this issue:

ipsn/go-secp256k1 is a fork of go-ethereum/crypto, which is used in filecoin-project/go-crypto (filecoin-project/go-crypto go.mod)

Because ipsn/go-secp256k1 is a fork of go-ethereum/crypto, it reuses the same C libraries, with the same symbols.

The result is that importing both go-ethereum/ethereum and filecoin-project/lotus in the same go module causes build errors when running clang. I expect relying on these two modules will be a not-so-uncommon use case among early FEVM builders, since there's helpful APIs in both.

Steb had two ideas:

"We might be able to just switch to go-ethereum/crypto/secp256k1.. We didn't do that before because go couldn't do "partial" dependencies. But now it can."
"Perform a mass symbol rename in go-secp256k1 as to avoid duplicate link conflicts"

Logging Information

/usr/local/Cellar/go/1.19.3/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
duplicate symbol '_secp256k1GoPanicError' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000005.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000018.o
duplicate symbol '_secp256k1GoPanicIllegal' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000005.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000018.o
duplicate symbol '_secp256k1_context_destroy' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_seckey_verify' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_verify' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_recoverable_signature_convert' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_nonce_function_default' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_recoverable_signature_serialize_compact' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_signature_serialize_compact' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_recoverable_signature_parse_compact' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_signature_parse_compact' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_recover' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_signature_serialize_der' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_signature_parse_der' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_sign' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ext_scalar_mul' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_privkey_tweak_mul' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_tweak_mul' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_context_set_error_callback' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_context_set_illegal_callback' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_context_randomize' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_signature_normalize' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_serialize' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_create' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_context_create' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_parse' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_context_clone' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_combine' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ecdsa_sign_recoverable' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_privkey_tweak_add' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_ec_pubkey_tweak_add' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_CURVE_B' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
duplicate symbol '_secp256k1_nonce_function_rfc6979' in:
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000008.o
    /var/folders/lf/szs6pm8n7xn5wx2s5_k52w2r0000gn/T/go-link-4286204756/000021.o
ld: 33 duplicate symbols for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)



### Repo Steps

_No response_

Schwartz10 commented 1 year ago

Hey @arajasek - i went ahead and just did the name remappings and things are working again on my end. A simple replace of https://github.com/ipsn/go-secp256k1 => https://github.com/glifio/go-secp256k1 v0.0.1 does the trick

jennijuju commented 1 year ago

@Kubuxu @magik6k thoughts?

Kubuxu commented 1 year ago

Let’s try go-ethereum , keeping it to release versions not git commit.

scotthconner commented 1 year ago

It kind of looks like here that @Schwartz10 fixed this with the symbol renaming. Arugably #1 may be a little cleaner as it reduces the dependencies requirements by 1, but is this fixed? Is there a PR for Jon's fix?

filecoin-project / lotus