fix go-modules/release process

[david415]

1. we do not understand exactly what is broken.
2. we observe that for example catchat can be built by some developers but not others
3. @mixmasala has a prescription for solving this issue and it is:

09:16 < masala> dawuud: maybe this go module problem is just because kimchi is listed in go.mod of authority 09:17 < masala> if i remove that and then go get -u -v, the dependencies for tests are not added to go.mod 09:17 < masala> ie, it doesnt get re-added 09:17 < masala> perhaps it's enough to just remove that dep from go.mod and then tag a new authority release and update everything 09:19 < dawuud> either way it's solved by not using kimchi 09:20 < dawuud> and since kimchi is optional i recommend not using it for the time being 09:21 < dawuud> so yeah... sounds like that plan could work 09:27 < masala> i hear what you are saying 09:27 < masala> but i dont want to remove unit tests from stuff without a replacement 09:28 < masala> we just need to make kimchi not be a top level dependency of the package 09:28 < masala> and that should resolve the circular import 09:35 < masala> 0988e94b7b62fe239fe05f0dc43abd79e56b23b8 is the commit that addd it 09:35 < masala> i guess when you update deps don't use the -t option 09:35 < masala> (for tests) 09:36 < masala> if you could fix that before you go to sleep that would be nice, since i cannot sign tags or fix this myself

10:32 < dawuud> we can easily replace the tests with docker based tests 10:33 < dawuud> i mean... yeah i can push a new signed tag 10:33 < dawuud> but did you push the commit that you want me to tag? 10:34 < dawuud> also it's hella late where i am but i'm up because insomnia 10:35 < masala> remove_kimchi_dep, 10:36 < masala> 87c4b4f2602ac2828428b656e12badd96d9d56f0 10:36 < masala> i just remove that line from go.mod 10:36 < masala> which should prevent kimchi from getting pulled in when authority is included 10:36 < masala> i believe this happens if you do a go get -u -v -t 10:37 < masala> which adds all the testing deps into the module deps, so we should avoid doing that 10:37 < masala> i'm not sure kimchi should even use modules, because it should not get pinned to some versions of packages 10:39 < masala> i also pushing a branch with same name for client 10:39 < masala> and looking for other repositories where kimchi is a top level dep 10:41 < masala> also memspool 10:42 < masala> so, memspool, client, and authority had kimchi in go.mod 10:43 < masala> anything depending on those repos will also need to be updated tho 10:51 < dawuud> i'm too tired to do this right now 10:51 < dawuud> but it sounds like you figured it out 10:56 < masala> ok 13:23 < muesli> masala: maybe "go mod tidy" would have been enough? that should remove unneeded/obsolete deps 15:36 < masala> hm let me see 15:36 < masala> ah that command fails with the same error due to whatever yawnings git server does 15:37 < masala> i think we just need to nuke those deps in go.mod manually 15:42 < masala> idk why go mod tidy starts with following the existing dependency chain, meh 15:44 < masala> it shouldn't be too much work to update the above repos and then the things that depend on them, i guess i'll just avoid messing around with stuff till after dawuud has recharged 18:23 < dawuud> i'd like to reiterate just how easy it is to write docker based tests... and you can compare in the "client" the kimchi vs docker tests. 18:24 < dawuud> you seem very attached to kimchi 18:24 < dawuud> and i'm not clear on the best course of action but i'm a fan of iterating towards it 18:25 < dawuud> so if this kimchi tries to take up several more days of your time i'd suggest temporarily dropping it 18:25 < dawuud> and we can come back to it later 18:25 < dawuud> on the other hand maybe you are close to fixing it? 18:29 < masala> i'm not focused on writing a replacement at the moment 18:29 < masala> the fix i proposed is to remove it from the top level dependency 18:30 < masala> it's quite clear that the breakage occurred on 8th april with the commit i outlined 18:31 < masala> tagged releases are signed with a key you control so i am unable to just fix it myself 19:21 < dawuud> no you are unable to fix it because you may not be totally familiar with our release process 19:22 < dawuud> thing is... you can sign tags with whatever key you want 19:22 < masala> that is an unusual release arrangement. 19:22 < dawuud> yes it is but it's better than "i can't fix it because i don't have the magical key" 19:23 < dawuud> people have to trust us 19:23 < dawuud> either my key or yours or whatever 19:23 < masala> you're right. i do not understand your release process. 19:24 < dawuud> ok so i'll do it now in that case 19:24 < dawuud> but eventually i'd like to transfer this knowledge to others 19:25 < dawuud> i've documented parts of the process but not the whole thing since it become more complex with go modules 19:25 < dawuud> so.. to get clear on what you want me to do... 19:26 < dawuud> you want me to make a new release with what's currently in authority master? 19:26 < dawuud> did any other repos get updated or what? 19:26 < dawuud> i'll be touching a cluster fuck of repos either way 19:27 < masala> i believe we just need to remove the dependency from the repos i outlined above and the repos that include those repos 19:27 < masala> yes 19:27 < masala> indeed it is a fuckload of repos

[david415]

user@computer:~/gopath/src/github.com/katzenpost/catshadow$ go get -u -v github.com/katzenpost/client@v0.0.4 go: finding github.com/agl/ed25519 latest go: finding github.com/katzenpost/chacha20 latest go: finding github.com/katzenpost/newhope latest go: finding git.schwanenlied.me/yawning/aez.git latest go: finding git.schwanenlied.me/yawning/bsaes.git latest go: finding golang.org/x/sys latest go: finding github.com/katzenpost/noise latest go: finding gopkg.in/op/go-logging.v1 latest go: finding golang.org/x/crypto latest go: finding golang.org/x/net latest ../../../../pkg/mod/github.com/katzenpost/core@v0.0.8/crypto/eddsa/eddsa.go:31:2: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519/extra25519

[david415]

user@computer:~/gopath/src/github.com/katzenpost/catshadow$ go mod tidy go: finding github.com/agl/ed25519 latest github.com/katzenpost/catshadow imports github.com/katzenpost/doubleratchet imports github.com/agl/ed25519: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519 github.com/katzenpost/catshadow imports github.com/katzenpost/doubleratchet imports github.com/agl/ed25519/extra25519: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519/extra25519 user@computer:~/gopath/src/github.com/katzenpost/catshadow$

[david415]

go mod tidy go: finding github.com/agl/ed25519 latest github.com/katzenpost/catshadow imports github.com/katzenpost/doubleratchet imports github.com/agl/ed25519: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519 github.com/katzenpost/catshadow imports github.com/katzenpost/doubleratchet imports github.com/agl/ed25519/extra25519: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519/extra25519

[david415]

user@computer:~/gopath/src/github.com/katzenpost/catchat$ go get -u -v github.com/katzenpost/catshadow@v0.0.5 go: finding github.com/katzenpost/catshadow v0.0.5 go: downloading github.com/katzenpost/catshadow v0.0.5 go: extracting github.com/katzenpost/catshadow v0.0.5 go: finding github.com/agl/ed25519 latest go: finding git.schwanenlied.me/yawning/aez.git latest go: finding git.schwanenlied.me/yawning/bsaes.git latest go: finding github.com/golang/protobuf v1.4.0 go: finding golang.org/x/crypto latest go: finding golang.org/x/sys latest go: finding github.com/katzenpost/noise latest go: finding github.com/awnumar/memcall latest go: finding gopkg.in/op/go-logging.v1 latest go: finding github.com/katzenpost/chacha20 latest go: finding golang.org/x/net latest go: finding github.com/katzenpost/newhope latest go: downloading github.com/golang/protobuf v1.4.0 go: extracting github.com/golang/protobuf v1.4.0 go: downloading google.golang.org/protobuf v1.21.0 go: extracting google.golang.org/protobuf v1.21.0 go: finding google.golang.org/protobuf v1.21.0 ../../../../pkg/mod/github.com/katzenpost/doubleratchet@v0.0.2/ratchet.go:14:2: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519 ../../../../pkg/mod/github.com/katzenpost/core@v0.0.8/crypto/eddsa/eddsa.go:31:2: module github.com/agl/ed25519@latest (v0.0.0-20200225211852-fd4d107ace12) found, but does not contain package github.com/agl/ed25519/extra25519

[david415]

builds are working but the go modules errors are disturbing. i do not understand the root problem yet.

[david415]

cyclic deps removed