Closed decanus closed 5 years ago
@oskarth can you try running this, I can't seem to build the client
Unable to build with make
or go get
:
> git pull decanus feature/data-sync
From github.com:decanus/status-console-client
* branch feature/data-sync -> FETCH_HEAD
Already up to date.
>
> make build
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
build github.com/status-im/status-console-client: cannot find module for path github.com/status-im/mvds
make: *** [Makefile:7: build] Error 1
> go get github.com/status-im/mvds
go: finding github.com/libp2p/go-libp2p-loggables v1.1.24
go: finding github.com/libp2p/go-libp2p v6.0.2+incompatible
go: finding github.com/libp2p/go-buffer-pool v0.1.1
go: finding github.com/ipfs/go-log v1.5.7
go: finding github.com/libp2p/go-libp2p-host v3.0.15+incompatible
go: finding github.com/libp2p/go-libp2p-metrics v2.1.7+incompatible
go: finding github.com/libp2p/go-flow-metrics v0.2.0
go: finding github.com/libp2p/go-libp2p-crypto v2.0.1+incompatible
go: finding github.com/libp2p/go-libp2p-interface-pnet v3.0.0+incompatible
go: finding github.com/jackpal/go-nat-pmp v1.0.1
go: github.com/libp2p/go-flow-metrics@v0.2.0: unknown revision v0.2.0
go: finding github.com/libp2p/go-sockaddr v1.0.3
go: github.com/libp2p/go-libp2p-loggables@v1.1.24: unknown revision v1.1.24
go: finding github.com/ipfs/go-datastore v3.2.0+incompatible
go: github.com/libp2p/go-buffer-pool@v0.1.1: unknown revision v0.1.1
go: github.com/libp2p/go-libp2p-metrics@v2.1.7+incompatible: unknown revision v2.1.7
go: github.com/libp2p/go-libp2p-crypto@v2.0.1+incompatible: unknown revision v2.0.1
go: finding github.com/libp2p/go-libp2p-secio v2.0.17+incompatible
go: finding github.com/ipfs/go-cid v0.9.0
go: github.com/libp2p/go-libp2p-host@v3.0.15+incompatible: unknown revision v3.0.15
go: finding github.com/libp2p/go-libp2p-protocol v1.0.0
go: finding github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324
go: github.com/ipfs/go-log@v1.5.7: unknown revision v1.5.7
go: finding github.com/libp2p/go-libp2p-interface-connmgr v0.0.21
go: github.com/libp2p/go-libp2p-interface-pnet@v3.0.0+incompatible: unknown revision v3.0.0
go: finding github.com/libp2p/go-libp2p-nat v0.8.8
go: github.com/libp2p/go-libp2p@v6.0.2+incompatible: unknown revision v6.0.2
go: finding github.com/libp2p/go-reuseport v0.1.18
go: github.com/libp2p/go-sockaddr@v1.0.3: unknown revision v1.0.3
go: finding github.com/libp2p/go-addr-util v2.0.7+incompatible
go: github.com/ipfs/go-datastore@v3.2.0+incompatible: unknown revision v3.2.0
go: finding github.com/ipfs/go-ipfs-util v1.2.8
go: github.com/libp2p/go-libp2p-protocol@v1.0.0: unknown revision v1.0.0
go: finding github.com/libp2p/go-maddr-filter v1.1.10
go: github.com/libp2p/go-libp2p-interface-connmgr@v0.0.21: unknown revision v0.0.21
go: finding github.com/libp2p/go-tcp-transport v2.0.16+incompatible
go: github.com/ipfs/go-cid@v0.9.0: unknown revision v0.9.0
go: finding github.com/libp2p/go-libp2p-peerstore v1.5.0
go: github.com/libp2p/go-libp2p-secio@v2.0.17+incompatible: unknown revision v2.0.17
go: finding github.com/multiformats/go-multibase v0.3.0
go: github.com/libp2p/go-libp2p-nat@v0.8.8: unknown revision v0.8.8
go: finding github.com/multiformats/go-multihash v1.0.8
go: github.com/libp2p/go-reuseport@v0.1.18: unknown revision v0.1.18
go: finding github.com/libp2p/go-reuseport-transport v0.1.11
go: finding github.com/multiformats/go-multistream v0.3.9
go: github.com/libp2p/go-addr-util@v2.0.7+incompatible: unknown revision v2.0.7
go: github.com/ipfs/go-ipfs-util@v1.2.8: unknown revision v1.2.8
go: finding github.com/libp2p/go-libp2p-circuit v2.2.2+incompatible
go: github.com/libp2p/go-tcp-transport@v2.0.16+incompatible: unknown revision v2.0.16
go: github.com/libp2p/go-maddr-filter@v1.1.10: unknown revision v1.1.10
go: finding github.com/libp2p/go-conn-security v0.1.15
go: finding github.com/jackpal/gateway v1.0.4
go: github.com/libp2p/go-libp2p-peerstore@v1.5.0: unknown revision v1.5.0
go: github.com/multiformats/go-multibase@v0.3.0: unknown revision v0.3.0
go: finding github.com/libp2p/go-mplex v0.2.30
go: github.com/multiformats/go-multihash@v1.0.8: unknown revision v1.0.8
go: finding github.com/libp2p/go-stream-muxer v3.0.1+incompatible
go: finding golang.org/x/text v0.3.0
go: finding golang.org/x/sys v0.0.0-20180926160741-c2ed4eda69e7
go: finding github.com/libp2p/go-conn-security-multistream v0.1.15
go: github.com/multiformats/go-multistream@v0.3.9: unknown revision v0.3.9
go: finding github.com/libp2p/go-libp2p-routing v2.7.1+incompatible
go: github.com/libp2p/go-reuseport-transport@v0.1.11: unknown revision v0.1.11
go: finding golang.org/x/net v0.0.0-20180524181706-dfa909b99c79
go: finding github.com/libp2p/go-libp2p-transport-upgrader v0.1.16
go: github.com/libp2p/go-libp2p-circuit@v2.2.2+incompatible: unknown revision v2.2.2
go: finding github.com/multiformats/go-multiaddr v1.2.7
go: github.com/libp2p/go-conn-security@v0.1.15: unknown revision v0.1.15
go: finding github.com/multiformats/go-multiaddr-dns v0.2.3
go: github.com/libp2p/go-mplex@v0.2.30: unknown revision v0.2.30
go: finding github.com/libp2p/go-libp2p-net v3.0.2+incompatible
go: github.com/libp2p/go-stream-muxer@v3.0.1+incompatible: unknown revision v3.0.1
go: finding github.com/libp2p/go-ws-transport v2.0.15+incompatible
go: github.com/libp2p/go-conn-security-multistream@v0.1.15: unknown revision v0.1.15
go: finding github.com/libp2p/go-msgio v0.0.6
go: finding github.com/libp2p/go-libp2p-transport v3.0.15+incompatible
go: github.com/libp2p/go-libp2p-routing@v2.7.1+incompatible: unknown revision v2.7.1
go: finding github.com/libp2p/go-libp2p-peer v2.4.0+incompatible
go: github.com/libp2p/go-libp2p-transport-upgrader@v0.1.16: unknown revision v0.1.16
go: github.com/multiformats/go-multiaddr@v1.2.7: unknown revision v1.2.7
go: github.com/multiformats/go-multiaddr-dns@v0.2.3: unknown revision v0.2.3
go: github.com/libp2p/go-ws-transport@v2.0.15+incompatible: unknown revision v2.0.15
go: github.com/libp2p/go-libp2p-net@v3.0.2+incompatible: unknown revision v3.0.2
go: github.com/libp2p/go-msgio@v0.0.6: unknown revision v0.0.6
go: github.com/libp2p/go-libp2p-transport@v3.0.15+incompatible: unknown revision v3.0.15
go: github.com/libp2p/go-libp2p-peer@v2.4.0+incompatible: unknown revision v2.4.0
go: error loading module requirements
Still unable to build
> make clean
make: *** No rule to make target 'clean'. Stop.
> make build
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
# github.com/status-im/status-console-client/vendor/github.com/ethereum/go-ethereum/crypto/secp256k1
vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/curve.go:42:10: fatal error: libsecp256k1/include/secp256k1.h: No such file or directory
#include "libsecp256k1/include/secp256k1.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/status-im/status-console-client/vendor/github.com/karalabe/hid
vendor/github.com/karalabe/hid/hid_enabled.go:23:11: fatal error: os/threads_posix.c: No such file or directory
#include "os/threads_posix.c"
^~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/status-im/status-console-client/vendor/gopkg.in/olebedev/go-duktape.v3
duk_logging.c: In function ‘duk__logger_prototype_log_shared’:
duk_logging.c:184:64: warning: ‘Z’ directive writing 1 byte into a region of size between 0 and 9 [-Wformat-overflow=]
sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
^
duk_logging.c:184:2: note: ‘sprintf’ output between 25 and 85 bytes into a destination of size 32
sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.year, (int) comp.month + 1, (int) comp.day,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.hours, (int) comp.minutes, (int) comp.seconds,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.milliseconds);
~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:7: build] Error 2
^ @mandrigin @dshulyak @adambabik do you know?
Would be great if we can get this merged as soon as possible
@oskarth use make vendor
for dependencies. go mod vendor
doesn't preserve non-go files, so it needs a bit of help. basically this error suggests that go-ethereum got updated/pulled by go mod and all c-deps were removed
not sure what this one is about
# github.com/status-im/status-console-client/vendor/gopkg.in/olebedev/go-duktape.v3
duk_logging.c: In function ‘duk__logger_prototype_log_shared’:
duk_logging.c:184:64: warning: ‘Z’ directive writing 1 byte into a region of size between 0 and 9 [-Wformat-overflow=]
sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
^
duk_logging.c:184:2: note: ‘sprintf’ output between 25 and 85 bytes into a destination of size 32
sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.year, (int) comp.month + 1, (int) comp.day,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.hours, (int) comp.minutes, (int) comp.seconds,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(int) comp.milliseconds);
~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:7: build] Error 2
> make vendor
go mod vendor
go: modules disabled inside GOPATH/src by GO111MODULE=auto; see 'go help modules'
make: *** [Makefile:32: vendor] Error 1
Sorry I'm not too familiar with the go module system, but this is what I see
> GO111MODULE=on make vendor
go mod vendor
$GOPATH/go.mod exists but should not
make: *** [Makefile:32: vendor] Error 1
For the ethereum error this workaround also worked:
cp -r \
"${GOPATH}/src/github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1" \
"vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/"
But didn't figure out the equivalent for other error. Btw, live discussion in #status-protocol
@oskarth go modules should be fixed now, could you please confirm?
@decanus Yay, yes make build
builds successfully now.
I also get the
> ./bin/status-term-client -create-key-pair
Your private key: 0x06e4f3572a0b47d2fc333f6172386474dbb4701031ea004aca451d6944bd35a7
> ./bin/status-term-client -keyhex=0x06e4f3572a0b47d2fc333f6172386474dbb4701031ea004aca451d6944bd35a7
file is encrypted or is not a database
>
that you get, IIRC
hey, sorry i will have to merge status-go update cause it solves certain issues in the client. i wanted to wait till this PR gets merged, but it doesn't seem that it will be merged today
@dshulyak I merged master, now I am running into the following error
build github.com/status-im/status-console-client: cannot load github.com/ethereum/go-ethereum/crypto/sha3: cannot find module providing package github.com/ethereum/go-ethereum/crypto/sha3
Any ideas?
it looks as if you were using old status-go version that tries to import that package. can you check that you have v0.25.0-beta.2 in your update?
@dshulyak I always have issues updating the local packages found in vendor? How do I do this?
so all versions described in go.mod file. i noticed several issues in your go.mod. status-go points to 0.16.4 which is very old. go-ethereum points to 1.8.27 and doesn't use our fork.
i suggest to use go.mod from master and update it with mvds dependency. i experimented with it and i am able to build your branch. so here is a go.mod file https://github.com/status-im/status-console-client/blob/datasync-mod/go.mod
additionally i had to use go-ethreum 1.8.23 version in mvds, otherwise it conflicts with version in console client (hence replace directive in my go.mod, you won't need it)
to update a single dep i use go get or i put it to go.mod file manually. for example with mvds i just added a line with mvds dep and replace directive for it. then make vendor worked without issues
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
build github.com/status-im/status-console-client: cannot load github.com/ethereum/go-ethereum/crypto/sha3: open /home/oskarth/go/src/github.com/status-im/status-console-client/vendor/github.com/ethereum/go-ethereum/crypto/sha3: no such file or directory
make: *** [Makefile:7: build] Error 1
^ works
Can we do make vendor in make build or does this not make sense?
@decanus do you want to push branch to status-im so we can push patches to this PR?
@oskarth I can't because I don't have permissions on the status repo
Also, when following your steps I get the following error:
go mod vendor
go: import "github.com/status-im/status-console-client" ->
import "github.com/status-im/status-go/node" ->
import "github.com/status-im/status-go/services/personal" ->
import "github.com/ethereum/go-ethereum/ethapi": looping trying to add package
make: *** [vendor] Error 1
@decanus Oops, added protocol team now, should work
When running make vendor
Latest with existing go.mod file:
go: import "github.com/status-im/status-console-client" ->
import "github.com/status-im/status-go/node" ->
import "github.com/status-im/status-go/services/personal" ->
import "github.com/ethereum/go-ethereum/ethapi": looping trying to add package
make: *** [Makefile:32: vendor] Error 1
Latest with @dshulyak go.mod file:
# github.com/status-im/status-console-client
./main.go:356:20: cannot use t (type *adapters.DataSyncWhisperTransport) as type mvds.Transport in argument to mvds.NewNode:
*adapters.DataSyncWhisperTransport does not implement mvds.Transport (wrong type for Watch method)
have Watch() mvds.Packet
want Watch() *mvds.Packet
make: *** [Makefile:7: build] Error 2
@decanus wdyt about merging partial progress? as long as other things don't break, then can fix bugs etc in future PRs
@oskarth I'd tend to agree, let's have @mandrigin throw another look at it along with @adambabik
Ping @adambabik @mandrigin
Unless this breaks master and it is under flag I'd rather we merge this and continue with smaller incremental fixes
@oskarth @decanus I need some introduction to understand how we want this to work.
I understand that data sync algorithm is implemented in https://github.com/status-im/mvds. Now, it uses Whisper to subscribe for messages. It needs already decoded message and inspects messages' payload to figure out what type of a message is received and what to do with it. Is this correct?
If that's the correct description, we may want to adjust the adapter a bit then. The problem is that currently in the adapter we decode message's payload using transit format. In the case of data sync layer, that's incorrect, right? Payload is not transit format but mvds format.
Generally, implementing this, you should not write a new transport unless you want something different than Whisper. Instead, there should be a mechanism to inject mvds which would handle the message, store mvds details somewhere (as it needs to be used later I believe) and return protocol.Message
.
@adambabik I would agree that we shouldn't have to write a new transport for this, how best would we handle it?
@adambabik if we do refactor, I'd prefer to merge this as is and then do the rest in a different PR, otherwise this will start getting pretty large.
@decanus I will suggest merging it to a long-lived branch like data-sync
instead of master
. I think it's crucial we have this transport thing figured out, however, I also agree there is no point in keeping this PR for longer.
I will adjust the code so it's possible to specify a custom decoding function instead of using transit decoder by default. That should allow us to refactor the code, remove the new transport and merge to master
.
Does it sound good?
@adambabik sounds good, maybe this should also fall into refactoring the transports so PFS is removed from them and put at a different layer in the code. Also I've created the branch data-sync, lets merge this in?
We will need to think about the Sync state and message storage, so we ensure we put messages somewhere where they are presisted if the application is closed. This is important for the sync state too, so we continue retransmitting once repopened, for this to work we should probably pass the start epoch to a node.
I will adjust the code so it's possible to specify a custom decoding function instead of using transit decoder by default. That should allow us to refactor the code, remove the new transport and merge to master.
@adambabik is there an issue for this?
There is a PR for this: https://github.com/status-im/status-console-client/pull/81
We will need to think about the Sync state and message storage, so we ensure we put messages somewhere where they are presisted if the application is closed. This is important for the sync state too, so we continue retransmitting once repopened, for this to work we should probably pass the start epoch to a node.
This will be possible with: https://github.com/status-im/mvds/pull/44 & https://github.com/status-im/mvds/pull/43
@adambabik @dshulyak I removed protocol buffers from the MVDS repo so it fails, how should I handle this best? put it in the console client make file or readd it to the github of MVDS?
you should have golang bindings generated from .proto file in your repo
This pull request implements Data Sync functionality using status-im/mvds, it can be enabled using the flag
ds
.PFS is not supported for the data sync prototype.