search

status-im / status-console-client

Status messaging console user interface
Mozilla Public License 2.0
10 stars 2 forks source link

feature/data-sync #34

Closed decanus closed 5 years ago

decanus commented 5 years ago

This pull request implements Data Sync functionality using status-im/mvds, it can be enabled using the flag ds.

PFS is not supported for the data sync prototype.

decanus commented 5 years ago

@oskarth can you try running this, I can't seem to build the client

oskarth commented 5 years ago

Unable to build with make or go get:

> git pull decanus feature/data-sync
From github.com:decanus/status-console-client
 * branch            feature/data-sync -> FETCH_HEAD
Already up to date.
> 
> make build
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
build github.com/status-im/status-console-client: cannot find module for path github.com/status-im/mvds
make: *** [Makefile:7: build] Error 1
> go get github.com/status-im/mvds
go: finding github.com/libp2p/go-libp2p-loggables v1.1.24
go: finding github.com/libp2p/go-libp2p v6.0.2+incompatible
go: finding github.com/libp2p/go-buffer-pool v0.1.1
go: finding github.com/ipfs/go-log v1.5.7
go: finding github.com/libp2p/go-libp2p-host v3.0.15+incompatible
go: finding github.com/libp2p/go-libp2p-metrics v2.1.7+incompatible
go: finding github.com/libp2p/go-flow-metrics v0.2.0
go: finding github.com/libp2p/go-libp2p-crypto v2.0.1+incompatible
go: finding github.com/libp2p/go-libp2p-interface-pnet v3.0.0+incompatible
go: finding github.com/jackpal/go-nat-pmp v1.0.1
go: github.com/libp2p/go-flow-metrics@v0.2.0: unknown revision v0.2.0
go: finding github.com/libp2p/go-sockaddr v1.0.3
go: github.com/libp2p/go-libp2p-loggables@v1.1.24: unknown revision v1.1.24
go: finding github.com/ipfs/go-datastore v3.2.0+incompatible
go: github.com/libp2p/go-buffer-pool@v0.1.1: unknown revision v0.1.1
go: github.com/libp2p/go-libp2p-metrics@v2.1.7+incompatible: unknown revision v2.1.7
go: github.com/libp2p/go-libp2p-crypto@v2.0.1+incompatible: unknown revision v2.0.1
go: finding github.com/libp2p/go-libp2p-secio v2.0.17+incompatible
go: finding github.com/ipfs/go-cid v0.9.0
go: github.com/libp2p/go-libp2p-host@v3.0.15+incompatible: unknown revision v3.0.15
go: finding github.com/libp2p/go-libp2p-protocol v1.0.0
go: finding github.com/huin/goupnp v0.0.0-20180415215157-1395d1447324
go: github.com/ipfs/go-log@v1.5.7: unknown revision v1.5.7
go: finding github.com/libp2p/go-libp2p-interface-connmgr v0.0.21
go: github.com/libp2p/go-libp2p-interface-pnet@v3.0.0+incompatible: unknown revision v3.0.0
go: finding github.com/libp2p/go-libp2p-nat v0.8.8
go: github.com/libp2p/go-libp2p@v6.0.2+incompatible: unknown revision v6.0.2
go: finding github.com/libp2p/go-reuseport v0.1.18
go: github.com/libp2p/go-sockaddr@v1.0.3: unknown revision v1.0.3
go: finding github.com/libp2p/go-addr-util v2.0.7+incompatible
go: github.com/ipfs/go-datastore@v3.2.0+incompatible: unknown revision v3.2.0
go: finding github.com/ipfs/go-ipfs-util v1.2.8
go: github.com/libp2p/go-libp2p-protocol@v1.0.0: unknown revision v1.0.0
go: finding github.com/libp2p/go-maddr-filter v1.1.10
go: github.com/libp2p/go-libp2p-interface-connmgr@v0.0.21: unknown revision v0.0.21
go: finding github.com/libp2p/go-tcp-transport v2.0.16+incompatible
go: github.com/ipfs/go-cid@v0.9.0: unknown revision v0.9.0
go: finding github.com/libp2p/go-libp2p-peerstore v1.5.0
go: github.com/libp2p/go-libp2p-secio@v2.0.17+incompatible: unknown revision v2.0.17
go: finding github.com/multiformats/go-multibase v0.3.0
go: github.com/libp2p/go-libp2p-nat@v0.8.8: unknown revision v0.8.8
go: finding github.com/multiformats/go-multihash v1.0.8
go: github.com/libp2p/go-reuseport@v0.1.18: unknown revision v0.1.18
go: finding github.com/libp2p/go-reuseport-transport v0.1.11
go: finding github.com/multiformats/go-multistream v0.3.9
go: github.com/libp2p/go-addr-util@v2.0.7+incompatible: unknown revision v2.0.7
go: github.com/ipfs/go-ipfs-util@v1.2.8: unknown revision v1.2.8
go: finding github.com/libp2p/go-libp2p-circuit v2.2.2+incompatible
go: github.com/libp2p/go-tcp-transport@v2.0.16+incompatible: unknown revision v2.0.16
go: github.com/libp2p/go-maddr-filter@v1.1.10: unknown revision v1.1.10
go: finding github.com/libp2p/go-conn-security v0.1.15
go: finding github.com/jackpal/gateway v1.0.4
go: github.com/libp2p/go-libp2p-peerstore@v1.5.0: unknown revision v1.5.0
go: github.com/multiformats/go-multibase@v0.3.0: unknown revision v0.3.0
go: finding github.com/libp2p/go-mplex v0.2.30
go: github.com/multiformats/go-multihash@v1.0.8: unknown revision v1.0.8
go: finding github.com/libp2p/go-stream-muxer v3.0.1+incompatible
go: finding golang.org/x/text v0.3.0
go: finding golang.org/x/sys v0.0.0-20180926160741-c2ed4eda69e7
go: finding github.com/libp2p/go-conn-security-multistream v0.1.15
go: github.com/multiformats/go-multistream@v0.3.9: unknown revision v0.3.9
go: finding github.com/libp2p/go-libp2p-routing v2.7.1+incompatible
go: github.com/libp2p/go-reuseport-transport@v0.1.11: unknown revision v0.1.11
go: finding golang.org/x/net v0.0.0-20180524181706-dfa909b99c79
go: finding github.com/libp2p/go-libp2p-transport-upgrader v0.1.16
go: github.com/libp2p/go-libp2p-circuit@v2.2.2+incompatible: unknown revision v2.2.2
go: finding github.com/multiformats/go-multiaddr v1.2.7
go: github.com/libp2p/go-conn-security@v0.1.15: unknown revision v0.1.15
go: finding github.com/multiformats/go-multiaddr-dns v0.2.3
go: github.com/libp2p/go-mplex@v0.2.30: unknown revision v0.2.30
go: finding github.com/libp2p/go-libp2p-net v3.0.2+incompatible
go: github.com/libp2p/go-stream-muxer@v3.0.1+incompatible: unknown revision v3.0.1
go: finding github.com/libp2p/go-ws-transport v2.0.15+incompatible
go: github.com/libp2p/go-conn-security-multistream@v0.1.15: unknown revision v0.1.15
go: finding github.com/libp2p/go-msgio v0.0.6
go: finding github.com/libp2p/go-libp2p-transport v3.0.15+incompatible
go: github.com/libp2p/go-libp2p-routing@v2.7.1+incompatible: unknown revision v2.7.1
go: finding github.com/libp2p/go-libp2p-peer v2.4.0+incompatible
go: github.com/libp2p/go-libp2p-transport-upgrader@v0.1.16: unknown revision v0.1.16
go: github.com/multiformats/go-multiaddr@v1.2.7: unknown revision v1.2.7
go: github.com/multiformats/go-multiaddr-dns@v0.2.3: unknown revision v0.2.3
go: github.com/libp2p/go-ws-transport@v2.0.15+incompatible: unknown revision v2.0.15
go: github.com/libp2p/go-libp2p-net@v3.0.2+incompatible: unknown revision v3.0.2
go: github.com/libp2p/go-msgio@v0.0.6: unknown revision v0.0.6
go: github.com/libp2p/go-libp2p-transport@v3.0.15+incompatible: unknown revision v3.0.15
go: github.com/libp2p/go-libp2p-peer@v2.4.0+incompatible: unknown revision v2.4.0
go: error loading module requirements
oskarth commented 5 years ago

Still unable to build

> make clean
make: *** No rule to make target 'clean'.  Stop.
> make build
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
# github.com/status-im/status-console-client/vendor/github.com/ethereum/go-ethereum/crypto/secp256k1
vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/curve.go:42:10: fatal error: libsecp256k1/include/secp256k1.h: No such file or directory
 #include "libsecp256k1/include/secp256k1.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/status-im/status-console-client/vendor/github.com/karalabe/hid
vendor/github.com/karalabe/hid/hid_enabled.go:23:11: fatal error: os/threads_posix.c: No such file or directory
  #include "os/threads_posix.c"
           ^~~~~~~~~~~~~~~~~~~~
compilation terminated.

# github.com/status-im/status-console-client/vendor/gopkg.in/olebedev/go-duktape.v3
duk_logging.c: In function ‘duk__logger_prototype_log_shared’:
duk_logging.c:184:64: warning: ‘Z’ directive writing 1 byte into a region of size between 0 and 9 [-Wformat-overflow=]
  sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
                                                                ^
duk_logging.c:184:2: note: ‘sprintf’ output between 25 and 85 bytes into a destination of size 32
  sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.year, (int) comp.month + 1, (int) comp.day,
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.hours, (int) comp.minutes, (int) comp.seconds,
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.milliseconds);
          ~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:7: build] Error 2
oskarth commented 5 years ago

^ @mandrigin @dshulyak @adambabik do you know?

Would be great if we can get this merged as soon as possible

dshulyak commented 5 years ago

@oskarth use make vendor for dependencies. go mod vendor doesn't preserve non-go files, so it needs a bit of help. basically this error suggests that go-ethereum got updated/pulled by go mod and all c-deps were removed

dshulyak commented 5 years ago

not sure what this one is about

# github.com/status-im/status-console-client/vendor/gopkg.in/olebedev/go-duktape.v3
duk_logging.c: In function ‘duk__logger_prototype_log_shared’:
duk_logging.c:184:64: warning: ‘Z’ directive writing 1 byte into a region of size between 0 and 9 [-Wformat-overflow=]
  sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
                                                                ^
duk_logging.c:184:2: note: ‘sprintf’ output between 25 and 85 bytes into a destination of size 32
  sprintf((char *) date_buf, "%04d-%02d-%02dT%02d:%02d:%02d.%03dZ",
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.year, (int) comp.month + 1, (int) comp.day,
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.hours, (int) comp.minutes, (int) comp.seconds,
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          (int) comp.milliseconds);
          ~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [Makefile:7: build] Error 2
oskarth commented 5 years ago 
> make vendor
go mod vendor
go: modules disabled inside GOPATH/src by GO111MODULE=auto; see 'go help modules'
make: *** [Makefile:32: vendor] Error 1

Sorry I'm not too familiar with the go module system, but this is what I see

oskarth commented 5 years ago 
> GO111MODULE=on make vendor
go mod vendor
$GOPATH/go.mod exists but should not
make: *** [Makefile:32: vendor] Error 1
oskarth commented 5 years ago

For the ethereum error this workaround also worked:

cp -r \
  "${GOPATH}/src/github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1" \
  "vendor/github.com/ethereum/go-ethereum/crypto/secp256k1/"

But didn't figure out the equivalent for other error. Btw, live discussion in #status-protocol

decanus commented 5 years ago

@oskarth go modules should be fixed now, could you please confirm?

oskarth commented 5 years ago

@decanus Yay, yes make build builds successfully now.

oskarth commented 5 years ago

I also get the

> ./bin/status-term-client -create-key-pair
Your private key: 0x06e4f3572a0b47d2fc333f6172386474dbb4701031ea004aca451d6944bd35a7
> ./bin/status-term-client -keyhex=0x06e4f3572a0b47d2fc333f6172386474dbb4701031ea004aca451d6944bd35a7
file is encrypted or is not a database
>

that you get, IIRC

dshulyak commented 5 years ago

hey, sorry i will have to merge status-go update cause it solves certain issues in the client. i wanted to wait till this PR gets merged, but it doesn't seem that it will be merged today

decanus commented 5 years ago

@dshulyak I merged master, now I am running into the following error 

build github.com/status-im/status-console-client: cannot load github.com/ethereum/go-ethereum/crypto/sha3: cannot find module providing package github.com/ethereum/go-ethereum/crypto/sha3

Any ideas?

dshulyak commented 5 years ago

it looks as if you were using old status-go version that tries to import that package. can you check that you have v0.25.0-beta.2 in your update?

decanus commented 5 years ago

@dshulyak I always have issues updating the local packages found in vendor? How do I do this?

dshulyak commented 5 years ago

so all versions described in go.mod file. i noticed several issues in your go.mod. status-go points to 0.16.4 which is very old. go-ethereum points to 1.8.27 and doesn't use our fork.

i suggest to use go.mod from master and update it with mvds dependency. i experimented with it and i am able to build your branch. so here is a go.mod file https://github.com/status-im/status-console-client/blob/datasync-mod/go.mod

additionally i had to use go-ethreum 1.8.23 version in mvds, otherwise it conflicts with version in console client (hence replace directive in my go.mod, you won't need it)

to update a single dep i use go get or i put it to go.mod file manually. for example with mvds i just added a line with mvds dep and replace directive for it. then make vendor worked without issues

oskarth commented 5 years ago 
GOFLAGS="-mod=vendor" go build -o ./bin/status-term-client .
build github.com/status-im/status-console-client: cannot load github.com/ethereum/go-ethereum/crypto/sha3: open /home/oskarth/go/src/github.com/status-im/status-console-client/vendor/github.com/ethereum/go-ethereum/crypto/sha3: no such file or directory
make: *** [Makefile:7: build] Error 1
oskarth commented 5 years ago
  1. Add go.mod above
  2. make vendor
  3. make build

^ works

Can we do make vendor in make build or does this not make sense?

@decanus do you want to push branch to status-im so we can push patches to this PR?

decanus commented 5 years ago

@oskarth I can't because I don't have permissions on the status repo

Also, when following your steps I get the following error:

go mod vendor
go: import "github.com/status-im/status-console-client" ->
    import "github.com/status-im/status-go/node" ->
    import "github.com/status-im/status-go/services/personal" ->
    import "github.com/ethereum/go-ethereum/ethapi": looping trying to add package
make: *** [vendor] Error 1
oskarth commented 5 years ago

@decanus Oops, added protocol team now, should work

oskarth commented 5 years ago

When running make vendor

Latest with existing go.mod file:

go: import "github.com/status-im/status-console-client" ->
    import "github.com/status-im/status-go/node" ->
    import "github.com/status-im/status-go/services/personal" ->
    import "github.com/ethereum/go-ethereum/ethapi": looping trying to add package
make: *** [Makefile:32: vendor] Error 1

Latest with @dshulyak go.mod file:

# github.com/status-im/status-console-client
./main.go:356:20: cannot use t (type *adapters.DataSyncWhisperTransport) as type mvds.Transport in argument to mvds.NewNode:
    *adapters.DataSyncWhisperTransport does not implement mvds.Transport (wrong type for Watch method)
        have Watch() mvds.Packet
        want Watch() *mvds.Packet
make: *** [Makefile:7: build] Error 2
oskarth commented 5 years ago

@decanus wdyt about merging partial progress? as long as other things don't break, then can fix bugs etc in future PRs

decanus commented 5 years ago

@oskarth I'd tend to agree, let's have @mandrigin throw another look at it along with @adambabik

oskarth commented 5 years ago

Ping @adambabik @mandrigin

Unless this breaks master and it is under flag I'd rather we merge this and continue with smaller incremental fixes

adambabik commented 5 years ago

@oskarth @decanus I need some introduction to understand how we want this to work.

I understand that data sync algorithm is implemented in https://github.com/status-im/mvds. Now, it uses Whisper to subscribe for messages. It needs already decoded message and inspects messages' payload to figure out what type of a message is received and what to do with it. Is this correct?

If that's the correct description, we may want to adjust the adapter a bit then. The problem is that currently in the adapter we decode message's payload using transit format. In the case of data sync layer, that's incorrect, right? Payload is not transit format but mvds format.

Generally, implementing this, you should not write a new transport unless you want something different than Whisper. Instead, there should be a mechanism to inject mvds which would handle the message, store mvds details somewhere (as it needs to be used later I believe) and return protocol.Message.

decanus commented 5 years ago

@adambabik I would agree that we shouldn't have to write a new transport for this, how best would we handle it?

decanus commented 5 years ago

@adambabik if we do refactor, I'd prefer to merge this as is and then do the rest in a different PR, otherwise this will start getting pretty large.

adambabik commented 5 years ago

@decanus I will suggest merging it to a long-lived branch like data-sync instead of master. I think it's crucial we have this transport thing figured out, however, I also agree there is no point in keeping this PR for longer.

I will adjust the code so it's possible to specify a custom decoding function instead of using transit decoder by default. That should allow us to refactor the code, remove the new transport and merge to master.

Does it sound good?

decanus commented 5 years ago

@adambabik sounds good, maybe this should also fall into refactoring the transports so PFS is removed from them and put at a different layer in the code. Also I've created the branch data-sync, lets merge this in?

decanus commented 5 years ago

We will need to think about the Sync state and message storage, so we ensure we put messages somewhere where they are presisted if the application is closed. This is important for the sync state too, so we continue retransmitting once repopened, for this to work we should probably pass the start epoch to a node.

oskarth commented 5 years ago

I will adjust the code so it's possible to specify a custom decoding function instead of using transit decoder by default. That should allow us to refactor the code, remove the new transport and merge to master.

@adambabik is there an issue for this?

adambabik commented 5 years ago

There is a PR for this: https://github.com/status-im/status-console-client/pull/81

decanus commented 5 years ago

We will need to think about the Sync state and message storage, so we ensure we put messages somewhere where they are presisted if the application is closed. This is important for the sync state too, so we continue retransmitting once repopened, for this to work we should probably pass the start epoch to a node.

This will be possible with: https://github.com/status-im/mvds/pull/44 & https://github.com/status-im/mvds/pull/43

decanus commented 5 years ago

@adambabik @dshulyak I removed protocol buffers from the MVDS repo so it fails, how should I handle this best? put it in the console client make file or readd it to the github of MVDS?

dshulyak commented 5 years ago

you should have golang bindings generated from .proto file in your repo