Improve experimental Nimbus support (sketch)

[oskarth]

Allow Nimbus to be used as the main node for real interactive/scripted chat.

(Blocked by https://github.com/status-im/status-console-client/pull/79 and related to https://github.com/status-im/status-nim/issues/1 - update: no longer blocked)

Acceptance criteria

Negotiable, minimally scoped improvement with follow up issues fine.

Hook up interactive public chats to use Nimbus node for posting messages.

Future enhancements

• Disable geth as well
• More use cases (discovery/add peers/1:1 etc)

[oskarth]

I compiled a rough list of packages/functions/types etc we currently rely on from geth and status-go in status-console-client. Perhaps this is useful in terms of coming up with a plan for getting Nimbus into the app? I suspect a lot of these already exist, and in some cases (util fns) we can do it gradually. Just to get a rough overview. https://gist.github.com/oskarth/bf3a523df8cef353990a4e500a8dde03

[kdeme]

I had a brief chat with @PombeirP and understood that for status-core the preferred way to start is to have a more lower level API. Let's say a direct Whisper API, similar to the existing Whisper RPC API (https://github.com/status-im/nimbus/blob/master/nimbus/rpc/whisper.nim#L8) .

I think the long term goal should be to have a "Status protocol" API in Nimbus, where the lower level work is done behind the scenes (whether that is Whisper now, or something else later). This is more in line with the basic PoC API that is provided now.

I understood the request to start with the lower level API is because the Status API is still quite subject to change. It is fine anyhow, as the lower level work has to be done anyway, and I think the first main goal would be to (stress) test the specific parts of the Nimbus code base, which will also be achieved this way.

This doesn't really matter for the Nimbus API, but I assume integration will then be done with the status-protocol-go repository and not with the status-console-client.

I do have some first questions/remarks regarding the Whisper API.

• Do we want to keep the same way of providing symmetric and asymmetric keys as is done with the RPC version? I'm talking about the system of adding a key and than receiving an ID, which is then used in the post, subscribe, etc. calls. I can't really tell if this is a handy system as I haven't developed on top of it.
• I think as a lower level API I should allow for all options in the calls (like is done in the RPC calls). But if preferred I could limit it (or set some defaults) to specific functionality used by Status
• I can (and probably will) provide both ways of retrieving messages via a registered callback or via polling (e.g getFilterMessages). What is used now or is the preferred way?

[pedropombeiro]

Thanks @kdeme. I'm answering below, however, @adambabik and @cammellos will have the most context as they are currently doing a ton of work on status-protocol-go.

• From what I see in the code, we still use the system to register both symmetric and asymmetric keys.
• I don't see usages of GetFilterMessages in the status-protocol-go project. Maybe the others can confirm this?

[adambabik]

I had a brief chat with @PombeirP and understood that for status-core the preferred way to start is to have a more lower level API. Let's say a direct Whisper API, similar to the existing Whisper RPC API.

I don't quite get "a direct Whisper API" part. Does it mean a higher level of abstraction built on top of Whisper API? If yes, I don't think that's the most useful thing to start with. However, if you think so, https://github.com/status-im/status-protocol-go/blob/master/transport/whisper/whisper_service.go can be considered as a higher level of abstraction.

Generally, the whole Status Protocol consist of few layers but they are tightly connected even with Whisper. In my opinion, the most beneficial would be to implement public chats first to get some intuition and learn about challenges.

Take a look, for example, at https://github.com/status-im/status-protocol-go/blob/90905dec7cba95e2b8c3b904d9eaebaeca57ad59/message_processor.go#L202-L228. This is the easiest type of a message -- a public message without encryption layer.

A little bit more complex are public messages including the encryption layer: https://github.com/status-im/status-protocol-go/blob/90905dec7cba95e2b8c3b904d9eaebaeca57ad59/message_processor.go#L164-L200 (note a call to p.protocol.BuildPublicMessage()).

[adambabik]

I don't see usages of GetFilterMessages in the status-protocol-go project. Maybe the others can confirm this?

We don't use it. Here is the stuff we use: https://github.com/status-im/status-protocol-go/blob/master/transport/whisper/whisper_service.go. We almost entirely rely on filters directly.

[kdeme]

I don't quite get "a direct Whisper API" part. Does it mean a higher level of abstraction built on top of Whisper API? If yes, I don't think that's the most useful thing to start with. However, if you think so, https://github.com/status-im/status-protocol-go/blob/master/transport/whisper/whisper_service.go can be considered as a higher level of abstraction.

With "direct" I meant a Whisper API that gives direct access/control to all or most of the Whisper functionality. Similar like the Whisper v6 RPC API does (but without the RPC part obviously). Post messages with all the options, filters with all the options, and although this is not really purely Whisper functionality, the system of key management.

However, it wouldn't have to be all as it is now in that API, things could be altered for convenience and depending on the usage in Status Protocol. I think this would be similar as the Whisper API that is used now inside https://github.com/status-im/status-protocol-go/blob/0dd1cd585befdc5be7d3a8f8665e94f45f50fc50/transport/whisper/whisper_service.go#L276 , but I would have to check in better detail what exists there.

fyi, I'm experimenting with this here: https://github.com/status-im/nimbus/pull/395 It is still very much testing/playing around with the possibilities to get a feel of what works well (and also trying/learning the go integration). The examples, while incomplete, show two quick versions of a "working" public chat.

Take a look, for example, at https://github.com/status-im/status-protocol-go/blob/90905dec7cba95e2b8c3b904d9eaebaeca57ad59/message_processor.go#L202-L228. This is the easiest type of a message -- a public message without encryption layer.

When you speak of encryption layer you mean the encryption on top of Whisper right? (This?). Not the actual whisper encryption. By the way, I thought that public messages didn't have this layer?

I don't see usages of GetFilterMessages in the status-protocol-go project. Maybe the others can confirm this?

We don't use it. Here is the stuff we use: https://github.com/status-im/status-protocol-go/blob/master/transport/whisper/whisper_service.go. We almost entirely rely on filters directly.

But I see calls there such as RetrieveAllMessages, RetrievePublicMessages, RetrievePrivateMessages which use a Retrieve call. My question here basically is if these calls are used actively to get the messages (basically polling), or if another system is used. In the examples mentioned above, a callback is called by the filter when messages arrive.

[adambabik]

With "direct" I meant a Whisper API that gives direct access/control to all or most of the Whisper functionality. Similar like the Whisper v6 RPC API does (but without the RPC part obviously).

I see. We have this in geth as a "Whisper service" and on top of it the JSON-RPC API is built on. And later as you noticed we added transport/whisper/whisper_service.go to abstract it even more.

When you speak of encryption layer you mean the encryption on top of Whisper right? (This?). Not the actual whisper encryption.

Correct.

By the way, I thought that public messages didn't have this layer?

They currently don't that's why I pointed to https://github.com/status-im/status-protocol-go/blob/90905dec7cba95e2b8c3b904d9eaebaeca57ad59/message_processor.go#L202-L228 first. This should be implemented now if you want to test connectivity with existing clients.

The wrapped version of public messages (https://github.com/status-im/status-protocol-go/blob/90905dec7cba95e2b8c3b904d9eaebaeca57ad59/message_processor.go#L164-L200) also does not encrypt the payload but wraps it and carries sender's public bundle information. I am not 100% sure though whether we gonna turn it on for V1 or not.

My question here basically is if these calls are used actively to get the messages (basically polling), or if another system is used. In the examples mentioned above, a callback is called by the filter when messages arrive.

These methods are supposed to be called in a loop by the caller. It gives a bit more control to the caller but we definitely should expose something like Stream method to avoid that. Still, it would boil down to doing it in a loop behind the scene.

It's a problem with geth Whisper implementation but in Nimbus one it might be possible to avoid polling.