communicate via websocket

[Mschnuff]

Work in Progress

cleaned up version of my attempts to communicate with the bloxberg network via websocket. this code could actually be readable. This also uses latest master branch as base. Sending a request triggers the listener to send a message (not for all requests). In the official documentation they use a different approach which did not work out for me:

private String getClientversion(WebSocketService webSocketService) throws ExecutionException, InterruptedException {
        System.out.println("enter method");
         Request<?, Web3ClientVersion> request = new Request<>(
                // Name of an RPC method to call
                "web3_clientVersion",
                // Parameters for the method. "web3_clientVersion" does not expect any
                Collections.<String>emptyList(),
                // Service that is used to send a request
                webSocketService,
                // Type of an RPC call to get an Ethereum client version
                Web3ClientVersion.class);
        System.out.println("request created");

// Send an asynchronous request via WebSocket protocol
        System.out.println("send request via websocket: " + request.toString());
         CompletableFuture<Web3ClientVersion> reply = webSocketService.sendAsync(
                request,
                Web3ClientVersion.class);
        System.out.println("request sent " + request.toString());

// Get result of the reply

        System.out.println("getting reply " + reply.toString() );
         return reply.get().getWeb3ClientVersion();

    }

This produces Execution exceptions (due to Timeout) when implemented into the program.

[Mschnuff]

ok. so inserting this into my test data: web3jwss.ethGetBlockByNumber(DefaultBlockParameter.valueOf(BigInteger.valueOf(33)), true) .sendAsync(); gives me a reply message by the network with all of the information on transactions, that we had before but as a String in json format. This string could be parsed or just printed out in its raw form. The resulting string is huge, because it contains a lot of raw data, so I will not post it here. edit: this gives smaller results: web3jwss.ethGetBlockByNumber(DefaultBlockParameter.valueOf(BigInteger.valueOf(33)), false) .sendAsync();

edit#2: added this with new commit. also outsources the websocketlistener into a separate class.

[kiview]

This is interesting but confusing, so this does not return the same objects as if an HTTP client is used? Interesting...

[Mschnuff]

yeah reply is still broken. As far as I can tell I have to use the listener for responses. before we used to do this:

String reply = someRequest.sendAsync().get().getWeb3ClientVersion();
handleStringInSomeWay();

now I do this: someRequest.sendAsync();

And then wait for message from Listener:

14:50:10.287 [WebSocketConnectReadThread-28] DEBUG org.web3j.protocol.websocket.WebSocketClient - Received message {"jsonrpc":"2.0","result":"Parity-Ethereum//v2.6.6-beta-5162bc2-20191205/x86_64-linux-gnu/rustc1.39.0","id":4} from server wss://websockets.bloxberg.org/

[Mschnuff]

this does not work on my system. will continue using default port for now.

URI uriWithoutPort = new URI (wssBloxberg);
URI uriWithPort = URIBuilder.fromURI(uriWithoutPort).withPort(8545).toURI();
webSocketClient = new WebSocketClient(uriWithPort);

[kiview]

This is not about your system, but about which port bloxberg has an open wss connection. So in this case it seems, it is 443, default https port, which is fine.

[Mschnuff]

100 first blocks via websocket

timestamp before first request: 2020-06-10 13:38:46.479
timestamp after latest reply: 2020-06-10 13:38:47.164

ok for 1000 blocks via http and websocket:

timestamps_wss_1: 2020-06-10 14:12:49.009
timestamps_wss_2: 2020-06-10 14:12:57.46
timestamps_http_1: 2020-06-10 14:12:51.717
timestamps_http_2: 2020-06-10 14:13:19.835

[kiview]

This looks pretty fast, or? Do you have the values for HTTP request as well?

[ghost]

We are reviewing the code with @moekappels, and: 1) With 1000 Blocks the timestamps are:

timestamps_wss_1: 2020-06-10 14:12:49.009
timestamps_wss_2: 2020-06-10 14:12:57.46
timestamps_http_1: 2020-06-10 14:12:51.717
timestamps_http_2: 2020-06-10 14:13:19.835

2) With 10000 Blocks:

timestamps_wss_1: 2020-06-10 15:15:37.438
timestamps_wss_2: 2020-06-10 15:15:43.341
timestamps_http_1: 2020-06-10 15:15:38.45
timestamps_http_2: 2020-06-10 15:20:28.272

[kiview]

If this performance change is measured correctly, this is awesome. We should definitely pursue this then.

A note, for simple timestamp you can use System.currentTimeMillis(), which gives a long, no need to construct any Date objects.

Just looking at this code, where do you interact with the Websocket response? Or does web3j work with it internally and cache it, so getEthBlock will hit an internal cache?

        for (int i = 0; i < 1000; i++) {
            // this gives us the whole transaction data we used to have as a String
            CompletableFuture future = web3jwss.ethGetBlockByNumber(DefaultBlockParameter.valueOf(BigInteger.valueOf(i)), false)
                    .sendAsync();
        }
        Date date2 = new Date();
        Timestamp ts2 = new Timestamp(date2.getTime());
        for(int j = 0; j < 1000; j++) {
            try {
                getEthBlock(BigInteger.valueOf(j));
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

[kiview]

No wait guys, what are you doing there? The second loops represents the HTTP interactions, correct? Then in the first loop, you just send messages over the Websocket without waiting for the response (fire and forget), you never resolve your Future!

You have to call get() on a Future at one point and resolve to actual data, a little bit more diligence, please.

[Mschnuff]

In the WebSocketListener I can use Jackson to convert a String into an Java-Object (ethblock in this case). The WebSocketListener could then put the ethblock into a collection or something. The problem is, it is not obvious if there are additional replies comming at a given point in time.

From WebSocketListenerMessageCatcher:

public void onMessage(String message) {
        currentMessage = message;
        try {            
            ethBlock = objectMapper.readValue(currentMessage, EthBlock.class);
            System.out.println("blocknumber: " + ethBlock.getBlock().getNumber().longValue());          

        } catch (JsonProcessingException e) {
            e.printStackTrace();
        }

    }

[kiview]

Yes, but this is happening on a different thread (probably?), but maybe for now just increase a counter (AtomicInt because of currency with different thread) and check for the counter in the test to do the measurement.

If we want to make good use of the asynchronous nature of listener (once we verified the performance), instead of using a collection normal approach would be to use an event-driven architecture internally as well.

[Mschnuff]

You have to call get() on a Future at one point and resolve to actual data, a little bit more diligence, please.

if i call get() on the 'future' like so:

web3jwss.ethGetBlockByNumber(DefaultBlockParameter.valueOf(BigInteger.valueOf(i)), false) .sendAsync().get(); I get a request timeout on the first block. That s why i am using the listener. The listener creates a timestamp after every response. I then get the latest timestamp from the listener after i am done with all the wss- and https-requests as the very last step.

[kiview]

Sounds good, will check when I am back at the PC.

[kiview]

Alright, checked the code now, and I see what happens there. Basically you have no mechanism in place that makes sure that the WebSocketListenerMessageCatcher received all the expected messages and blocks when you access the last timestamp. I can just be any timestamp to be frankly, so you set up a race condition there.

I will push a change using a CountdownLatch to show how to easily solve this.

Also much easier to just do the stuff you are doing in getSomeWssTestData() in one of the test classes (e.g. BloxbergClientTests), to have stuff at least a bit cleaner.

[kiview]

Alright, so WS is indeed so incredibly fast, the race condition was not very likely in the first place! OMG, is WS fast, this is awesome :rocket:

WS connection took 340 ms for 1000 requests
HTTP connection took 29672 ms for 1000 requests

[Mschnuff]

So I pushed a new commit. had lots problems and new error messages that I didnt quite understand. Progress therefore was very slow. I rearranged the architecture of the program a little bit. I feel it is a little bit weird that the the filewriter is now used as an argument for the constructor of the wssClient but it seems that that can't helped. Writing out the files is not yet implemented. the listener gives a single response for the whole batch. this response cannot be displayed in the terminal. the program needs to be run with: mvn package > log-file.log