search

crossbario / autobahn-js

WAMP in JavaScript for Browsers and NodeJS
http://crossbar.io/autobahn
MIT License
1.43k stars 228 forks source link

Resposes keep getting slower when dealing with large data #543

Closed ondrej-vipo closed 3 years ago

ondrej-vipo commented 3 years ago

I have client that keeps calling function that returns large amount of data. After receiving the data, it prints the time needed to be received (offset). Even though that server is responding with the same data, the time needed for data to be received keeps getting larger.

The problem seems to be only in the browser. It seems to be working fine in the nodejs.

Output

speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507516  with offset  2 , data sent at  1616507514 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507532  with offset  5 , data sent at  1616507527 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507551  with offset  8 , data sent at  1616507543 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507573  with offset  11 , data sent at  1616507562 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507599  with offset  15 , data sent at  1616507584 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507629  with offset  19 , data sent at  1616507610 , size =  5700008
speed.html:22 calling getlargedata ...
speed.html:25 Received at  1616507663  with offset  23 , data sent at  1616507640 , size =  5700008
speed.html:22 calling getlargedata ...

Client:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Autobahn test</title>
        <meta charset="utf-8">
        <script src="autobahn.js"></script> <!-- autobahn-browser@20.9.2 -->
    </head>
    <body>
        <script>
            document.write('Open console!');
            (function(){
                var timer = null;
                var connection = new autobahn.Connection({
                    url: "ws://127.0.0.1:9990",
                    realm: 'realm1'
                }); 

                var getLargeData = function(){
                    timer = setTimeout(function(){
                        console.info('calling getlargedata ...');
                        session.call('com.example.getlargedata', []).then(function(data){
                            var now = Math.trunc((new Date).getTime()/1000);
                            console.info('Received at ', now, ' with offset ', now - data.time, ', data sent at ', data.time, ', size = ', data.data.length);
                            getLargeData(); // lets get large data again in 10s;
                        });
                    }, 10000);
                }

                connection.onclose = function(){
                    console.error('connection closed');
                    session = null;
                    if (timer !== null){
                        clearTimeout(timer);
                        timer = null;
                    }
                }

                connection.onopen = function (sess, details) {
                    console.log("Connected");
                    session = sess;
                    getLargeData();
                }
                connection.open();
            }());
        </script>
    </body>
</html>

Server (part):

    public function __construct()
    {
        parent::__construct("realm1");
        $this->largeData = 'START'.str_repeat('This is test string', 300000).'END';
    }
    public function onSessionStart($session, $transport)
    {
        $session->register('com.example.getlargedata', function($args){
            return ['time' => time(), 'data' => $this->largeData];
        });
    }

This problem is also happening at #542 Please let me know if more info is needed. I can provide full source of server if needed.

oberstet commented 3 years ago

my guess would be this is your app code problem .. not related to autobahn

ondrej-vipo commented 3 years ago

@oberstet I hava managed to replicate the problem also with crossbar.io

Server:

from autobahn.twisted.component import Component, run
from autobahn.twisted.util import sleep
from twisted.internet.defer import inlineCallbacks
import os
import argparse
from time import time 

url = os.environ.get('CBURL', 'ws://localhost:8080/ws')
realmv = os.environ.get('CBREALM', 'realm1')
print(url, realmv)
component = Component(transports=url, realm=realmv)

@component.on_join
@inlineCallbacks
def joined(session, details):
    print("session ready")
    counter = 0
    largedata = "START" + "This is a test string"*300000 + "END";

    def getlargedata():
        print("Sending large data...");
        return {"time":(int)(time()), "data": largedata}

    try:
        yield session.register(getlargedata, 'com.example.getlargedata')
        print("procedure registered")
    except Exception as e:
        print("could not register procedure: {0}".format(e)) 

if __name__ == "__main__":
    run([component])

Client:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Autobahn test</title>
        <meta charset="utf-8">
        <script src="autobahn.js"></script> <!-- autobahn-browser@20.9.2 -->
    </head>
    <body>
        <script>
            document.write('Open console!');
            (function(){
                var timer = null;

                var connection = new autobahn.Connection({
                    url: "ws://localhost:8080/ws",
                    realm: 'realm1'
                }); 

                var getLargeData = function(){
                    timer = setTimeout(function(){
                        console.info('calling getlargedata ...');
                        session.call('com.example.getlargedata', []).then(function(data){
                            var now = Math.trunc((new Date).getTime()/1000);
                            console.info('Received at ', now, ' with offset ', now - data.time, ', data sent at ', data.time, ', size = ', data.data.length);
                            getLargeData(); // lets get large data again in 10s;
                        });
                    }, 10000);
                }

                connection.onclose = function(){
                    console.error('connection closed');
                    session = null;
                    if (timer !== null){
                        clearTimeout(timer);
                        timer = null;
                    }
                }

                connection.onopen = function (sess, details) {
                    console.log("Connected");
                    session = sess;
                    getLargeData();
                }
                connection.open();
            }());
        </script>
    </body>
</html>
oberstet commented 3 years ago

if you receive a WAMP RPC call result, and when there is no WebSocket auto-fragmentation is in place, that will occupy the receiving client thread - eg no pubsub event can be process while the single RPC result is still being received and processed

you can use:

to improve that

ondrej-vipo commented 3 years ago

@oberstet hello,

I have tested both suggeted things and I do not see any overall improvement.

I have tested auto-fragmentation by setting "auto_fragment_size" option in the crossway config and the progressive call by the changing both front-end (with "receive_progress: true" option) and backend (by yelding data chunks in loop with the progress function).

Just to remind this: this issue only occures with autobahn-browser. There is no problem with autobahn-js in nodejs "frontend" (even without progressive calls and auto-fragmentation). And the issue is the same whether I am using crossway or thruway backend. - That's why I am thinking this is a bug in the autobahn-browser.

I am ready to provide more information if needed.

oberstet commented 3 years ago

And the issue is the same whether I am using crossway or thruway backend. - That's why I am thinking this is a bug in the autobahn-browser.

most likely is buggy app code (your code) ;) sorry, I don't have time to debug that.

if you still think there is a bug in the library, we need a new (failing) unit test to demonstrate the issue and its resolution

https://github.com/crossbario/autobahn-js/tree/master/packages/autobahn/test