Flask-SocketIO backend unable to recieve message string more than a certain length.

[talalz94]

Description

I am creating a simple application that takes in web cam-stream from an in-browser client side script, converts each frame into a string format and sends it over the backend server consisting of Flask-SocketIO. The issue is that when the string length exceeds a certain length, the messages are not recieved at the backend and Im talking upto 60 kb max string and at a rate of 1 message per 5 seconds, just to test out the connection. Kindly have a look below for further details.

index.html (Client Side)

<div id="container">
   <canvas id="canvasOutput"></canvas>
   <video autoplay="true" id="videoElement"></video>
</div>
<div class = 'video'>
   <img id="image">
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js" integrity="sha512-bLT0Qm9VnAYZDflyKcBaQ2gg0hSYNQrJ8RilYldYQ1FxQYoCLtUjuuRuZo+fjqhx/qtq/1itJ0C2ejDxltZVFg==" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.1.2/socket.io.js" integrity="sha512-iZIBSs+gDyTH0ZhUem9eQ1t4DcEn2B9lHxfRMeGQhyNdSUz+rb+5A3ummX6DQTOIs1XK0gOteOg/LPtSo9VJ+w==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>
<script type="text/javascript" charset="utf-8">
   $(document).ready(function() {

   var socket = io('http://localhost:5000');

   socket.on('connect', function(){
       console.log("Connected...!", socket.connected)
   });

   const video = document.querySelector("#videoElement");

   video.width = 500;
   video.height = 375; ;

   if (navigator.mediaDevices.getUserMedia) {
       navigator.mediaDevices.getUserMedia({ video: true })
       .then(function (stream) {
           video.srcObject = stream;
           video.play();
       })
       .catch(function (err0r) {
           console.log(err0r)
           console.log("Something went wrong!");
       });
   }

   function capture(video, scaleFactor) {
   if(scaleFactor == null){
       scaleFactor = 1;
   }
   var w = video.videoWidth * scaleFactor;
   var h = video.videoHeight * scaleFactor;
   var canvas = document.createElement('canvas');
       canvas.width  = w;
       canvas.height = h;
   var ctx = canvas.getContext('2d');
       ctx.drawImage(video, 0, 0, w, h);
   return canvas;
   }

   const FPS = 30;

   setInterval(() => {

       var type = "image/png";
       var video_element = document.getElementById("videoElement");
       var frame = capture(video_element, 0.25);
       var data = frame.toDataURL(type);

       console.log('image type: ', typeof data);
       console.log('image str len: ', data.length);

       socket.emit('image', data);

   }, 5000);

   socket.on('response_back', function(image){
       const image_id = document.getElementById('image');
       image_id.src = image;

   });
   });
</script>

app.py (Server side)

from flask import Flask, render_template
from flask_socketio import SocketIO, emit
#from flask_cors import CORS
import numpy as np
import imutils
#from io import StringIO
from PIL import Image
import base64
import cv2
import io

app = Flask(__name__)
socketio = SocketIO(app, cors_allowed_origins="*", logger=True)

@app.route('/', methods=['POST', 'GET'])
def index():
    return render_template('index.html')

@socketio.on('image')
def image(data_image):
    sbuf = io.StringIO()
    sbuf.write(data_image)
### decode and convert into image
    b = io.BytesIO(base64.b64decode(data_image))

    pimg = Image.open(b)

    ## converting RGB to BGR, as opencv standards
    sframe = cv2.cvtColor(np.array(pimg), cv2.COLOR_RGB2BGR)
    frame = np.array(pimg)
    # Process the image frame
    frame = imutils.resize(frame, width=700)
    frame = cv2.flip(frame, 1)

    imgencode = cv2.imencode('.jpg', frame)[1]

    # base64 encode
    stringData = base64.b64encode(imgencode).decode('utf-8')
    b64_src = 'data:image/jpg;base64,'
    stringData = b64_src + stringData

    # emit the frame back
    emit('response_back', stringData)

if __name__ == '__main__':
    socketio.run(app, host='127.0.0.1', debug=True)

Server Logs

C:\Python39\python39.exe "C:/Users/Talal Zahid/Desktop/Idenfo/Liveness Detection/webjs_flask_socket/app.py"
 * Serving Flask app 'app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 845-107-437
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [06/Jul/2021 05:53:53] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [06/Jul/2021 05:53:54] "GET /socket.io/?EIO=4&transport=polling&t=NfvJg1O HTTP/1.1" 200 -
127.0.0.1 - - [06/Jul/2021 05:53:54] "POST /socket.io/?EIO=4&transport=polling&t=NfvJg6g&sid=ss6f4I3fOcnsJQY4AAAA HTTP/1.1" 200 -
127.0.0.1 - - [06/Jul/2021 05:53:54] "GET /socket.io/?EIO=4&transport=polling&t=NfvJg6h&sid=ss6f4I3fOcnsJQY4AAAA HTTP/1.1" 200 -
127.0.0.1 - - [06/Jul/2021 05:53:54] "GET /socket.io/?EIO=4&transport=polling&t=NfvJgBX&sid=ss6f4I3fOcnsJQY4AAAA HTTP/1.1" 200 -
message handler error
Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\engineio\server.py", line 603, in _trigger_event
    return self.handlers[event](*args)
  File "C:\Python39\lib\site-packages\socketio\server.py", line 740, in _handle_eio_message
    pkt = packet.Packet(encoded_packet=data)
  File "C:\Python39\lib\site-packages\socketio\packet.py", line 40, in __init__
    self.attachment_count = self.decode(encoded_packet)
  File "C:\Python39\lib\site-packages\socketio\packet.py", line 111, in decode
    self.data = self.json.loads(ep)
  File "C:\Python39\lib\site-packages\engineio\json.py", line 16, in loads
    return original_loads(*args, **kwargs)
  File "C:\Python39\lib\json\__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "C:\Python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python39\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)
127.0.0.1 - - [06/Jul/2021 05:54:03] "GET /socket.io/?EIO=4&transport=websocket&sid=8sxFs9IeZyAmGn1XAAAC HTTP/1.1" 500 -
Traceback (most recent call last):
  File "C:\Python39\Lib\site-packages\flask\app.py", line 2088, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Python39\Lib\site-packages\flask_socketio\__init__.py", line 45, in __call__
    return super(_SocketIOMiddleware, self).__call__(environ,
  File "C:\Python39\Lib\site-packages\engineio\middleware.py", line 60, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "C:\Python39\Lib\site-packages\socketio\server.py", line 573, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "C:\Python39\Lib\site-packages\engineio\server.py", line 392, in handle_request
    packets = socket.handle_get_request(
  File "C:\Python39\Lib\site-packages\engineio\socket.py", line 103, in handle_get_request
    return getattr(self, '_upgrade_' + transport)(environ,
  File "C:\Python39\Lib\site-packages\engineio\socket.py", line 158, in _upgrade_websocket
    return ws(environ, start_response)
  File "C:\Python39\Lib\site-packages\engineio\async_drivers\threading.py", line 23, in __call__
    return self.app(self)
  File "C:\Python39\Lib\site-packages\engineio\socket.py", line 236, in _websocket_handler
    pkt = packet.Packet(encoded_packet=p)
  File "C:\Python39\Lib\site-packages\engineio\packet.py", line 27, in __init__
    self.decode(encoded_packet)
  File "C:\Python39\Lib\site-packages\engineio\packet.py", line 63, in decode
    self.packet_type = int(encoded_packet[0])
ValueError: invalid literal for int() with base 10: 'm'
Exception in thread Thread-17:
Traceback (most recent call last):
  File "C:\Python39\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "C:\Python39\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python39\lib\site-packages\simple_websocket\ws.py", line 90, in _thread
    self.connected = self._handle_events()
  File "C:\Python39\lib\site-packages\simple_websocket\ws.py", line 117, in _handle_events
    self.sock.send(out_data)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine

Client logs

Flask-SocketIO Versions Flask-SocketIO 5.1.0 python-engineio 4.2.0 python-socketio 5.3.0

Additional Context I am running a windows machine and as I have mentioned earlier, the string length seems to be causing the issue as if I send in any string that is greater than x bytes, I get the same errors.

Looking forward to hearing from you soon. Thanks in advance :)

[miguelgrinberg]

Yes, this appears to be an issue in the simple-websocket package. I'll look into it.

[miguelgrinberg]

Can I ask you to install simple-websocket from the main branch and confirm the fix? Thanks!

[talalz94]

@miguelgrinberg Thank you for the response brother. I have a few more questions if you can kindly address them:

1) I was never using simple-websocket in the first place, I was only using flask-socketio. So why are you asking me to use simple-websocket if the issue was in this library?

2) So if I use simple-websocket instead of Flask-socketio, is the client-browser supported like the use case I have mentioned above?

[miguelgrinberg]

I was never using simple-websocket in the first place

I see references to simple-websocket in your error stack traces, so you are most definitely using simple-websocket. This is a package that provides the websocket server to Flask-SocketIO, to you it is an indirect dependency. You don't have to change anything, just upgrade simple-websocket (which you already have installed) to the version that's on the main branch here on GitHub.

[talalz94]

I was never using simple-websocket in the first place

I see references to simple-websocket in your error stack traces, so you are most definitely using simple-websocket. This is a package that provides the websocket server to Flask-SocketIO, to you it is an indirect dependency. You don't have to change anything, just upgrade simple-websocket (which you already have installed) to the version that's on the main branch here on GitHub.

Yes, this has fixed the problem that I was having. Many thanks :)

Just one more thing. If you dont mind commenting on my approach towards building a multi-client app using Flask-SocketIO that essentially takes in web-cam stream from multiple users simultaneoulsy and processes them. Do you think the multi-session functionality would suffice?

[miguelgrinberg]

Do you think the multi-session functionality would suffice?

What do you mean by multi-session?

[talalz94]

Do you think the multi-session functionality would suffice?

What do you mean by multi-session?

Sorry its not exactly multi-session, I meant I want multiple clients to connect to the same server and I just wanted to confirm my existing approach and Flask-SocketIO is compatible with that>

[miguelgrinberg]

@talalz94 it should be fine as long as you don't store anything in the server. It seems you just accept a frame, process it and return the results, without retaining anything, correct? That is safe, as your server is stateless.