SocketIO: Video Streaming from server to client

[Fizmath]

Dear Miguel:

Hello

This is from your video streaming page + emit , server side (i removed the Return):

@app.route('/video_feed')
def video_feed():
       socketio.emit('from_flask', Response(gen(), mimetype='multipart/x-mixed-replace; 
                                                boundary=frame'),namespace='/test')

here the problems are Response and the Generator mixing with the event-driven SocketIO

client

 socket.on('from_flask', function (data) { 
          $("#image").attr("src",data) ;
           });

html

<img id="image" src="video_feed">

Would you please help me ? i know something is wrong here but i cant't figure it

Thanks a million for your great work

[miguelgrinberg]

Generators are a Flask feature, they are not supported for Socket.IO events. If you want to send the video frames over Socket.IO you have to send them one by one as independent events, and then you need to implement the display of these frames in JavaScript for the client.

If you want to take advantage of the Motion JPEG streaming support in browsers, then you have to do it as I do it in my articles, using HTTP.

[Fizmath]

So, i should do something with the generator before feeding into the SocketIO , right ? for example passing it to the Redis then emitting one by one from Redis list ... Any other possible solutions ?

[miguelgrinberg]

You can just treat the generator as an iterable, and then in each iteration emit an event with the frame payload.

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame})

You will also need to change the generator to not include all the boundary stuff, which is not part of the video frame and is only necessary when streaming via HTTP.

[Fizmath]

Oh! greetings ! , thank you very much for your instant response .. I used this

https://github.com/dxue2012/python-webcam-flask

If you open it's demo you will see the mirrored image from the server , but if you or someone else simultaneously opens another browser you will see that the returned image from the other clients got mixed and shuffled , that is because the returned video is not sending through the SocketIO , also look how simply the video has been sent from the browser tho the SERVER through the SocketIO

I am struggling to find a way to send the returned video 1 - through the SocketIO , 2 - individually to the specific client ( you know : request.sid and room = id )

I will be very happy if i'd implent this task

Thank you again for your kind replies though thousand of miles away ....

[miguelgrinberg]

I don't think the example that you pointed out sends video through Socket.IO. This project is heavily based on my articles, and it uses HTTP to stream: https://github.com/dxue2012/python-webcam-flask/blob/master/app.py#L46-L49.

[Fizmath]

Yes , it does not send video from the server to the client throuth Socket.IO , but it sends from the browser to the server through Socket.IO : https://github.com/dxue2012/python-webcam-flask/blob/master/static/js/main.js#L19

am i right ?

[miguelgrinberg]

Yes, it sends a single frame back to the server, encoded as a data url. It is actually very similar to what I mentioned above, just put the jpeg data in an event.

[Fizmath]

OK )) thank you very much and also note that this is an good example showing how your protocol can be implemented on HEROKU .

[Fizmath]

This works fine and machine-guns base64 encoding from the flask server to the browser through SocketIO :

@app.route('/video_feed')
def video_feed():
    for video_frame in gen():
        socketio.emit('from_flask',{'data': video_frame} ,namespace='/test')

The base64 encoded JPEG video_frame starts with :

 /9j/4AAQSkZJRgABAQAAAQABAAD/2.....

the browser receives the encoding by :

  socket.on('from_flask', function (data) {
          $("#imageElement").attr("src","data:image/jpeg;base64,"+data);

it seems to me that something is missing in the above code , further on ,in the HTML :

<img id="imageElement" src="video_feed">

we have the route src="video_feed" and {"src","data:image/jpeg;base64,"+data} , they might conflict. The browser shows errors :

note the streaming frames 698 in seconds ! and [object ] ....

Thank you very much for your attentions

[miguelgrinberg]

I see two problems.

You seem to be mixing two different methods of streaming. If you use the browser support for motion jpeg like I did using HTTP, then you have to use the generator response. If you want to stream over Socket.IO, then you have to create your image without a URL, and then each time your socket.io client receives a frame it can refresh the src attribute.

The second, and most important, problem is that you are not handling the jpeg data correctly. The server sends a payload with the format {data: video_frame}, so on the client, you have to do this:

socket.on('from_flask', function (payload) {
    $("#imageElement").attr("src","data:image/jpeg;base64,"+payload['data']);
}

[Fizmath]

i describe and update new problem in the next comment :

[miguelgrinberg]

Sorry I don't understand what you want to do. If you want to emit to a single client you can, just set the room argument to the sid value of that client.

[Fizmath]

the client opens his browser , webcam captures his video and sends through Socket.IO to the server , the server handles the incoming images and reflexes the modified images through the SocketIO :

users = []

@socketio.on('connect', namespace='/test')
def test_connect():

    users[:] = []
    users.append(request.sid)
    app.logger.info("client connected")
..

@app.route('/video_feed')
def video_feed():
    for video_frame in gen():
        print(users[0])
        socketio.emit('from_flask',{'data': video_frame} , namespace='/test' , room = users[0] )

every time a new client connects, his webcam sends video , the single element users list updates and this sid get fed into the from_flask event , room = user[0] , but things got messed up : while the first client smoothly sees his reflected video ,suddenly the second client connects and he receives not his reflected images from the server but the reflected images of the first client , and even the reflected images of both clients got mixed as if there were no individual channels for them

[miguelgrinberg]

You need separate background threads to handle the video for each client, and you need a way for clients to authenticate, so that the server knows which video stream to send when the client connects to the video feed route.

[Fizmath]

Thank you very very much . By your advice now i know what to do : i just start learning the subjects background threads and authentication

[Fizmath]

Is it possible to use threaded=True to handle this problem ?

[miguelgrinberg]

The threaded option handles requests in their own threads. This is different, since you have background threads for the video streams, running separately from client requests.

[Fizmath]

in camera class we have thread daemon

class Camera(object):
    def __init__(self, makeup_artist):
        self.to_process = []
        self.to_output = []
        self.makeup_artist = makeup_artist

        thread = threading.Thread(target=self.keep_processing, args=())
        thread.daemon = True
        thread.start()

    def process_one(self):
        if not self.to_process:
            # return to exit whole function
            return

        # input is an ascii string. 
        input_str = self.to_process.pop(0)

        # convert it to a pil image
        input_img = base64_to_pil_image(input_str)

        ################## where the hard work is done ############
        # output_img is an PIL image
        output_img = self.makeup_artist.apply_makeup(input_img)

        # output_str is a base64 string in ascii
        output_str = pil_image_to_base64(output_img)

        # convert eh base64 string in ascii to base64 string in _bytes_

        self.to_output.append(output_str)
        #self.to_output.append(binascii.a2b_base64(output_str))

    def keep_processing(self):
        while True:
            self.process_one()
            sleep(0.01)

    def enqueue_input(self, input):
        self.to_process.append(input)

    def get_frame(self):
        while not self.to_output:
            sleep(0.05)
        return self.to_output.pop(0)

you mean that i should separate threads in this class ? or should write threading code in initiation connect event ?

@socketio.on('connect', namespace='/test')
def test_connect():
    app.logger.info("client connected")

i really got perplexed for a few days finding the solution ... thank you very much for you advises .. i am waiting for your guides ....

[miguelgrinberg]

I cannot give you specific details, as I don't know the requirements of your application. But based on what you told me so far, you need a separate Camera instance, with its corresponding background thread, for each client.

[Fizmath]

OK , thank you . I'll try . Let's assume that the Camera class defined in the above comment receives frames from a client , the thread.daemon starts and then suddenly second client sends his frames , where and how should i code the background thread to separate the processing ?

[miguelgrinberg]

You are asking me to provide a solution that I never intended to implement myself, so I really don't know. There are many different ways to implement multiple streams, I would need to know the particulars of your application. Without knowing details, I expect you would have a different Camera instance for each client, but how to implement that depends on your needs.

[Fizmath]

Here is a fundamental question :
Assuming that we have imported theCamera module and make a single instance of it

from sys import stdout
from makeup_artist import Makeup_artist
import logging
from flask import Flask, render_template, Response , request, session
from flask_socketio import SocketIO, emit
from camera import Camera
from utils import base64_to_pil_image, pil_image_to_base64

app = Flask(__name__)
app.logger.addHandler(logging.StreamHandler(stdout))
app.config['SECRET_KEY'] = 'secret!'
app.config['DEBUG'] = True
socketio = SocketIO(app)

users = []
camera = Camera(Makeup_artist())

@socketio.on('input image', namespace='/test')
def test_message(input):
    input = input.split(",")[1]
     camera.enqueue_input(input)

@socketio.on('connect', namespace='/test')
def test_connect():

    users..append(request.sid)
    print(users)
    app.logger.info("client connected")

@app.route('/')
def index():
    """Video streaming home page."""
    return render_template('index.html')

def gen():
    app.logger.info("starting to generate frames!")
    while True:
        frame = camera.get_frame() #pil_image_to_base64(camera.get_frame())
        yield frame

        # yield (b'--frame\r\n'
        #        b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
    """Video streaming route. Put this in the src attribute of an img tag."""
    for video_frame in gen():
        socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test' )

if __name__ == '__main__':
    socketio.run(app)

the global camera instance appears twice : inside input image event and inside gen() method . Once the server runs the camera get uploaded just a single time no matter how many clients connect ,any idea how we can create new instances of Cameraevery time a client connects ?

[miguelgrinberg]

This is what I suggested a few days ago:

You need separate background threads to handle the video for each client, and you need a way for clients to authenticate, so that the server knows which video stream to send when the client connects to the video feed route.

Once you know who each user is, you can maintain a camera object for each, or if you prefer, use a single camera object with one background thread per client.

[Fizmath]

Can i do this ?:

...
from makeup_artist import Makeup_artist
from camera import Camera
.
.

@socketio.on('connect', namespace='/test')
def test_connect():
    session['instance'] = Camera(Makeup_artist())     # pass instance of Camera to flask session
    app.logger.info("client connected")

@socketio.on('input image', namespace='/test')   
def test_message(input):
    input = input.split(",")[1]
    session['instance'].enqueue_input(input)             #  then use that
.
.

is it legal ?

[miguelgrinberg]

Yes, nothing wrong with that.

[Fizmath]

Yes, it works fine inside the socketio.on and gives camera object but inside app.route gives an error:

# here put the socketio  code from the above comment 

@app.route('/')
def index():
    """Video streaming home page."""
    return render_template('index.html')

def gen():
    """Video streaming generator function."""
    app.logger.info("starting to generate frames!")
    while True:
        frame = session['instance'].get_frame() # gives error , see below
        yield frame

[miguelgrinberg]

Did you read the limitations with the session? Changes made in a socket.io handler are not visible in HTTP handlers. See https://flask-socketio.readthedocs.io/en/latest/#access-to-flask-s-context-globals for details.

[Fizmath]

I delayed to find the answer and the issue closed , would you please kindly give me some guidance :

Here is a fundamental question : Assuming that we have imported theCamera module and make a single instance of it

from sys import stdout
from makeup_artist import Makeup_artist
import logging
from flask import Flask, render_template, Response , request, session
from flask_socketio import SocketIO, emit
from camera import Camera
from utils import base64_to_pil_image, pil_image_to_base64

app = Flask(__name__)
app.logger.addHandler(logging.StreamHandler(stdout))
app.config['SECRET_KEY'] = 'secret!'
app.config['DEBUG'] = True
socketio = SocketIO(app)

users = []
camera = Camera(Makeup_artist())

@socketio.on('input image', namespace='/test')
def test_message(input):
    input = input.split(",")[1]
     camera.enqueue_input(input)

@socketio.on('connect', namespace='/test')
def test_connect():

    users..append(request.sid)
    print(users)
    app.logger.info("client connected")

@app.route('/')
def index():
    """Video streaming home page."""
    return render_template('index.html')

def gen():
    app.logger.info("starting to generate frames!")
    while True:
        frame = camera.get_frame() #pil_image_to_base64(camera.get_frame())
        yield frame

        # yield (b'--frame\r\n'
        #        b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed')
def video_feed():
    """Video streaming route. Put this in the src attribute of an img tag."""
    for video_frame in gen():
        socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test' )

if __name__ == '__main__':
    socketio.run(app)

the global camera instance appears twice : inside input image event and inside gen() method . Once the server runs the camera get uploaded just a single time no matter how many clients connect ,any idea how we can create new instances of Cameraevery time a client connects ?

1 -the camera instance is imported once , how can i create a new instance for every new client , knowing his sid from inside socketio ? 2 - the thread starts inside camera class: https://github.com/miguelgrinberg/Flask-SocketIO/issues/778#issuecomment-422047892

how can i create a new thread for a newly connected client?

Thank you very much for your kind remarks

[miguelgrinberg]

This is really a situation that I have never thought about in detail, so it is hard for me to give you advice. My suggestion was that you add authentication, so that you know who each user is. This is necessary so that you can recognize the same user when it contacts the server via socket.io and HTTP. Without authentication you have no way to match requests and socket events as coming from the same user.

Once you have that, one way to keep track of all the camera instances is to use a global dictionary:

cameras = {}

# to add a camera
cameras[username] = Camera()

Sorry I can't be more specific. You keep asking me about a specific solution, but unfortunately I have no code to show you that does what you want.

[Fizmath]

Thank you very much , i tried the above method but the desired result not achieved
I am racking my brains out of tune solving this problem , i will inform you as soon as a get the solution

[Fizmath]

Here is a general question : first i post a session value :

cameras = {}

@app.route('/session', methods=['POST'])
def session_access():
    global camera
    if 'session' in data:
        session['value'] = data['session']
        username = session.get('value', '')
        cameras[username] = Camera()
    return '', 204

after the above Post i want the below route to be uploaded ! because i first set the camera object in the above code . should i use redirect? should i use Login Required Decorator http://flask.pocoo.org/docs/1.0/patterns/viewdecorators/ ?

@app.route('/video_feed')
def video_feed():
    global camera
    for video_frame in gen(cameras[username]):
        socketio.emit('from_flask',{'data': video_frame.decode()} , namespace='/test', room = sidd )

[miguelgrinberg]

I don't understand what you are trying to do, sorry. Your first function references a data variable that is undefined. Your second function references username which is also undefined. I'm really confused, does this code run at all?

I don't understand your question either. The /video_feed route is likely going to be invoked from a <img> tag in a web page, right? So you will never redirect to it.

[Fizmath]

so , any idea ? i am still stuck in this quagmire

[miguelgrinberg]

@Fizmath I don't know, sorry. I don't understand your code. I tried to explain what you need to do to the best of my ability. As I said before, I don't have sample code to show you, I have never implemented what you need.

[Fizmath]

Ok. thanks

[LazyerIJ]

hello. did u solve this problem? I'm trying perfectly same thing but i can't know how to do it. thanks.

[thisisashukla]

Hi @Fizmath did you get good enough latency from socketio.emit method for video streaming. i am doing the same but getting high latency. Link to my question https://stackoverflow.com/questions/57088967/how-to-speed-up-flask-socketio-emit/57207738#57207738

[thisisashukla]

You can just treat the generator as an iterable, and then in each iteration emit an event with the frame payload.

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame})

You will also need to change the generator to not include all the boundary stuff, which is not part of the video frame and is only necessary when streaming via HTTP.

This method gave me poor FPS while streaming a recorded video to browser from flask server.

[rajdip34]

Hey anyone solves this problem ?

[AhmedBhati]

I have just done live video-streaming with flask-socketio, uploaded on the github just check, might help...https://github.com/AhmedBhati/video-streaming-with-flask-socketio

[whikwon]

@thisisashukla I also have high latency when using socketio.emit method did you find the solution?

[AhmedBhati]

@whikwon if you are using eventlet as the server for sending frames put broadcast=True in emit

for video_frame in gen():
    emit('my_video_frame', {'data': video_frame}, broadcast=True)

[whikwon]

@AhmedBhati I can't find the difference with, without broadcast argument.

Is there anything wrong with my code?

app.py

import base64
import time
import cv2
from flask import Flask, render_template, request
from flask_socketio import SocketIO, emit
import eventlet

eventlet.monkey_patch()
app = Flask(__name__)
socketio = SocketIO(app, logger=True, async_mode='eventlet')

@socketio.on('connect', namespace='/web')
def connect_web():
    print('[INFO] Web client connected: {}'.format(request.sid))

@socketio.on('disconnect', namespace='/web')
def disconnect_web():
    print('[INFO] Web client disconnected: {}'.format(request.sid))

@socketio.on('connect', namespace='/local')
def connect_cv():
    print('[INFO] CV client connected: {}'.format(request.sid))

@socketio.on('disconnect', namespace='/local')
def disconnect_cv():
    print('[INFO] CV client disconnected: {}'.format(request.sid))

@app.route('/')
def index():
    return render_template('index.html', async_mode=socketio.async_mode)

@socketio.on('stream_request')
def stream_video(message):
    socketio.emit('stream_response', message, namespace='/web', broadcast=True)
    return 

if __name__ == '__main__':
    socketio.run(app, host="127.0.0.1", port=5000, debug=True)

camera.py

import datetime
from threading import Thread
import base64
import time
import cv2
import socketio

sio = socketio.Client(logger=True)
sio.connect('http://127.0.0.1:5000',
            transports=['websocket'],
            namespaces=['/local'])

@sio.event
def connect():
    print('[INFO] Successfully connected to server')

@sio.event
def connect_error():
    print('[INFO] Failed to connect to server.')

@sio.event
def disconnect():
    print('[INFO] Disconnected from server.')

def encode_image(image):
    image = cv2.imencode('.jpg', image)[1].tobytes()
    image = base64.b64encode(image).decode('utf-8')
    image = f"data:image/jpeg;base64,{image}"
    return image

class FPS:
    def __init__(self):
        # store the start time, end time and total number of frames
        # that were examined between the start and end intervals
        self._start = None
        self._end = None
        self._numFrames = 0

    def start(self):
        # start the timer
        self._start = datetime.datetime.now()
        return self

    def stop(self):
        # stop the timer
        self._end = datetime.datetime.now()

    def update(self):
        # increment the total number of frames examined during the
        # start and end intervals
        self._numFrames += 1

    def elapsed(self):
        # return the total number of seconds between the start and 
        # end interval
        return (self._end - self._start).total_seconds()

    def fps(self):
        # compute the (approximate) frames per second
        return self._numFrames / self.elapsed()

class WebCamVideoStream:
    def __init__(self, src=0):
        # initialize the video camera stream and read the first frame 
        # from the stream
        self.stream = cv2.VideoCapture(src)
        (self.grabbed, self.frame) = self.stream.read()

        # initialize the variable used to inidicate if the thread 
        # should be stopped
        self.stopped = False

    def start(self):
        # start the thread to read frames from the video stream
        Thread(target=self.update, args=()).start()
        return self

    def update(self):
        # keep looping infinitely until the thread is stopped
        while True:
            # if the thread indicator variable is set, stop the thread
            if self.stopped:
                return
            # otherwise read the next frame from the stream
            (self.grabbed, self.frame) = self.stream.read()

    def read(self):
        # return the frame most recently read
        return self.frame

    def stop(self):
        # indicate that the thread should be stopped
        self.stopped = True

def main():
    # SLOW VERSION
    # cap = cv2.VideoCapture(0)
    # cap.set(cv2.CAP_PROP_FRAME_WIDTH, 600)
    # cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 600)
    stream = WebCamVideoStream(src=0).start()
    i = 0
    while True:
        # ret, frame = cap.read()
        frame = stream.read()
        frame = cv2.resize(frame, (600, 600))
        sio.emit('stream_request', {'image': encode_image(frame)})
        k = cv2.waitKey(1) 
        if k == 27: 
            break
        i += 1
        time.sleep(0.05)

    stream.stop()
    # cap.release()
    cv2.destroyAllWindows()

if __name__ == "__main__":
    main()

[AhmedBhati]

@whikwon can you send your index.html so i would run this code up and you can also refer the code written by me on video streaming using flask-socketio

[whikwon]

@AhmedBhati Here's my index.html file.

<html>
<head>
    <title>SocketIO</title>
    <script src="//cdnjs.cloudflare.com/ajax/libs/socket.io/2.2.0/socket.io.js" integrity="sha256-yr4fRk/GU1ehYJPAs8P4JlTgu0Hdsp4ZKrx8bDEDC3I=" crossorigin="anonymous"></script>
    <style>
        .container {
            display: grid;
            grid-template-rows: repeat(2, 1fr);
            grid-template-columns: repeat(2, 1fr);
        }
        .contents {
            background: white;
        }
        .images {
            background: #665c9c;
        }
    </style>
</head>
<body>
    <div class="container">
        <div class="images">
            <img id="streamed-image" src="">
        </div>
        <div class="contents">
            <button id='play-btn'>PLAY</button>
        </div>
    </div>
    <script type="text/javascript" charset="utf-8">
        const namespace = '/web';
        const socket = io(namespace);
        socket.on('stream_response', (msg) => {
            document.querySelector('#streamed-image').src = msg.image;
        });
    </script>
</body>
</html>

[whikwon]

@AhmedBhati I've ran your source code and it is faster than mine. What is the difference between them?

[AhmedBhati]

i have executed your code, @whikwon Firstly reduce the frame size the smaller the better, secondly remove the delay time.sleep(0.05) from the code.

[whikwon]

@AhmedBhati If I remove the delay, lagging are getting more and more. If I use flask Response instead like the code below, I found socketio between javascript and python doesn't work. Is it normal?

def video_generator():
    import imagezmq
    subscriber = imagezmq.ImageHub('tcp://127.0.0.1:5555', REQ_REP=False)
    while True:
        camera_id, frame = subscriber.recv_image()
        frame = cv2.imencode('.jpg', frame)[1].tobytes()
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')

@app.route('/video_feed/', methods=['POST', 'GET'])
def stream_imagezmq():
    return Response(video_generator(), mimetype='multipart/x-mixed-replace; boundary=frame')

[AhmedBhati]

@whikwon yeah but this is based on http protocol not on websockets, I'll send your modified code also. run it once an see

[whikwon]

@AhmedBhati Thank you so much, What is the big difference between http and websocket? Is there impact on performance? or else?