Python CLI client question

[ptdecker]

I'm only posting this question here because it is unclear where there is a better forum for it. If some other place is better while still avoiding the StackOverflow noise, please let me know and accept my apologies.

I have a situation where we have a Flask server that uses flask-sock for its websocket implementation. I'm trying to build a stand-alone CLI client for it in Python whose purpose, coupled with a server end-point, is to support streaming large files to the server through a websocket.

I thought I had it all implemented, but had one small puff of smoke that turned out to be a fire once I enabled debugging on the server side. I think I learned that the client I was trying to use is a Sockets.IO client and was causing the server to through a connection error at the end of the session. I can send the file just fine, but upon completion I get a 500 connection error on the server. I may have this totally wrong.

I think I pinpointed it to the package I am using on the client (https://pypi.org/project/websockets/).

I've spent so much time on this and at this point am feeling lost being so close to a solution, but yet so far.

Any help would be appreciated. Do I understand the situation? Should I be using a different client? I don't have a choice on the server side.

Below are the server and client apps and the server and client logs when running.

Server:

from flask import Flask
from flask_sock import Sock
import json
import os
import sys

app = Flask(__name__)
app.config['SOCK_SERVER_OPTIONS'] = {'ping_interval': 25}
app.config['SSL_DISABLE'] = True

sock = Sock(app)

@sock.route('/upload')
def upload(ws):
    file_metadata: json = json.loads(ws.receive().decode("utf-8"))
    if 'name' not in file_metadata:
        print(f"ERROR! Metadata does not contain expected 'name' attribute: '{file_metadata}'")
        sys.exit()
    if 'size' not in file_metadata:
        print(f"ERROR! Metadata does not contain expected 'size' attribute: '{file_metadata}'")
        sys.exit()
    print(f"{file_metadata['size']} bytes expected .", end="")
    with open(file_metadata['name'], "wb") as f:
        received_len: int = 0
        while received_len < file_metadata['size']:
            received: bytes = ws.receive()
            received_len += len(received)
            f.write(received)
            print(".", end="")
    print(f". received {received_len}", end="")
    written_to_disk = os.path.getsize(file_metadata['name'])
    print(f", {written_to_disk} written to '{file_metadata['name']}'")

if __name__ == '__main__':
    app.run(debug=True)

Client:

import websockets
import asyncio
import json
import os
import sys

async def send(uri: str, filename: str, buffer_size: int = 4096):
    if not uri:
        print(f"ERROR! 'uri' cannot be empty")
        sys.exit()
    if not filename:
        print(f"ERROR! 'filename' cannot be empty")
        sys.exit()
    print(f"Sending '{filename}'", end="")
    async with websockets.connect(uri) as websocket:
        filesize: int = os.path.getsize(filename)
        metadata: json = {"name": filename, "size": filesize}
        print(f" {filesize} bytes .", end="")
        await websocket.send(json.dumps(metadata).encode("utf-8"))
        with open(filename, "rb") as f:
            while True:
                bytes_read: bytes = f.read(buffer_size)
                if not bytes_read:
                    break
                await websocket.send(bytes_read)
                print('.', end='')
        print(". sent")

if __name__ == '__main__':
    asyncio.run(send('ws://localhost:5000/upload', 'sample_data_roster.csv'))

Server console:

/Users/ptdecker/PycharmProjects/socketServer/venv/bin/python /Users/ptdecker/PycharmProjects/socketServer/main.py
 * Serving Flask app 'main'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 120-408-945
69767 bytes expected .................... received 69767, 69767 written to 'sample_data_roster.csv'
127.0.0.1 - - [16/Aug/2022 17:08:07] "GET /upload HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/ptdecker/PycharmProjects/socketServer/venv/lib/python3.8/site-packages/flask/app.py", line 2548, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/ptdecker/PycharmProjects/socketServer/venv/lib/python3.8/site-packages/flask/app.py", line 2532, in wsgi_app
    return response(environ, start_response)
  File "/Users/ptdecker/PycharmProjects/socketServer/venv/lib/python3.8/site-packages/flask_sock/__init__.py", line 83, in __call__
    raise ConnectionError()
ConnectionError

Client console:

/Users/ptdecker/PycharmProjects/socketClient/venv/bin/python /Users/ptdecker/PycharmProjects/socketClient/main.py
Sending 'sample_data_roster.csv' 69767 bytes .................... sent

Process finished with exit code 0

[miguelgrinberg]

You theory does not really agree with the logs that you are showing.

First of all, there is no Socket.IO here. Your server (Flask-Sock) uses WebSocket, and your client (the websockets package) also uses WebSocket. This is fine.

The ConnectionError exception that you are getting is only used when your web server is the Flask development web server. This is a long story, but to make it short, the Flask development web server has been proven difficult when a route upgrades to WebSocket. Newer releases of Flask and Werkzeug use this ConnectionError exception as an attempt to end the WebSocket call gracefully. On older versions of these packages, the error shows up in the log, but it is still benign, as it just indicates that the connection with the client ended.

So that's it. I really have no indication that there is a problem. When you deploy your application on a production web server, none of this will happen and your WebSocket calls will end cleanly.

[ptdecker]

Ah, very interesting @miguelgrinberg. Thank you! How this came to light is if debug mode is off resulting in that error not being immediately visible on the server side, the connection from my client stays open for a long time (perhaps 20 seconds or so). That problem is what put me on this path when a very capable coworker of mine couldn't figure it out either and didn't see anything wrong on the surface with my code. This side effect went away immediately when I enable debug mode which led me to then try to find documentation on the "connection error" error. That goose chase took me down the Sockets.IO route and since much of this is new to me, I don't know the difference really. So, you're note goes a long way to explaining it. We can live with that delay on the client side if it goes away in a production setting. Thank you for taking the time to respond. We can probably close this question because of it. I appreciate your help.