search

FreeTAKTeam / FreeTakServer

Situational Awareness Server compatible with TAK clients
Eclipse Public License 2.0
674 stars 170 forks source link

1.9: REST API: postPresence creates - Socket OSError: Too many open files #198

Closed Securitybits-io closed 2 years ago

Securitybits-io commented 3 years ago

Running FTS 1.9 and utilizing the RestAPI postPresence endpoint in a external EUD/Bot eventually creates a DoS condition as the sqlalchemy is unable to open the database due to too many open files.

python3[1574]: Traceback (most recent call last):  
python3[1574]:   File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap  
python3[1574]:     self.run()  
python3[1574]:   File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run  
python3[1574]:     self._target(*self._args, **self._kwargs)
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/RestAPI.py", line 1628, in startup
python3[1574]:     socketio.run(app, host=IP, port=Port)
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/flask_socketio/__init__.py", line 583, in run
python3[1574]:     run_server()
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/flask_socketio/__init__.py", line 577, in run_server
python3[1574]:     eventlet.wsgi.server(eventlet_socket, app,
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 990, in server
python3[1574]:     client_socket, client_addr = sock.accept()
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 224, in accept
python3[1574]:     res = socket_accept(fd)
python3[1574]:   File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 61, in socket_accept
python3[1574]:     return descriptor.accept()
python3[1574]:   File "/usr/lib/python3.8/socket.py", line 292, in accept
python3[1574]:     fd, addr = self._accept()
python3[1574]: OSError: [Errno 24] Too many open files
brothercorvo commented 3 years ago

Change the number of  open files in your OS, reduce the frequency of updates. Sent from my Galaxy Tab® E -------- Original message --------From: Christoffer Claesson @.> Date: 2021-09-11 1:53 PM (GMT-04:00) To: FreeTAKTeam/FreeTakServer @.> Cc: Subscribed @.> Subject: [FreeTAKTeam/FreeTakServer] 1.9: REST API: postPresence creates - Socket OSError: Too many open files (#198) Running FTS 1.9 and utilizing the RestAPI postPresence endpoint in a external EUD/Bot eventually creates a DoS condition as the sqlalchemy is unable to open the database due to too many open files. python3[1574]: Traceback (most recent call last):
python3[1574]: File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
python3[1574]: self.run()
python3[1574]: File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
python3[1574]: self._target(self._args, self._kwargs) python3[1574]: File "/usr/local/lib/python3.8/dist-packages/FreeTAKServer/controllers/services/RestAPI.py", line 1628, in startup python3[1574]: socketio.run(app, host=IP, port=Port) python3[1574]: File "/usr/local/lib/python3.8/dist-packages/flask_socketio/init.py", line 583, in run python3[1574]: run_server() python3[1574]: File "/usr/local/lib/python3.8/dist-packages/flask_socketio/init.py", line 577, in run_server python3[1574]: eventlet.wsgi.server(eventlet_socket, app, python3[1574]: File "/usr/local/lib/python3.8/dist-packages/eventlet/wsgi.py", line 990, in server python3[1574]: client_socket, client_addr = sock.accept() python3[1574]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 224, in accept python3[1574]: res = socket_accept(fd) python3[1574]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenio/base.py", line 61, in socket_accept python3[1574]: return descriptor.accept() python3[1574]: File "/usr/lib/python3.8/socket.py", line 292, in accept python3[1574]: fd, addr = self._accept() python3[1574]: OSError: [Errno 24] Too many open files

—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/FreeTAKTeam/FreeTakServer/issues/198", "url": "https://github.com/FreeTAKTeam/FreeTakServer/issues/198", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Securitybits-io commented 3 years ago

The limit of Openfiles are ~10million, the updates are coming (depending on the amount of active users) 6-10 PUTs each 30 seconds. So it feels more like the handles are not being closed properly?

brothercorvo commented 3 years ago

With the has app we post 5 updates per second and works in 1.9. Try to us3 the post geo object function and see if you observe the sameSent from my Galaxy Tab® E -------- Original message --------From: Christoffer Claesson @.> Date: 2021-09-11 3:42 PM (GMT-04:00) To: FreeTAKTeam/FreeTakServer @.> Cc: Corvo @.>, Comment @.> Subject: Re: [FreeTAKTeam/FreeTakServer] 1.9: REST API: postPresence creates - Socket OSError: Too many open files (#198) The limit of Openfiles are ~10million, the updates are coming (depending on the amount of active users) 6-10 PUTs each 30 seconds. So it feels more like the handles are not being closed properly?

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/FreeTAKTeam/FreeTakServer/issues/198#issuecomment-917454697", "url": "https://github.com/FreeTAKTeam/FreeTakServer/issues/198#issuecomment-917454697", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

naman108 commented 2 years ago

have you managed to replicate on 1.9.1? furthermore after approximately how long does this happen?

Securitybits-io commented 2 years ago

have you managed to replicate on 1.9.1? furthermore after approximately how long does this happen?

Ill see if i can get a couple users to test the issue after approx 1 hour -ish the server died

naman108 commented 2 years ago

ok so I just wrote a test which runs 6000 put requests in a row without failure, this marginally greater than 4 clients at 20 cots per minute. is there another way I can reproduce or could you send me your client so I can use that to reproduce?

brothercorvo commented 2 years ago

closing this for now