alikureishy / Automata-Capstone

Capstone repository (pulled from /verichev)
MIT License
2 stars 0 forks source link

Socket timeouts consistently causing the server node to die within a few mins of running the simulation #13

Open alikureishy opened 5 years ago

alikureishy commented 5 years ago

127.0.0.1 - - [10/Jul/2019 23:00:25] "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1" 200 0 60.000150 (19109) wsgi exited, is_accepting=True Traceback (most recent call last): File "/data/code/nd/self-driving-car/collab/Automata-Capstone/ros/src/styx/server.py", line 68, in eventlet.wsgi.server(eventlet.listen(('', 4567)), app) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/wsgi.py", line 952, in server client_socket, client_addr = sock.accept() File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/greenio/base.py", line 227, in accept self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/greenio/base.py", line 207, in _trampoline mark_as_closed=self._mark_as_closed) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/hubs/init.py", line 163, in trampoline return hub.switch() File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 295, in switch return self.greenlet.switch() socket.timeout: timed out [styx_server-2] process has died [pid 19109, exit code 1, cmd /data/code/nd/self-driving-car/collab/Automata-Capstone/ros/src/styx/server.py __name:=styx_server __log:=/home/safdar/.ros/log/04bd1b84-a3a1-11e9-b4c2-309c23a3d554/styx_server-2.log]. log file: /home/safdar/.ros/log/04bd1b84-a3a1-11e9-b4c2-309c23a3d554/styx_server-2*.log

alikureishy commented 5 years ago

The last working commit for the 'master' branch was: commit d8659015bc9c208b9a37d9d32785360c04346de9 Author: evgenii.verichev evgenii.verichev@grabtaxi.com Date: Wed Jul 3 00:08:59 2019 +0800 dbw mkz

alikureishy commented 5 years ago

The issue above has crept in during one of these commits below:

commit d03545490635cca0cd9000d3393bff2120c9cc46 Merge: 6f636d6 78cb257 Author: Szilard Bessenyei bessszilard@gmail.com Date: Tue Jul 9 06:23:43 2019 +0200 Merge pull request #6 from safdark/saving_images_from_simulator
Saving images from simulator

commit 78cb257073d1fe07b743c41530702dfde5069afc (origin/saving_images_from_simulator, saving_images_from_simulator) Author: evgenii.verichev evgenii.verichev@grabtaxi.com Date: Mon Jul 8 23:41:26 2019 +0800 saving images from simulator

commit 6f636d6d1b4d363565d203da7e3b1f1e3311e39c Merge: d865901 c4015cc Author: Eugene Verichev eugeneverichev@inbox.ru Date: Mon Jul 8 03:06:12 2019 +0800 Merge pull request #1 from Verichev/adding_classifier Adding classifier

commit c4015cc772bde9c6c0f832b1306a1ca02ad0b6bc Author: evgenii.verichev evgenii.verichev@grabtaxi.com Date: Mon Jul 8 03:01:27 2019 +0800 adding classifier impl

bessszilard commented 5 years ago

Is this error similar? https://github.com/llSourcell/How_to_simulate_a_self_driving_car/issues/34

alikureishy commented 5 years ago

I don't think so, because the timeout does not happen on an older commit ... (with the same environment).

alikureishy commented 5 years ago

Fyi, I merged the PR for this issue (from branch "cleaning_consts_impl_in_detector") into mainline, but the socket timeout is still there:

127.0.0.1 - - [11/Jul/2019 19:33:43] "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1" 200 0 60.000223 (12033) wsgi exited, is_accepting=True Traceback (most recent call last): File "/data/code/nd/self-driving-car/collab/Automata-Capstone/ros/src/styx/server.py", line 68, in eventlet.wsgi.server(eventlet.listen(('', 4567)), app) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/wsgi.py", line 952, in server client_socket, client_addr = sock.accept() File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/greenio/base.py", line 227, in accept self._trampoline(fd, read=True, timeout=self.gettimeout(), timeout_exc=_timeout_exc) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/greenio/base.py", line 207, in _trampoline mark_as_closed=self._mark_as_closed) File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/hubs/init.py", line 163, in trampoline return hub.switch() File "/home/safdar/anaconda3/envs/CarND-Capstone/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 295, in switch return self.greenlet.switch() socket.timeout: timed out

naveedhd commented 5 years ago

I am not getting this error.. is there a way to reproduce it? does it happen on start for you or after some time?

alikureishy commented 5 years ago

It happens after crossing 1-2 traffic lights

On Fri, Jul 12, 2019, 1:28 AM Naveed Usmani notifications@github.com wrote:

I am not getting this error.. is there a way to reproduce it? does it happen on start for you or after some time?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/safdark/Automata-Capstone/issues/13?email_source=notifications&email_token=AAJDKA3B75N4DXSZL7LBH43P7A6DZA5CNFSM4IAWKZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZZDAQQ#issuecomment-510799938, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJDKA4MOILHOZCFICNQUF3P7A6DZANCNFSM4IAWKZDA .

alikureishy commented 5 years ago

This seems to happen because of some issue with the simulator (not the server). I don't think there's anything on the server that is causing this.