dustinfast / PTC-Sim

A PTC Back Office Server with web interface, track/locomotive simulators, and EMP messaging subsystem.
MIT License
3 stars 4 forks source link

Heroku deployment unstable #9

Open dustinfast opened 5 years ago

dustinfast commented 5 years ago

Port binding fails with ADDRESS ALREADY IN USE and after a minute or two of runtime, memory usage exceeds 500mb limit (presumably as a result of multiple threads failing/restarting due to port bind issue).

Heroku web log output -

2018-11-23T18:28:09.088733+00:00 app[web.1]: Exception in thread Thread-15:
2018-11-23T18:28:09.088737+00:00 app[web.1]: Traceback (most recent call last):
2018-11-23T18:28:09.088739+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/threading.py", line 801, in __bootstrap_inner
2018-11-23T18:28:09.088741+00:00 app[web.1]:     self.run()
2018-11-23T18:28:09.088743+00:00 app[web.1]:   File "/app/lib_messaging.py", line 353, in run
2018-11-23T18:28:09.088745+00:00 app[web.1]:     sock.bind((BROKER, SEND_PORT))
2018-11-23T18:28:09.088747+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/socket.py", line 228, in meth
2018-11-23T18:28:09.088749+00:00 app[web.1]:     return getattr(self._sock,name)(*args)
2018-11-23T18:28:09.088751+00:00 app[web.1]: error: [Errno 98] Address already in use

Heroku main log -

2018-11-23T18:29:16.367777+00:00 heroku[web.1]: Process running mem=533M(104.3%)
2018-11-23T18:29:16.367829+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
dustinfast commented 5 years ago

Tried reducing number of concurrent locos to 1 (truncated static/track/locos.json) with no effect. Behavior suggests more than one instance of the application is starting?

dustinfast commented 5 years ago

Added better exception handling - instead of erroring out on [Errno 98], the two places it was occurring now output either !!ERROR 9-1!! or !!ERROR 9-2!! and exit. After restart, these error msgs did shown up recurringly (implying an exit) but the issue kept occurring. Eventualylly mem filled and application restarted with 3 functioning locos, until mem fills up again.

Note: When run locally (with ./sim_bos.py) it still runs fine, with no error messages occurring.

Behavior still suggests more than one instance of the application is starting in the hosted environment.