search

LexPredict / lexpredict-contraxsuite

LexPredict ContraxSuite
GNU Affero General Public License v3.0
167 stars 62 forks source link

After upgrade to 1.0.6 (from deploy script), Locate fails #26

Open dwmcqueen opened 6 years ago

dwmcqueen commented 6 years ago

When trying to run Locate, I get the following error:

`NFO 2018-02-14 15:34:15,262 | Start task "Locate", id=24 INFO 2018-02-14 15:34:15,270 | Deleted 0 UrlUsage objects INFO 2018-02-14 15:34:15,274 | Deleted 0 TextUnitTag(tag=url) INFO 2018-02-14 15:34:15,277 | Deleted 0 DateDurationUsage objects INFO 2018-02-14 15:34:15,280 | Deleted 0 TextUnitTag(tag=duration) INFO 2018-02-14 15:34:15,282 | Deleted 0 CourtUsage objects INFO 2018-02-14 15:34:15,285 | Deleted 0 TextUnitTag(tag=court) INFO 2018-02-14 15:34:15,288 | Deleted 0 CopyrightUsage objects INFO 2018-02-14 15:34:15,290 | Deleted 0 TextUnitTag(tag=copyright) INFO 2018-02-14 15:34:15,293 | Deleted 0 DefinitionUsage objects INFO 2018-02-14 15:34:15,296 | Deleted 0 TextUnitTag(tag=definition) INFO 2018-02-14 15:34:15,298 | Deleted 0 RatioUsage objects INFO 2018-02-14 15:34:15,301 | Deleted 0 TextUnitTag(tag=ratio) INFO 2018-02-14 15:34:15,304 | Deleted 0 CurrencyUsage objects INFO 2018-02-14 15:34:15,306 | Deleted 0 TextUnitTag(tag=currency) INFO 2018-02-14 15:34:15,309 | Deleted 0 PartyUsage objects INFO 2018-02-14 15:34:15,311 | Deleted 0 TextUnitTag(tag=party) INFO 2018-02-14 15:34:15,314 | Deleted 0 DateUsage objects INFO 2018-02-14 15:34:15,317 | Deleted 0 TextUnitTag(tag=date) INFO 2018-02-14 15:34:15,320 | Deleted 0 TrademarkUsage objects INFO 2018-02-14 15:34:15,324 | Deleted 0 TextUnitTag(tag=trademark) INFO 2018-02-14 15:34:15,327 | Deleted 0 DistanceUsage objects INFO 2018-02-14 15:34:15,330 | Deleted 0 TextUnitTag(tag=distance) INFO 2018-02-14 15:34:15,332 | Deleted 0 PercentUsage objects INFO 2018-02-14 15:34:15,335 | Deleted 0 TextUnitTag(tag=percent) INFO 2018-02-14 15:34:15,338 | Deleted 0 AmountUsage objects INFO 2018-02-14 15:34:15,340 | Deleted 0 TextUnitTag(tag=amount) INFO 2018-02-14 15:34:15,343 | Deleted 0 GeoEntityUsage objects INFO 2018-02-14 15:34:15,345 | Deleted 0 GeoAliasUsage objects INFO 2018-02-14 15:34:15,348 | Deleted 0 TextUnitTag(tag=geoentity) INFO 2018-02-14 15:34:15,351 | Deleted 0 CitationUsage objects INFO 2018-02-14 15:34:15,354 | Deleted 0 TextUnitTag(tag=citation) INFO 2018-02-14 15:34:15,356 | Deleted 0 RegulationUsage objects INFO 2018-02-14 15:34:15,359 | Deleted 0 TextUnitTag(tag=regulation) INFO 2018-02-14 15:34:15,362 | Deleted 0 TermUsage objects INFO 2018-02-14 15:34:15,365 | Deleted 0 TextUnitTag(tag=term) INFO 2018-02-14 15:34:23,614 | Run location of [url; duration; court; definition; ratio; distance; party; date; trademark; copyright; percent; amount; geoentity; currency; citation; regulation; term]. INFO 2018-02-14 15:34:23,617 | Locate in [paragraphs and sentences]. INFO 2018-02-14 15:34:23,629 | Found 39238 Text Units. Added 39238 subtasks. ERROR 2018-02-14 15:35:09,604 | ERROR ERROR 2018-02-14 15:35:09,638 | OperationalError: [Errno 111] Connection refused ERROR 2018-02-14 15:35:09,640 | Traceback (most recent call last): File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 494, in _ensured return fun(*args, **kwargs) File "/opt/ve/lib/python3.5/site-packages/kombu/messaging.py", line 203, in _publish mandatory=mandatory, immediate=immediate, File "/opt/ve/lib/python3.5/site-packages/amqp/channel.py", line 1734, in _basic_publish (0, exchange, routing_key, mandatory, immediate), msg File "/opt/ve/lib/python3.5/site-packages/amqp/abstract_channel.py", line 50, in send_method conn.frame_writer(1, self.channel_id, sig, args, content) File "/opt/ve/lib/python3.5/site-packages/amqp/method_framing.py", line 166, in write_frame write(view[:offset]) File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 258, in write self._write(s) ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors yield File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 515, in _ensured reraise_as_library_errors=False, File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection callback) File "/opt/ve/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time return fun(*args, **kwargs) File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect return self.connection File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection self._connection = self._establish_connection() File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection conn = self.transport.establish_connection() File "/opt/ve/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection conn.connect() File "/opt/ve/lib/python3.5/site-packages/amqp/connection.py", line 282, in connect self.transport.connect() File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 109, in connect self._connect(self.host, self.port, self.connect_timeout) File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 150, in _connect self.sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/lexpredict-contraxsuite/contraxsuite_services/apps/task/tasks.py", line 210, in run self.process(kwargs) File "/opt/lexpredict-contraxsuite/contraxsuite_services/apps/task/tasks.py", line 822, in process taskid='%d%s' % (self.task.id, fast_uuid())) File "/opt/ve/lib/python3.5/site-packages/celery/app/task.py", line 536, in apply_async options File "/opt/ve/lib/python3.5/site-packages/celery/app/base.py", line 737, in send_task amqp.send_task_message(P, name, message, options) File "/opt/ve/lib/python3.5/site-packages/celery/app/amqp.py", line 554, in send_task_message properties File "/opt/ve/lib/python3.5/site-packages/kombu/messaging.py", line 181, in publish exchange_name, declare, File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 527, in _ensured errback and errback(exc, 0) File "/usr/lib/python3.5/contextlib.py", line 77, in exit self.gen.throw(type, value, traceback) File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors sys.exc_info()[2]) File "/opt/ve/lib/python3.5/site-packages/vine/five.py", line 178, in reraise raise value.with_traceback(tb) File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors yield File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 515, in _ensured reraise_as_library_errors=False, File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection callback) File "/opt/ve/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time return fun(*args, **kwargs) File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect return self.connection File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection self._connection = self._establish_connection() File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection conn = self.transport.establish_connection() File "/opt/ve/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection conn.connect() File "/opt/ve/lib/python3.5/site-packages/amqp/connection.py", line 282, in connect self.transport.connect() File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 109, in connect self._connect(self.host, self.port, self.connect_timeout) File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 150, in _connect self.sock.connect(sa) kombu.exceptions.OperationalError: [Errno 111] Connection refused

INFO 2018-02-14 15:35:09,646 | End of main task "Locate", id=24 INFO 2018-02-14 15:35:09,654 | There are 39001 uncompleted subtasks remaining. Please wait.`

dwmcqueen commented 6 years ago

I upgraded to 1.06 by git pull ing the latest deploy (to add in new fabricrc and python requirements) and then ran "update-local.sh" as the local (non-root) user. The script appeared to run correctly and the site loaded (with 1.0.6 indicated as the new version).

reddalexx commented 6 years ago

Such errors usually mean that celery doesn't work. possible solution:

  1. check that celery works: celery status -A apps
  2. restart it if needed: celery multi restart 2 -A apps -f celery.log -B
  3. check that rabbitmg works: sudo rabbitmqctl status then restart if needed: sudo rabbitmqctl restart

but as I think restarting celery should fix the issue if it doesn't help I need logs: celery.log, /var/log/uwsgi/contrax.log

dwmcqueen commented 6 years ago

It looks like Celery is running as well as rabbitmq. Attached are the files you mentioned, celery.log contrax.log

reddalexx commented 6 years ago

Perhaps it didn't work at 15:35:09 but as I see from celery.log celery works good since that time [2018-02-14 15:35:29,106: INFO/MainProcess] Connected to amqp://contrax1:@127.0.0.1:5672/contrax1_vhost [2018-02-14 15:35:29,108: INFO/MainProcess] Connected to amqp://contrax1:@127.0.0.1:5672/contrax1_vhost

may be there was some issue with rabbitmq in that time, at least celery.log says that [2018-02-14 15:35:29,093: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...

dwmcqueen commented 6 years ago

Something did happen and now tasks are hung. I think I am going to wipe the server and move to the docker deployment.