search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 479 forks source link

solr mongo-connector index for error #777

Open nyf1993 opened 6 years ago

nyf1993 commented 6 years ago

Traceback (most recent call last): File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request httplib_response = conn.getresponse() File "d:\python\lib\http\client.py", line 1331, in getresponse response.begin() File "d:\python\lib\http\client.py", line 297, in begin version, status, reason = self._read_status() File "d:\python\lib\http\client.py", line 258, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "d:\python\lib\socket.py", line 586, in readinto return self._sock.recv_into(b) socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\requests\adapters.py", line 440, in send timeout=timeout File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "d:\python\lib\site-packages\urllib3\util\retry.py", line 357, in increment raise six.reraise(type(error), error, _stacktrace) File "d:\python\lib\site-packages\urllib3\packages\six.py", line 686, in reraise raise value File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen chunked=chunked) File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 389, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 309, in _raise_timeout raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value) urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\pysolr.py", line 366, in _send_request timeout=self.timeout) File "d:\python\lib\site-packages\requests\sessions.py", line 555, in post return self.request('POST', url, data=data, json=json, kwargs) File "d:\python\lib\site-packages\requests\sessions.py", line 508, in request resp = self.send(prep, send_kwargs) File "d:\python\lib\site-packages\requests\sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "d:\python\lib\site-packages\requests\adapters.py", line 521, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60) 2017-12-25 13:37:05,460 [CRITICAL] mongo_connector.oplog_manager:670 - Exception during collection dump Traceback (most recent call last): File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request six.raise_from(e, None) File "", line 2, in raise_from File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request httplib_response = conn.getresponse() File "d:\python\lib\http\client.py", line 1331, in getresponse response.begin() File "d:\python\lib\http\client.py", line 297, in begin version, status, reason = self._read_status() File "d:\python\lib\http\client.py", line 258, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "d:\python\lib\socket.py", line 586, in readinto return self._sock.recv_into(b) socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\requests\adapters.py", line 440, in send timeout=timeout File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "d:\python\lib\site-packages\urllib3\util\retry.py", line 357, in increment raise six.reraise(type(error), error, _stacktrace) File "d:\python\lib\site-packages\urllib3\packages\six.py", line 686, in reraise raise value File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen chunked=chunked) File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 389, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "d:\python\lib\site-packages\urllib3\connectionpool.py", line 309, in _raise_timeout raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value) urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\pysolr.py", line 366, in _send_request timeout=self.timeout) File "d:\python\lib\site-packages\requests\sessions.py", line 555, in post return self.request('POST', url, data=data, json=json, kwargs) File "d:\python\lib\site-packages\requests\sessions.py", line 508, in request resp = self.send(prep, send_kwargs) File "d:\python\lib\site-packages\requests\sessions.py", line 618, in send r = adapter.send(request, **kwargs) File "d:\python\lib\site-packages\requests\adapters.py", line 521, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\mongo_connector\util.py", line 35, in wrapped return f(*args, kwargs) File "d:\python\lib\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 294, in bulk_upsert self.solr.add(batch, add_kwargs) File "d:\python\lib\site-packages\pysolr.py", line 891, in add overwrite=overwrite, handler=handler) File "d:\python\lib\site-packages\pysolr.py", line 478, in _update return self._send_request('post', path, message, {'Content-type': 'text/xml; charset=utf-8'}) File "d:\python\lib\site-packages\pysolr.py", line 370, in _send_request raise SolrError(error_message % (url, err)) pysolr.SolrError: Connection to server 'http://172.16.100.63:8000/solr/test_mongo/update/' timed out: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "d:\python\lib\site-packages\mongo_connector\oplog_manager.py", line 625, in do_dump upsert_all(dm) File "d:\python\lib\site-packages\mongo_connector\oplog_manager.py", line 611, in upsert_all mapped_ns, long_ts) File "d:\python\lib\site-packages\mongo_connector\util.py", line 46, in wrapped reraise(new_type, exc_value, exc_tb) File "d:\python\lib\site-packages\mongo_connector\compat.py", line 9, in reraise raise exctype(str(value)).with_traceback(trace) File "d:\python\lib\site-packages\mongo_connector\util.py", line 35, in wrapped return f(*args, kwargs) File "d:\python\lib\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 294, in bulk_upsert self.solr.add(batch, add_kwargs) File "d:\python\lib\site-packages\pysolr.py", line 891, in add overwrite=overwrite, handler=handler) File "d:\python\lib\site-packages\pysolr.py", line 478, in _update return self._send_request('post', path, message, {'Content-type': 'text/xml; charset=utf-8'}) File "d:\python\lib\site-packages\pysolr.py", line 370, in _send_request raise SolrError(error_message % (url, err)) mongo_connector.errors.OperationFailed: Connection to server 'http://172.16.100.63:8000/solr/test_mongo/update/' timed out: HTTPConnectionPool(host='172.16.100.63', port=8000): Read timed out. (read timeout=60) 2017-12-25 13:37:05,465 [ERROR] mongo_connector.oplog_manager:678 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['172.16.99.248:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='r0'), 'local'), 'oplog.rs') 2017-12-25 13:37:05,857 [ERROR] mongo_connector.connector:398 - MongoConnector: OplogThread <OplogThread(Thread-2, started 24072)> unexpectedly stopped! Shutting down

nyf1993 commented 6 years ago

Mongodb inserts about 400 data, and each data is about 10m. The above error occurred and solr error: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space