search

nansencenter / django-geo-spaas-harvesting

Harvest data into a GeoSPaaS catalog
GNU General Public License v3.0
1 stars 1 forks source link

"lost synchronisation" with the database when running multiple harvesters in parallel #75

Closed aperrin66 closed 3 years ago

aperrin66 commented 3 years ago

When running multiple harvesters in parallel, the main process sometimes ends with the following error:

2021-01-25 15:26:15,365 - geospaas_harvesting.daemon - ERROR - An unexpected error occurred
Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: lost synchronization with server: got message type "

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/venv/lib/python3.7/site-packages/geospaas_harvesting/harvest.py", line 208, in launch_harvest
    harvester.harvest()
  File "/venv/lib/python3.7/site-packages/geospaas_harvesting/harvesters.py", line 77, in harvest
    self._ingester.ingest(self._current_crawler)
  File "/venv/lib/python3.7/site-packages/geospaas_harvesting/ingesters.py", line 270, in ingest
    if self._uri_exists(download_url):
  File "/venv/lib/python3.7/site-packages/geospaas_harvesting/ingesters.py", line 78, in _uri_exists
    return DatasetURI.objects.filter(uri=uri).exists()
  File "/venv/lib/python3.7/site-packages/django/db/models/query.py", line 777, in exists
    return self.query.has_results(using=self.db)
  File "/venv/lib/python3.7/site-packages/django/db/models/sql/query.py", line 538, in has_results
    return compiler.has_results()
  File "/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1121, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/venv/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1151, in execute_sql
    cursor.execute(sql, params)
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/venv/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: lost synchronization with server: got message type "

This might be caused by the combination of Django and multiprocessing.