search

PnX-SI / gn_module_export

Module GeoNature d'export
GNU General Public License v3.0
7 stars 10 forks source link

Impossibilité d'exporter quelque soit le format #200

Closed marineteneur closed 1 year ago

marineteneur commented 1 year ago

Bonjour,

J'ai depuis peu refait une installation de GeoNature 2.12.3 à neuf et quand je veux faire un export (en n'importe quel format) j'ai une erreur 500 Internal Server Error, alors que de son coté l'api fonctionne.

J'ai fait la montée en version de la dépendance Utils-Flask-SQLAlchemy en v0.3.5 mais rien ne change.

Dans le log geonature j'ai cette erreur qui s'affiche :

`[2023-09-07 14:20:18 +0200] [2419573] [ERROR] Exception on /exports/2/json [POST] Traceback (most recent call last): File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 611, in connect sock = self.retry.call_with_retry( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry return do() File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 612, in lambda: self._connect(), lambda error: self.disconnect(error) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 677, in _connect raise err File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 665, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 119, in reconnect_on_error yield File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 169, in _consume_from self._pubsub.subscribe(key) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1592, in subscribe ret_val = self.execute_command("SUBSCRIBE", *new_channels.keys()) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1433, in execute_command self.connection = self.connection_pool.get_connection( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 1387, in get_connection connection.connect() File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 617, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 611, in connect sock = self.retry.call_with_retry( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry return do() File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 612, in lambda: self._connect(), lambda error: self.disconnect(error) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 677, in _connect raise err File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 665, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors yield File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/base.py", line 787, in send_task self.backend.on_task_call(P, task_id) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 365, in on_task_call self.result_consumer.consume_from(task_id) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 161, in consume_from return self.start(task_id) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 139, in start self._consume_from(initial_task_id) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 169, in _consume_from self._pubsub.subscribe(key) File "/usr/lib/python3.9/contextlib.py", line 135, in exit self.gen.throw(type, value, traceback) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 122, in reconnect_on_error self._ensure(self._reconnect_pubsub, ()) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 376, in ensure return retry_over_time( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/utils/functional.py", line 312, in retry_over_time return fun(*args, *kwargs) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/backends/redis.py", line 106, in _reconnect_pubsub metas = self.backend.client.mget(self.subscribed_to) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/commands/core.py", line 1893, in mget return self.execute_command("MGET", args, options) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 1235, in execute_command conn = self.connection or pool.get_connection(command_name, options) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 1387, in get_connection connection.connect() File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 617, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/flask_cors/decorator.py", line 128, in wrapped_function resp = make_response(f(*args, *kwargs)) File "/home/geonatureadmin/geonature/backend/geonature/core/gn_permissions/decorators.py", line 56, in decorated_view return view_func(args, kwargs) File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/blueprint.py", line 173, in getOneExportThread generate_export.delay( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay return self.apply_async(args, kwargs) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/task.py", line 575, in apply_async return app.send_task( File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/base.py", line 788, in send_task amqp.send_task_message(P, name, message, **options) File "/usr/lib/python3.9/contextlib.py", line 135, in exit self.gen.throw(type, value, traceback) File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors raise ConnectionError(str(exc)) from exc kombu.exceptions.OperationalError: Error 111 connecting to localhost:6379. Connection refused. `

Et dans le log geonature-worker :

[2023-09-07 13:39:29,460: INFO/Beat] beat: Starting... [2023-09-07 13:39:30,358: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 111 connecting to localhost:6379. Connection refused.. Trying again in 2.00 seconds... (1/100)

camillemonchicourt commented 1 year ago

Quelle version du module Exports ?

camillemonchicourt commented 1 year ago

As-tu bien redémarré les services après l'installation du module ?

sudo systemctl restart geonature
sudo systemctl restart geonature-worker
marineteneur commented 1 year ago

J'ai la version 1.5.2. Oui j'ai tout relancé avec la commande suivante : sudo systemctl restart taxhub usershub postgresql geonature geonature-worker apache2

TheoLechemia commented 1 year ago

Celery n'a pas l'air de tourner ? Ou alors peut être pas sur le bon port ? Que renvoie la commande : sudo systemctl status redis

marineteneur commented 1 year ago

Merci beaucoup ! en effet redis était inactif, après un sudo systemctl start redis le problème a été réglé :)

maximetoma commented 1 year ago

J'ai de nouveau eu un problème d'export des CSV dernièrement, j'ai vérifié le status de redis qui était actif J'ai quand même effectué un restart sur le service de redis, et les exports CSV ont de nouveau fonctionnés

Je ne sais pas ce qui peut provoquer ça.... :/

Avant le restart

redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; preset: enabled) Active: active (running) since Sat 2023-05-06 07:13:34 CEST; 5 months 19 days ago Docs: http://redis.io/documentation, man:redis-server(1) Main PID: 1434416 (redis-server) Status: "Ready to accept connections" Tasks: 5 (limit: 38353) Memory: 4.0M CPU: 9h 7min 8.751s CGroup: /system.slice/redis-server.service └─1434416 "/usr/bin/redis-server 127.0.0.1:6379"

Après le restart

redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; preset: enabled) Active: active (running) since Tue 2023-10-24 15:26:45 CEST; 14s ago Docs: http://redis.io/documentation, man:redis-server(1) Main PID: 1901904 (redis-server) Status: "Ready to accept connections" Tasks: 5 (limit: 38353) Memory: 8.0M CPU: 57ms CGroup: /system.slice/redis-server.service └─1901904 "/usr/bin/redis-server 127.0.0.1:6379"

maximetoma commented 1 year ago

Après observation, c'est le worker qui pose problème J'ai encore redémarrer aujourd'hui pour que ça marche

Avant le restart du worker

[2023-10-25 10:38:23,672: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 628, in start
    c.loop(*c.loop_args())
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 1326, in on_readable
    self.cycle.on_readable(fileno)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 562, in on_readable
    chan.handlers[type]()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 906, in _receive
    ret.append(self._receive_one(c))
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/kombu/transport/redis.py", line 916, in _receive_one
    response = c.parse_response()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 796, in parse_response
    response = self._execute(conn, try_read)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 772, in _execute
    return conn.retry.call_with_retry(
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 49, in call_with_retry
    fail(error)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 774, in <lambda>
    lambda error: self._disconnect_raise_connect(conn, error),
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 761, in _disconnect_raise_connect
    raise error
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 773, in <lambda>
    lambda: command(*args, **kwargs),
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/client.py", line 794, in try_read
    return conn.read_response(disconnect_on_error=False, push_request=True)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/connection.py", line 493, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/resp2.py", line 15, in read_response
    result = self._read_response(disable_decoding=disable_decoding)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/resp2.py", line 25, in _read_response
    raw = self._buffer.readline()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/socket.py", line 115, in readline
    self._read_from_socket()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/redis/_parsers/socket.py", line 68, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2023-10-25 10:38:23,679: WARNING/MainProcess] /home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered back to the queue.
You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss setting.
In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2023-10-25 10:38:23,683: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 111 connecting to localhost:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)

[2023-10-25 10:38:25,692: INFO/MainProcess] Connected to redis://localhost:6379/0
[2023-10-25 10:38:25,697: INFO/MainProcess] mingle: searching for neighbors
[2023-10-25 10:38:26,706: INFO/MainProcess] mingle: all alone
[2023-10-25 10:38:33,900: INFO/MainProcess] Task gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba] received
[2023-10-25 10:38:33,900: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba]: Generate export 189...
[2023-10-25 10:38:37,007: INFO/ForkPoolWorker-7] Send email notification to maxime.toma (maxime.toma@lpo.fr)
[2023-10-25 10:38:37,008: INFO/MainProcess] Task geonature.core.notifications.tasks.send_notification_mail[0a6fcd54-6361-494c-8d5a-07804cbfbb59] received
[2023-10-25 10:38:37,009: INFO/ForkPoolWorker-8] Launch mail.
[2023-10-25 10:38:37,010: INFO/ForkPoolWorker-7] Send database notification to maxime.toma
[2023-10-25 10:38:37,012: ERROR/ForkPoolWorker-7] Task gn_module_export.tasks.generate_export[5c5626ce-4ffb-48b6-a1aa-f0d06b759fba] raised unexpected: ValueError("Invalid fields for <Schema(many=False)>: {'geom_2154'}.")
Traceback (most recent call last):
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/geonature/celery_app.py", line 12, in __call__
    return self.run(*args, **kwargs)
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/tasks.py", line 62, in generate_export
    export_data_file(export_id, file_name, export_url, format, id_role, filters)
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils_export.py", line 153, in export_data_file
    raise exp
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils_export.py", line 140, in export_data_file
    export_as_file(
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils/export.py", line 29, in export_as_file
    _export_as_file(
  File "/home/geonatureadmin/gn_module_export/backend/gn_module_export/utils/export.py", line 75, in _export_as_file
    func_dict[file_format](
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/export.py", line 43, in export_csv
    schema = schema_class(only=only or None)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla/schema.py", line 51, in __init__
    super().__init__(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/utils_flask_sqla_geo/schema.py", line 175, in __init__
    super().__init__(*args, only=only, exclude=exclude, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow_sqlalchemy/load_instance_mixin.py", line 45, in __init__
    super().__init__(*args, **kwargs)
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 404, in __init__
    self._init_fields()
  File "/home/geonatureadmin/geonature/backend/venv/lib/python3.9/site-packages/marshmallow/schema.py", line 979, in _init_fields
    raise ValueError(message)
ValueError: Invalid fields for <Schema(many=False)>: {'geom_2154'}.
[2023-10-25 10:38:37,768: INFO/ForkPoolWorker-8] Task geonature.core.notifications.tasks.send_notification_mail[0a6fcd54-6361-494c-8d5a-07804cbfbb59] succeeded in 0.7593830302357674s: None
[2023-10-25 10:39:25,456: INFO/MainProcess] beat: Shutting down...

Après le restart du worker

[2023-10-25 10:39:29,871: INFO/Beat] beat: Starting...
[2023-10-25 10:39:29,871: INFO/MainProcess] Connected to redis://localhost:6379/0
[2023-10-25 10:39:29,872: INFO/MainProcess] mingle: searching for neighbors
[2023-10-25 10:39:30,878: INFO/MainProcess] mingle: all alone
[2023-10-25 10:39:30,892: INFO/MainProcess] celery@ad337eea-02e7-4865-bd16-9a14049e635c.alx.host ready.
[2023-10-25 10:39:38,133: INFO/MainProcess] Task gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b] received
[2023-10-25 10:39:38,134: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b]: Generate export 189...
[2023-10-25 10:39:50,656: INFO/ForkPoolWorker-7] Send email notification to maxime.toma (maxime.toma@lpo.fr)
[2023-10-25 10:39:50,659: INFO/MainProcess] Task geonature.core.notifications.tasks.send_notification_mail[46d4d612-c651-4664-907c-103daabf7efd] received
[2023-10-25 10:39:50,660: INFO/ForkPoolWorker-8] Launch mail.
[2023-10-25 10:39:50,661: INFO/ForkPoolWorker-7] Send database notification to maxime.toma
[2023-10-25 10:39:50,664: INFO/ForkPoolWorker-7] gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b]: Export 189 generated.
[2023-10-25 10:39:50,664: INFO/ForkPoolWorker-7] Task gn_module_export.tasks.generate_export[44c74def-413a-4fb8-8255-305ddf187e5b] succeeded in 12.530796222388744s: None
[2023-10-25 10:39:51,246: INFO/ForkPoolWorker-8] Task geonature.core.notifications.tasks.send_notification_mail[46d4d612-c651-4664-907c-103daabf7efd] succeeded in 0.5856118053197861s: None