search

gip-inclusion / data-inclusion

data·inclusion aggrège les données de l'insertion sociale et professionnelle
https://api.data.inclusion.beta.gouv.fr/api/v0/docs
MIT License
6 stars 1 forks source link

chore(pipeline) : Attempt at fixing the API import #325

Closed vperron closed 6 days ago

vperron commented 1 week ago

For an unknown reason, sometimes the pg_dump fails to finish.

Let's add a few retries.

Result:

[2024-10-23, 07:14:12 UTC] {process_utils.py:194} INFO - Tunnel created
[2024-10-23, 07:14:13 UTC] {process_utils.py:194} INFO - ('pg_dump postgresql://airflaw:ko8B-xxxxxxx@127.0.0.1:33657/data_inclus_1511?sslmode=prefer --format=custom --clean --if-exists --no-owner --no-privileges --table api__requests --table api__communes --table api__services --table api__structures --file /tmp/tmpxjt25_ta',)
[2024-10-23, 07:14:13 UTC] {process_utils.py:194} INFO - b''
[2024-10-23, 07:14:13 UTC] {process_utils.py:194} INFO - b'pg_dump: error: connection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\nconnection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\n'
[2024-10-23, 07:14:16 UTC] {process_utils.py:194} INFO - ('pg_dump postgresql://airflaw:ko8B-xxxxxx@127.0.0.1:33657/data_inclus_1511?sslmode=prefer --format=custom --clean --if-exists --no-owner --no-privileges --table api__requests --table api__communes --table api__services --table api__structures --file /tmp/tmpxjt25_ta',)
[2024-10-23, 07:14:17 UTC] {process_utils.py:194} INFO - b''
[2024-10-23, 07:14:17 UTC] {process_utils.py:194} INFO - b'pg_dump: error: connection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\nconnection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\n'
[2024-10-23, 07:14:20 UTC] {process_utils.py:194} INFO - ('pg_dump postgresql://airflaw:ko8B-xxxxxxx@127.0.0.1:33657/data_inclus_1511?sslmode=prefer --format=custom --clean --if-exists --no-owner --no-privileges --table api__requests --table api__communes --table api__services --table api__structures --file /tmp/tmpxjt25_ta',)
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - b''
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - b'pg_dump: error: connection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\nconnection to server at "127.0.0.1", port 33657 failed: FATAL:  password authentication failed for user "airflaw"\n'
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - Traceback (most recent call last):
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-callxglv8768/script.py", line 90, in import_data_inclusion_api
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -     raise exc
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-callxglv8768/script.py", line 70, in import_data_inclusion_api
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -     subprocess.run(
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -   File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -     raise CalledProcessError(retcode, process.args,
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - subprocess.CalledProcessError: Command '('pg_dump postgresql://airflaw:ko8B-Lsqb62JS_WdoFZD@127.0.0.1:33657/data_inclus_1511?sslmode=prefer --format=custom --clean --if-exists --no-owner --no-privileges --table api__requests --table api__communes --table api__services --table api__structures --file /tmp/tmpxjt25_ta',)' returned non-zero exit status 1.
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - 
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO - The above exception was the direct cause of the following exception:
[2024-10-23, 07:14:21 UTC] {process_utils.py:194} INFO -

En staging: image

Les retry ont l'air efficaces.

vperron commented 6 days ago

J'avoue, non, j'ai filé directement sur tenacity après avoir tenté le coup des TCP keepalives dans la connection string.

En effet ça sera plus simple avec ça, je tente et si ça vaut le coup j'en profiterai pour nettoyer tenacity qui a priori ne devrait plus etre là.

vperron commented 6 days ago

J'ai vérifié localement que ça se comportait comme prévu avec les retry Airflow.

vmttn commented 6 days ago

rebased

vperron commented 6 days ago

Ah bah je venais de rebase ^^

On Thu, Oct 24, 2024 at 3:34 PM vmttn @.***> wrote:

Merged #325 https://github.com/gip-inclusion/data-inclusion/pull/325 into main.

— Reply to this email directly, view it on GitHub https://github.com/gip-inclusion/data-inclusion/pull/325#event-14851712219, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAVUKVZ6GJLJ2OXXSMVKH3Z5DZOJAVCNFSM6AAAAABQOD24GCVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUHA2TCNZRGIZDCOI . You are receiving this because you authored the thread.Message ID: @.***>

-- Victor