search

SocialGouv / cdtn-admin

Veille & administration des contenus pour le code du travail numérique
https://cdtn-admin.fabrique.social.gouv.fr
6 stars 1 forks source link

Perf: trigger #363

Closed lionelB closed 3 years ago

lionelB commented 3 years ago

Les triggers que l'on a rajouté ont un gros impact sur les perfs actuelle notamment au niveau de l'ingester Pour une mise a jour des fiche service-public(fiches-vdd) et convention-collective (kali-data)

m-maillot commented 3 years ago

Je pense que l'on peut essayer de monitorer le call pour voir où sont les impacts. Je sais qu'il y a des outils spécifiques, faut que je rejette un oeil.

Ensuite, on peut aussi aller voir du coté du LISTEN/NOTIFY afin de rendre asynchrone les parties qui peuvent être couteuse (https://www.postgresql.org/docs/current/sql-notify.html)

lionelB commented 3 years ago

on peut aussi jouer sur la config des trigger (passer le trigger de la table documents en STATEMENT) et voir pourquoi le trigger delete a autant d'impact

revolunet commented 3 years ago

C'est moche. bon c'est vraiment un "nice to have" en sécu, on a aucune obligation pour l'instant donc a dropper si c'est génant imho

lionelB commented 3 years ago

pour le clean, on pourra le render async une fois que cette issue sera traité coté hasura https://github.com/hasura/graphql-engine/issues/5050 ca va permettre d'appeler l'api hasura depuis un cron trigger

lionelB commented 3 years ago

document row trigger , clean trigger enabled

doc availability time is part of the update time and correspond to the time we update is_available of all documents for a given source

total time update kali doc availability update fiche-sp doc availbility
1074.88s 11:29.590 4:30.081 1:23.863 6:21.143
515.57s 3:48.858 1:00.237 4:42.772 1:27.756
876.74s 9:09.422 4:24.368 5:20.112 1:26.926
310.23s 1:59.333 56.546s 3:04.810 1:28.694
343.75s 2:29.834 1:02.877 3:08.729 1:34.256
586.06s 4:13.267 1:11.017 5:26.881 1:38.038
329.08s 2:14.170 1:06.155 3:08.322 1:29.937
330.78s 2:11.619 1:06.262 3:13.475 1:31.232
665.79s 4:17.038 1:08.053 6:42.855 1:36.397
1034.16s 8:16.946 1:04.978 8:50.430 1:34.369

document statement trigger , clean trigger enabled

total time update kali doc availability update fiche-sp doc availbility
314.25s. 2:20.478 485.559ms 2:47.765 1.263s
317.07s. 4:39.204 693.496ms 30.812s 1.335s
248.17s. 1:19.215 715.925ms 2:41.129 1.496s
70.82s. 47.112s 579.367ms 19.744s 1.017s
43.10s. 10.609s 440.339ms 25.986s 1.080s
37.05s. 11.540s 435.406ms 19.483s 1.255s
37.26s. 10.926s 414.791ms 19.786s 1.078s
49.47s. 23.687s 630.209ms 19.936s 1.050s
72.19s. 11.127s 407.912ms 54.671s 796.512ms
38.33s. 11.105s 565.029ms 21.194s 843.012ms

document row trigger , clean trigger disabled

total time update kali doc availability update fiche-sp doc availbility
36.11s. 10.768s 534.919ms 18.941s 1.861s
169.13s. 2:20.982 548.87ms 21.736s 1.900s
34.88s. 10.259s 701.218ms 19.122s 1.873s
35.06s. 10.005s 493.307ms 19.832s 2.561s
37.03s. 10.468s 467.427ms 18.589s 1.394s
33.33s. 9.582s 527.142ms 18.474s 1.505s
42.93s. 15.556s 550.377ms 22.178s 1.402s
35.17s. 10.110s 660.531ms 19.130s 1.563s
33.72s. 9.711s 569.831ms 18.278s 1.612s
36.18s. 9.896s 597.879ms 18.512s 1.270s

document statement trigger , clean trigger disabled

total time update kali doc availability update fiche-sp doc availbility
41.62s. 17.311s 535.911ms 17.741s 1.006s
32.97s. 9.659s 340.785ms 17.940s 946.285ms
32.04s. 9.352s 325.003ms 17.711s 656.129ms
32.17s. 9.629s 386.088ms 17.299s 689.77ms
67.76s. 9.538s 344.907ms 49.843s 805.983ms
32.52s. 9.349s 345.769ms 17.498s 866.862ms
31.51s. 9.464s 354.525ms 17.026s 667.716ms
33.35s. 11.118s 307.166ms 17.524s 989.212ms
32.46s. 9.562s 337.581ms 17.443s 810.866ms
34.99s. 9.425s 242.462ms 19.396s 798.301ms

document trigger disabled , clean trigger disabled

total time update kali doc availability update fiche-sp doc availbility
32.42s. 9.770s 369.034ms 17.697s 975.39ms
32.86s. 9.793s 439.488ms 17.776s 709.365ms
32.94s. 9.718s 532.084ms 18.155s 1.787s
52.29s. 9.514s 290.812ms 37.144s 761.313ms
33.38s. 11.032s 477.728ms 17.879s 1.309s
33.24s. 9.545s 320.044ms 17.843s 681.371ms
32.53s. 9.569s 312.139ms 17.337s 753.395ms
33.00s. 10.737s 1.360s 16.659s 634.863ms
40.15s. 9.401s 239.012ms 24.691s 702.132ms
33.81s. 9.698s 397.824ms 18.078s 1.329s
m-maillot commented 3 years ago

Peux tu décrire ce que mesure les colonnes doc availability ? Comment doit on lire ce type de données 1:02.877 dans certaines colonnes ? Que signifie le 1: ?

m-maillot commented 3 years ago

Tu peux oublier la première question 😄

lionelB commented 3 years ago

1:02.877 correspond a 1minute 02 secondes

SocialGroovyBot commented 3 years ago

:tada: This issue has been resolved in version 1.0.0-alpha.11 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

SocialGroovyBot commented 3 years ago

:tada: This issue has been resolved in version 1.0.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: