search

line / promgen

Promgen is a configuration file generator for Prometheus
https://line.github.io/promgen/
MIT License
1.06k stars 150 forks source link

Table 'promgen_ruleannotation' is truncated after migrate command on 0.58.x #485

Open mysterious-maniac opened 7 months ago

mysterious-maniac commented 7 months ago

Some context for this is that the docker Promgen image connects to a MySQL database on the host.

I had earlier experienced an issue re:

# docker exec -it 194fa5c8c5c9 promgen migrate
SystemCheckError: System check identified some issues:

ERRORS:
promgen.Rule: (fields.E180) MySQL does not support JSONFields.
promgen.Rule: (fields.E180) MySQL does not support JSONFields.

But it was resolved by updating the MySQL DB version from 5.6 -> 5.7.44 (ref: "JSONField is supported on MariaDB 10.2.7+, MySQL 5.7.8+, Oracle, PostgreSQL, and SQLite (with the JSON1 extension enabled)."

When running the newly pulled image (0.58.1/0.58.2), I also run the migrate command (docker exec -it promgen migrate):

Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, promgen, sessions, sites, social_django
Running migrations:
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying authtoken.0003_tokenproxy... OK
  Applying promgen.0021_shard_load... OK
  Applying promgen.0022_rule_labels_annotations... OK
  Applying social_django.0009_auto_20191118_0520... OK
  Applying social_django.0010_uid_db_index... OK
  Applying social_django.0011_alter_id_fields... OK

When a config change is made on the promgen ui, the celery worker outputs an error regarding the promgen_ruleannotation table not existing anymore:

 Task promgen.tasks.write_rules[5d8d9910-0925-428b-b254-916d34350526] raised unexpected: ProgrammingError(1146, "Table 'promgendocker2.promgen_ruleannotation
' doesn't exist")
Traceback (most recent call last):
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1146, "Table 'promgendocker2.promgen_ruleannotation' doesn't exist")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/promgen38/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/promgen38/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/promgen-0.38/promgen/tasks.py", line 114, in write_rules
    fp.write(prometheus.render_rules())
  File "/usr/src/promgen-0.38/promgen/prometheus.py", line 80, in render_rules
    'rulelabel_set',
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1625, in prefetch_related_objects
    obj_list, additional_lookups = prefetch_one_level(obj_list, prefetcher, lookup, level)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1738, in prefetch_one_level
    prefetcher.get_prefetch_queryset(instances, lookup.get_current_queryset(level)))
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 627, in get_prefetch_queryset
    for rel_obj in queryset:
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1100, in execute_sql
    cursor.execute(sql, params)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/promgen38/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 71, in execute
    return self.cursor.execute(query, args)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/opt/promgen38/lib/python3.6/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'promgendocker2.promgen_ruleannotation' doesn't exist")

I had to rollback my promgen database in MySQL and found out that when the migrate command is executed, for whatever reason the promgen_ruleannotation table is truncated from the database:

Before docker migrate:

| auth_group                 |
| auth_group_permissions     |
| auth_permission            |
| auth_user                  |
| auth_user_groups           |
| auth_user_user_permissions |
| authtoken_token            |
| django_admin_log           |
| django_content_type        |
| django_migrations          |
| django_session             |
| django_site                |
| promgen_alert              |
| promgen_alerterror         |
| promgen_alertlabel         |
| promgen_audit              |
| promgen_defaultexporter    |
| promgen_exporter           |
| promgen_farm               |
| promgen_filter             |
| promgen_host               |
| promgen_probe              |
| promgen_project            |
| promgen_prometheus         |
| promgen_rule               |
| promgen_ruleannotation     |
| promgen_rulelabel          |
| promgen_sender             |
| promgen_service            |
| promgen_shard              |
| promgen_url                |
| social_auth_association    |
| social_auth_code           |
| social_auth_nonce          |
| social_auth_partial        |
| social_auth_usersocialauth |
+----------------------------+
36 rows in set (0.01 sec)

After docker migrate:

| auth_group                 |
| auth_group_permissions     |
| auth_permission            |
| auth_user                  |
| auth_user_groups           |
| auth_user_user_permissions |
| authtoken_token            |
| django_admin_log           |
| django_content_type        |
| django_migrations          |
| django_session             |
| django_site                |
| promgen_alert              |
| promgen_alerterror         |
| promgen_alertlabel         |
| promgen_audit              |
| promgen_defaultexporter    |
| promgen_exporter           |
| promgen_farm               |
| promgen_filter             |
| promgen_host               |
| promgen_probe              |
| promgen_project            |
| promgen_prometheus         |
| promgen_rule               |
| promgen_sender             |
| promgen_service            |
| promgen_shard              |
| promgen_url                |
| social_auth_association    |
| social_auth_code           |
| social_auth_nonce          |
| social_auth_partial        |
| social_auth_usersocialauth |
+----------------------------+
34 rows in set (0.00 sec)

Is this the expected behaviour? I do not know why the promgen_ruleannoation or the promgen_rulelabel tables would dissapear.

This issue is preventing me from running this version of Promgen (however if https://github.com/line/promgen/issues/484 gets resolved, this will be a non-issue as well).

Thanks in advanced!