DB sync fails due to validation error

[mconlow]

Here's my environment:

Django             3.2.21
django-peeringdb   3.3.0
peeringdb          2.0.0

when processing the [campus] object, I get:

    raise ValidationError(_("Invalid value: {}").format(value))
django.core.exceptions.ValidationError: ['Invalid value: 240 VAC']

[netixx]

Same Issue here (same versions).

Here is the full trace (not sure if it adds anything):

  pdb.updater.update_all(rs, since=None, fetch_private=True)
  File "/usr/local/lib/python3.11/site-packages/peeringdb/_update.py", line 237, in update_all
    self._handle_incremental_sync(entries, res)
  File "/usr/local/lib/python3.11/site-packages/peeringdb/_update.py", line 187, in _handle_incremental_sync
    obj, _ = self.create_obj(row, res)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/peeringdb/_update.py", line 139, in create_obj
    set_many_relations(self.backend, res, obj, row)
  File "/usr/local/lib/python3.11/site-packages/peeringdb/_sync.py", line 72, in set_many_relations
    B.set_relation_many_to_many(obj, fname, objs)
  File "/usr/local/lib/python3.11/site-packages/django_peeringdb/client_adaptor/backend.py", line 137, in set_relation_many_to_many
    relation.set(objs)  # Django 2.x
    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 908, in set
    old_objs = set(self.using(db).all())
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 121, in __iter__
    for row in compiler.results_iter(results):
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1500, in apply_converters
    value = converter(value, expression, connection)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_peeringdb/fields.py", line 53, in from_db_value
    self.clean_choices(values)
  File "/usr/local/lib/python3.11/site-packages/django_peeringdb/fields.py", line 29, in clean_choices
    raise ValidationError(_("Invalid value: {}").format(value))
django.core.exceptions.ValidationError: ['Invalid value: 240 VAC']

[netixx]

I got around that issue by cleaning (truncate table cascade) all the tables.

I guess at some point something was synced that was invalid, and made this crash ?

[vegu]

the validation for the field was changed substantially a few weeks back, think cache server may have still been serving old values for that field, i can't reproduce this now, so hopefully resolved?

[fdomain]

@vegu I've just run into this issue

peeringdb sync --fetch-private;                                                                                                             1 ✘  12m 12s  peeringdb-py   6   09:22:08 
Syncing to https://www.peeringdb.com/api
[org] Fetching from API 
[org] Processing 898 objects
[campus] Fetching from API 
[campus] Processing 4 objects
[fac] Fetching from API 
[fac] Processing 293 objects
Fetching dangling relationship Organization 37212
Traceback (most recent call last):
  File "/home/f.domain/dev/peeringdb-py/.venv/bin/peeringdb", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/cli.py", line 68, in main
    return handler(config=cfg, **vars(options))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/commands.py", line 20, in _wrapped
    r = func(*a, **k)
        ^^^^^^^^^^^^^
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/commands.py", line 262, in handle
    client.updater.update_all(rs, since, fetch_private=kwargs["fetch_private"])
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/_update.py", line 237, in update_all
    self._handle_incremental_sync(entries, res)
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/_update.py", line 188, in _handle_incremental_sync
    self.copy_object(obj)
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/_update.py", line 48, in copy_object
    self.clean_obj(old)
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/_update.py", line 69, in clean_obj
    raise e
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/peeringdb/_update.py", line 59, in clean_obj
    self.backend.clean(obj)
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/django_peeringdb/client_adaptor/backend.py", line 142, in clean
    obj.full_clean()
  File "/home/f.domain/dev/peeringdb-py/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 1502, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'available_voltage_services': ['Invalid value: 400 VAC']}

[jackcarrozzo]

+1

[vegu]

we ran into this on an old database and it seems to be something that requires a local migration and can't rely on the remote api to provide the new valid values as the validation error isn't raised in the model cleaning function, but when setting the field value, breaking even simply doing a Facility.objects.all() query.

could try to tuck on a local data migration, that doesn't update the updated field of the facility and removes the old voltage values.

and something to keep in mind when changing validation values on other fields going forward.

[vegu]

@leovegoda identified as a bug

[leovegoda]

Moving to Ready for Implementation as this has been identified as a bug by a developer

[dant89]

Same issue experienced by LINX with our DB cache script:

 django.core.exceptions.ValidationError: {'available_voltage_services': ['Invalid value: 400 VAC']}

Syncing to https://www.peeringdb.com/api
[org] Fetching from API 
[org] Processing 0 objects
[campus] Fetching from API 
[campus] Processing 0 objects
[fac] Fetching from API 
[fac] Processing 742 objects
Traceback (most recent call last):
  File "/srv/conda/envs/peeringdb_sync/bin/peeringdb", line 8, in <module>
    sys.exit(main())

Truncating peeringb_facility and re-runing fixed this issue, not ideal solution though.