vegu
commented
6 years ago There are several sync issues of these nature in the current client, and a rewrite is being planned at https://github.com/peeringdb/peeringdb/issues/158. Until then, the only sure way we have found to ensure that your local copy is proper is to re create the database and run a fresh sync.
grizz
Calling "peeringdb sync" to synchronize database to local copy leads to a fatal error due to a missing foreign key constraint. Obviously there is somewhere a reference to an ix with ix_id 1176 which seems to be missing in peeringdb itself:
peeringdb sync
Operations to perform: Synchronize unmigrated apps: django_peeringdb Apply all migrations: (none) Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: No migrations to apply. syncing from https://www.peeringdb.com/api only tables False Starting new HTTPS connection (1): www.peeringdb.com https://www.peeringdb.com:443 "GET /api/org?since=1513801662&limit=0 HTTP/1.1" 200 271 org last update 1513801662 1 changed data to be processed 1 Starting new HTTPS connection (1): www.peeringdb.com https://www.peeringdb.com:443 "GET /api/fac?since=1513770362&limit=0 HTTP/1.1" 200 24 fac last update 1513770362 0 changed data to be processed 0 Starting new HTTPS connection (1): www.peeringdb.com https://www.peeringdb.com:443 "GET /api/net?since=1513800747&limit=0 HTTP/1.1" 200 618 net last update 1513800747 1 changed data to be processed 1 Starting new HTTPS connection (1): www.peeringdb.com https://www.peeringdb.com:443 "GET /api/ix?since=1513766134&limit=0 HTTP/1.1" 200 562 ix last update 1513766134 1 changed data to be processed 1 Starting new HTTPS connection (1): www.peeringdb.com https://www.peeringdb.com:443 "GET /api/ixfac?since=1513726747&limit=0 HTTP/1.1" 200 1097 ixfac last update 1513726747 8 changed data to be processed 8 {'ix': [u'internet exchange instance with id 1176 does not exist.']} : errors: {'ix': [u'internet exchange instance with id 1176 does not exist.']} ix: Missing Object, dict: {'_validators': [], 'auto_created': False, 'serialize': True, '_unique': False, 'unique_for_year': None, 'blank': False, 'help_text': u'', 'null': False, 'to_fields': ['id'], 'db_tablespace': '', 'db_index': True, 'is_relation': True, 'unique_for_month': None, 'unique_for_date': None, 'primary_key': False, 'concrete': True, 'swappable': True, 'max_length': None, 'rel':, 'from_fields': [u'self'], 'verbose_name': u'ix', '_choices': [], 'creation_counter': 88, 'editable': True, 'related_model': <class 'django_peeringdb.models.concrete.InternetExchange'>, 'error_messages': {u'unique': <django.utils.functional.proxy object at 0x7f3504480590>, u'invalid': <django.utils.functional.proxy object at 0x7f350440b050>, u'invalid_choice': <django.utils.functional.proxy object at 0x7f3504581710>, u'blank': <django.utils.functional.proxy object at 0x7f3504480550>, u'null': <django.utils.functional.proxy object at 0x7f3504480510>, u'unique_for_date': <django.utils.functional.proxy object at 0x7f35044805d0>}, '_related_fields': [(, )], '_error_messages': None, 'db_constraint': True, '_verbose_name': None, 'name': 'ix', 'db_column': None, 'default': <class django.db.models.fields.NOT_PROVIDED at 0x7f35045c8b48>, 'attname': u'ix_id', 'column': u'ix_id', 'model': <class 'django_peeringdb.models.concrete.InternetExchangeFacility'>, 'opts': }
ix.1176 not found locally, trying to fetch object...
Starting new HTTPS connection (1): www.peeringdb.com
https://www.peeringdb.com:443 "GET /api/ix/1176?depth=0 HTTP/1.1" 404 33
Traceback (most recent call last):
File "/usr/bin/peeringdb", line 11, in
load_entry_point('peeringdb==0.6.0', 'console_scripts', 'peeringdb')()
File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in call
return self.main(args, kwargs)
File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, kwargs)
File "/usr/lib/python2.7/site-packages/peeringdb/cli.py", line 167, in sync
db.sync()
File "/usr/lib/python2.7/site-packages/peeringdb/localdb.py", line 124, in sync
call_command('pdb_sync', interactive=False)
File "/usr/lib/python2.7/site-packages/django/core/management/init.py", line 120, in call_command
return command.execute(*args, *defaults)
File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
output = self.handle(args, options)
File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 92, in handle
self.sync(tables, pk, limit=limit)
File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 99, in sync
self.update_db(cls, self.get_objs(cls, pk=pk, **kwargs))
File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 178, in update_db
self._sync(cls, row)
File "/usr/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 162, in _sync
r = self.rpc.get(field, int(m.group(1)), depth=0)
File "/usr/lib/python2.7/site-packages/twentyc/rpc/client.py", line 114, in get
return self._load(self._request(typ, id=id, params=kwargs))
File "/usr/lib/python2.7/site-packages/twentyc/rpc/client.py", line 85, in _load
self._throw(res, data)
File "/usr/lib/python2.7/site-packages/twentyc/rpc/client.py", line 65, in _throw
raise NotFoundException("%d %s" % (res.status_code, err))
twentyc.rpc.client.NotFoundException: 404 Not found.
I wonder how this exception can occure as I thougt the local database created would be an exact clone of the live-database. Also I think this Exception should be caught and not lead to a fatal error for the synchronization.