client sync issues

[lukastribus]

Hello,

I have peeringdb-py 0.6.1 from pipi:

lukas@dev:~/peeringdb$ cat ~/.peeringdb/config.yaml
database:
  engine: sqlite3
  name: peeringdb.sqlite3
peeringdb:
  password: ''
  timeout: 0
  url: https://www.peeringdb.com/api
  user: ''
lukas@dev:~/peeringdb$ pipenv run pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
certifi==2019.9.11
chardet==3.0.4
Click==7.0
Django==1.8.19
django-countries==5.5
django-handleref==0.1.5
django-inet==0.3.2
django-peeringdb==0.4.0
future==0.18.0
idna==2.8
ipaddress==1.0.22
munge==0.6.0
peeringdb==0.6.1
PyYAML==5.1.2
requests==2.22.0
six==1.12.0
twentyc.rpc==0.3.5
urllib3==1.25.6
lukas@dev:~/peeringdb$

Initial sync to sqlite3 takes more than 3 hours after which it fails in with a 404 Not Found error:

{'org': [u'organization instance with id 21434 does not exist.']} : errors: {'org': [u'organization instance with id 21434 does not exist.']}
org: Missing Object, dict: {'_validators': [], 'auto_created': False, 'serialize': True, '_unique': False, 'unique_for_year': None, 'blank': False, 'help_text': u'', 'null': False, 'to_fields': ['id'], 'db_tablespace': '', 'db_index': True, 'is_relation': True, 'unique_for_month': None, 'unique_for_date': None, 'primary_key': False, 'concrete': True, 'swappable': True, 'max_length': None, 'rel': <ManyToOneRel: django_peeringdb.network>, 'from_fields': [u'self'], 'verbose_name': u'org', '_choices': [], 'creation_counter': 86, 'validators': [], 'editable': True, 'related_model': <class 'django_peeringdb.models.concrete.Organization'>, 'error_messages': {u'unique': <django.utils.functional.__proxy__ object at 0x7f2b682e1810>, u'invalid': <django.utils.functional.__proxy__ object at 0x7f2b682682d0>, u'invalid_choice': <django.utils.functional.__proxy__ object at 0x7f2b689efb90>, u'blank': <django.utils.functional.__proxy__ object at 0x7f2b682e17d0>, u'null': <django.utils.functional.__proxy__ object at 0x7f2b682e1790>, u'unique_for_date': <django.utils.functional.__proxy__ object at 0x7f2b682e1850>}, '_related_fields': [(<django.db.models.fields.related.ForeignKey: org>, <django.db.models.fields.AutoField: id>)], '_error_messages': None, 'db_constraint': True, '_verbose_name': None, 'name': 'org', 'db_column': None, 'default': <class django.db.models.fields.NOT_PROVIDED at 0x7f2b68ad2bb0>, 'attname': u'org_id', 'column': u'org_id', 'model': <class 'django_peeringdb.models.concrete.Network'>, 'opts': <Options for Network>}
org.21434 not found locally, trying to fetch object...
Starting new HTTPS connection (1): www.peeringdb.com:443
https://www.peeringdb.com:443 "GET /api/org/21434?depth=0 HTTP/1.1" 404 33
Traceback (most recent call last):
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/bin/peeringdb", line 10, in <module>
    sys.exit(cli())
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/peeringdb/cli.py", line 164, in sync
    db.sync()
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/peeringdb/localdb.py", line 124, in sync
    call_command('pdb_sync', interactive=False)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 92, in handle
    self.sync(tables, pk, limit=limit)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 99, in sync
    self.update_db(cls, self.get_objs(cls, pk=pk, **kwargs))
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 178, in update_db
    self._sync(cls, row)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/django_peeringdb/management/commands/pdb_sync.py", line 162, in _sync
    r = self.rpc.get(field, int(m.group(1)), depth=0)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/twentyc/rpc/client.py", line 114, in get
    return self._load(self._request(typ, id=id, params=kwargs))
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/twentyc/rpc/client.py", line 85, in _load
    self._throw(res, data)
  File "/home/lukas/.local/share/virtualenvs/peeringdb-3r7qYvHI/local/lib/python2.7/site-packages/twentyc/rpc/client.py", line 65, in _throw
    raise NotFoundException("%d %s" % (res.status_code, err))
twentyc.rpc.client.NotFoundException: 404 Not found.

It is unclear:

• is the initial sync supposed to run for more than 3 hours?
• did the sync abort when hitting the 404 (did the traceback abort the entire job, or just part of it)?
• does peeringdb-py handle those 404 error gracefully, or is the resulting database out of sync?
• I can see some refactoring is in the tree, but hasn't released to pipi; can you we get this released or is something blocking this?

[bcavns01]

For reference, this looks to be the same issue as these: https://github.com/peeringdb/peeringdb-py/issues/21 https://github.com/peeringdb/peeringdb/issues/574

[arnoldnipper]

Is that still relevant, @lukastribus?

[lukastribus]

Closing.