ValueError: Circular reference detected

[emirot]

I'm getting a circular reference detected using a simple example.

python --version : Python 3.6.4

Code :

import records
db = records.Database('postgres://postgres:postgres@localhost:5432/postgres')
rows = db.query('select * from employees')
print(rows[0])

pip freeze :

click==6.7
docopt==0.6.2
et-xmlfile==1.0.1
Flask==1.0.2
itsdangerous==0.24
jdcal==1.4
Jinja2==2.10
MarkupSafe==1.0
odfpy==1.3.6
openpyxl==2.5.7
psycopg2==2.7.5
PyYAML==3.13
records==0.5.2
SQLAlchemy==1.2.11
tablib==0.12.1
unicodecsv==0.14.1
Werkzeug==0.14.1
xlrd==1.1.0
xlwt==1.3.0

Error :

/Users/turo/Documents/repositories/test/nolanenv/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installin
g from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "server.py", line 14, in <module>
    print(str(rows[0]))
  File "/Users/turo/Documents/repositories/test/nolanenv/lib/python3.6/site-packages/records.py", line 45, in __repr__
    return '<Record {}>'.format(self.export('json')[1:-1])
  File "/Users/turo/Documents/repositories/test/nolanenv/lib/python3.6/site-packages/records.py", line 96, in export
    return self.dataset.export(format, **kwargs)
  File "/Users/turo/Documents/repositories/test/nolanenv/lib/python3.6/site-packages/tablib/core.py", line 467, in export
    return export_set(self, **kwargs)
  File "/Users/turo/Documents/repositories/test/nolanenv/lib/python3.6/site-packages/tablib/formats/_json.py", line 30, in export_set
    return json.dumps(dataset.dict, default=date_handler)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

[plombardi89]

Also running into this issue for a simple select * from <table> query. If I name individual columns in my table it works fine...

atomicwrites==1.1.5
attrs==18.1.0
certifi==2018.8.24
chardet==3.0.4
click==6.7
docopt==0.6.2
et-xmlfile==1.0.1
idna==2.7
jdcal==1.4
more-itertools==4.2.0
odfpy==1.3.6
openpyxl==2.5.7
pluggy==0.6.0
psycopg2-binary==2.7.5
py==1.5.4
pytest==3.6.3
pyyaml==3.13
records==0.5.2
requests==2.19.1
ruamel.yaml==0.15.70
six==1.11.0
sqlalchemy==1.2.12
tablib==0.12.1
unicodecsv==0.14.1
urllib3==1.23
xlrd==1.1.0
xlwt==1.3.0

[emirot]

@plombardi89 it might be an issue in https://github.com/kennethreitz/tablib

[plombardi89]

Yea it's definitely an issue down in tablib, but it's occurring in this library. It actually appears to be a date parsing issue based on some limited hacking I did in _json.py in tablib.

[SupermanIsEvious]

Yes, this is a result of the date_parser function used in tablib for json encoding. Functions used for extending default behavior in JSONEncoder should never return the object unchanged.

In my case, I had converted an entry into a set, which resulted in this error. You can work around the bug in tablib by converting the value causing the error into a Python type that is JSON serializable (in my case, for instance, I converted it to a list).

[tym-xqo]

I don't think it's necessarily the only case, but I happened to notice this issue coming up particularly when the query results include a Postgres interval data type column, which gets mapped to datetime.timedelta on the Python side. I think the 'circular reference' error message is misleading - I suspect this error occurs whenever a result is not caught by the special tablib parsing for decimals, UUIDs and datetimes, but still is not JSON serializable. 😕

[tym-xqo]

Related: I'm also not clear why records is passing stuff through the tablib JSON serializer when not actually exporting to JSON 🤔

Edit: oh, it's built into the Record.__repr__ method. Huh.