Datetime.time to bson convertion fails in mongodb

[phipse]

First Traceback, second print of feature.name and feature attribute.

Either the datetime type has to get changed in the feature or it must be decoded to something readable for the database in the summarist.

I would prefer the latter, as the datetime format is easily convertible.

Maybe the Summarist should do parameter checks on every data, before it is written into the database.

EDIT: Reproduction:

• $ export PYTHONPATH=/path/to/complexlab_ai/:$PYTHONPATH
• $ python init.py -- in src/starter/
• to get the print msgs add 'print feature.name; print feature.attributes;' in summarist/init.py

First: ------------------------------------------------- Traceback (most recent call last): File "init.py", line 78, in extractorStream( extractDataPath, crawlerList ) File "init.py", line 31, in extractorStream summ.process( extractResult.itervalues().next() ) File "/home/phipse/Uni/Hauptstudium/complexlab_ai/src/summarist/init.py", line 9, in process self.db.features.insert({"name": feature.name, "attributes": feature.attributes}) # TODO ensureIndex(name, id) File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 359, in insert continue_on_error, self.uuid_subtype), safe) File "/usr/lib64/python2.7/site-packages/pymongo/message.py", line 80, in insert encoded = [bson.BSON.encode(doc, check_keys, uuid_subtype) for doc in docs] File "/usr/lib64/python2.7/site-packages/bson/__init.py", line 567, in encode return cls(_dict_to_bson(document, check_keys, uuid_subtype)) File "/usr/lib64/python2.7/site-packages/bson/init.py", line 476, in _dict_to_bson elements.append(_element_to_bson(key, value, check_keys, uuid_subtype)) File "/usr/lib64/python2.7/site-packages/bson/init.py", line 409, in _element_to_bson return BSONARR + name + _dict_to_bson(as_dict, check_keys, uuid_subtype, False) File "/usr/lib64/python2.7/site-packages/bson/init.py", line 476, in _dict_to_bson elements.append(_element_to_bson(key, value, check_keys, uuid_subtype)) File "/usr/lib64/python2.7/site-packages/bson/init.py", line 466, in _element_to_bson type(value)) bson.errors.InvalidDocument: cannot convert value of type <type 'datetime.date'> to bson

Second:------------------------------------------------------------ None ['decreasing', datetime.date(2004, 2, 25), datetime.date(2004, 2, 27), -0.25] None ['decreasing', datetime.date(2012, 2, 10), datetime.date(2012, 2, 22), -2.7300000000000004] None ['decreasing', datetime.date(1984, 9, 10), datetime.date(1984, 9, 13), -0.05999999999999961] None ['decreasing', datetime.date(1992, 12, 17), datetime.date(1992, 12, 23), -0.10999999999999988] None ['increasing', datetime.date(1984, 9, 12), datetime.date(1984, 9, 25), -0.370000000000001] None ['increasing', datetime.date(2005, 11, 9), datetime.date(2005, 11, 15), -0.75] None ['increasing', datetime.date(2008, 8, 19), datetime.date(2008, 8, 22), -0.04000000000000625] None ['increasing', datetime.date(1992, 1, 23), datetime.date(1992, 1, 24), -1.0] None ['increasing', datetime.date(1994, 5, 19), datetime.date(1994, 5, 20), -0.5] None ['decreasing', datetime.date(2009, 12, 11), datetime.date(2009, 12, 16), -0.10000000000000053]

[phipse]

Quick'n'Dirty: add str() around feature.attributes: self.db.features.insert({"name": feature.name, "attributes": str(feature.attributes)})

Next problem: File "/home/phipse/Uni/Hauptstudium/complexlab_ai/src/summarist/init.py", line 10, in process if self.db.features.count % 1000: TypeError: unsupported operand type(s) for %: 'instancemethod' and 'int'

[phipse]

... self.db.features.count() nicht self.db.features.count

[phipse]

Summary: str(Datetime.date) solves the bson conversion issue, but this code isn't even valid python code. So you can you please go back and fix it? You can test it by executing 'python init.py in src/starter/, it will crash in the summarist.

[pschwede]

is it fixed now?

[farao]

Yes and no, phipse just put a str() around it in the summarist, which makes it work/not fail, but which is not what we want :-) Is there a integer- or in general number representation of Dates, that we can use? Maybe we can already use this number representation in the mask, when generating the feature? Or we find a datetime-class that is accepted and converted by mongodb....

Am 30.03.2013 11:37, schrieb Peter Schwede:

is it fixed now?

— Reply to this email directly or view it on GitHub https://github.com/phipse/complexlab_ai/issues/12#issuecomment-15672526.

[pschwede]

>>> int(float(datetime.date(2013,3,30).strftime("%s.%f"))*1000)  # UTC datetime[1]
1364598000000

[1] http://bsonspec.org/#/specification