histrio / py-couchdb

Modern pure python CouchDB Client.
https://pycouchdb.readthedocs.org/
Other
120 stars 43 forks source link

TypeError: __init__() missing 1 required positional argument: 'history' #71

Open dhulke opened 6 years ago

dhulke commented 6 years ago

I have a web crawling app that creates a buffer of about 500 documents and bulk saves them to couch. Whenever I'm getting too many page errors, all of a sudden, for some unknown reason, this error pops up. I don't understand why page load errors would interfere with bulk save since I only buffer documents that have been correctly loaded. Any idea why this is happening?

Traceback (most recent call last): File "fetch.py", line 28, in fetcher.fetch() File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 31, in fetch loop.run_until_complete(self.run()) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", line 467, in run_until_complete return future.result() File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 43, in run await asyncio.gather(all_jobs) File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 55, in cnpj_job self.flush() File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\fetcher.py", line 63, in flush self.cnpjcouchdb.save_bulk(list(self.couch_buffer)) File "C:\Users\danilo\Projects\CNPJReceita\cnpjreceita\cnpjcouchdb.py", line 25, in save_bulk return self.db.save_bulk(docs) File "C:\Users\danilo\Projects\CNPJReceita\venv\lib\site-packages\pycouchdb\client.py", line 403, in save_bulk _docs = copy.deepcopy(docs) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy y = copier(x, memo) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 215, in _deepcopy_list append(deepcopy(a, memo)) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 150, in deepcopy y = copier(x, memo) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 240, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 180, in deepcopy y = _reconstruct(x, memo, rv) File "C:\Users\danilo\AppData\Local\Programs\Python\Python36\lib\copy.py", line 274, in _reconstruct y = func(*args) TypeError: init() missing 1 required positional argument: 'history'

histrio commented 6 years ago

Hi, @dhulke ! Can you show what kind of object you trying to save as bulk? It seems to be some issues in object reconstruction. We need to make it reproducible. Thanks!

dhulke commented 6 years ago

I haven't been able to find a consistent way to reproduce this error. Some times it happens very frequently and other times it doesn't happen at all for over a week. All I have are simple json objects I'm retrieving through aiohttp response.json(). Some of the text has weird characters, but that isn't consistently causing this error. I tried looking for the reduce functions which are getting called at y = func(*args) but I haven't found a single one that requires a history attribute. Is there any other way we could try and find the cause for this without being able to consistently reproduce it?

dhulke commented 6 years ago

Or if we at least knew what function expects the history attribute we could probably infer what the error is