search

rheostat2718 / unladen-swallow

Automatically exported from code.google.com/p/unladen-swallow
Other
0 stars 0 forks source link

Cannot incrementally unpickle from a file-like object #60

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
The latest development version of Django's tests fail to pass under unladen
swallow, specifically the cache tests with the following errors:

======================================================================
ERROR: test_add (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
144, in test_add
    self.assertEqual(self.cache.get("addkey1"), "value")
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_data_types (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
212, in test_data_types
    self.assertEqual(self.cache.get("stuff"), stuff)
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_decr (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
194, in test_decr
    self.assertEqual(self.cache.decr('answer'), 42)
  File "/home/alex/python_path_sym/django/core/cache/backends/base.py",
line 84, in decr
    return self.incr(key, -delta)
  File "/home/alex/python_path_sym/django/core/cache/backends/base.py",
line 75, in incr
    new_value = self.get(key) + delta
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_delete (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
165, in test_delete
    self.assertEqual(self.cache.get("key1"), "spam")
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_expiration (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
224, in test_expiration
    self.assertEqual(self.cache.get("expire2"), "newvalue")
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_get_many (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
158, in test_get_many
    self.assertEqual(self.cache.get_many(['a', 'c', 'd']), {'a' : 'a', 'c'
: 'c', 'd' : 'd'})
  File "/home/alex/python_path_sym/django/core/cache/backends/base.py",
line 57, in get_many
    val = self.get(k)
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_incr (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
185, in test_incr
    self.assertEqual(self.cache.incr('answer'), 42)
  File "/home/alex/python_path_sym/django/core/cache/backends/base.py",
line 75, in incr
    new_value = self.get(key) + delta
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_simple (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
137, in test_simple
    self.assertEqual(self.cache.get("key"), "value")
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

======================================================================
ERROR: test_unicode (regressiontests.cache.tests.FileBasedCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/alex/django_src/tests/regressiontests/cache/tests.py", line
237, in test_unicode
    self.assertEqual(self.cache.get(key), value)
  File
"/home/alex/python_path_sym/django/core/cache/backends/filebased.py", line
50, in get
    return pickle.load(f)
RuntimeError: Ran out of input

----------------------------------------------------------------------

Original issue reported on code.google.com by alex.gay...@gmail.com on 13 Jun 2009 at 10:01

GoogleCodeExporter commented 9 years ago 
I have seen the exact same error on the ZODB3 tests (the object database which 
uses pickle protocol one to 
persist Python objects).

In that case they only happened for special C extension types and I wasn't able 
to craft a minimal test case 
showing the problem.

Original comment by hanno...@gmail.com on 14 Jun 2009 at 1:46

GoogleCodeExporter commented 9 years ago 
Yeah, this is a known issue with the changes we made to the cPickle module. 
I'll try
to get a fix in this week.

Original comment by collinw on 15 Jun 2009 at 6:50

GoogleCodeExporter commented 9 years ago 
Patch ready, need to test it against Django trunk@head.

Original comment by collinw on 27 Jun 2009 at 3:02

GoogleCodeExporter commented 9 years ago 
Django trunk@head now passes at a cost of 10% unpickling performance. Oh well. 
Patch 
mailed for review.

Original comment by collinw on 27 Jun 2009 at 7:24

GoogleCodeExporter commented 9 years ago 
Issue 7 has been merged into this issue.

Original comment by collinw on 29 Jun 2009 at 10:01

GoogleCodeExporter commented 9 years ago 
Part of the problem was fixed in r686. There's still a problem with reading 
from real
files, as opposed to file-like objects, that probably has to do with EOF 
caching.

Django trunk@head now passes, though. ZODB still shows regressions, as does 
cvs2svn
(which tests all kinds of corner cases).

Original comment by collinw on 1 Jul 2009 at 11:18

GoogleCodeExporter commented 9 years ago 
The actual problem was that we need to hold an extra reference to each object 
in the 
Pickler's memo for as long as the Pickler object lives. A fixed has been 
mailed, which 
makes cvs2svn's test suite pass.

Original comment by collinw on 3 Jul 2009 at 12:12

GoogleCodeExporter commented 9 years ago 
The refcount bug was fixed in r690. That sucked to find.

Original comment by collinw on 4 Jul 2009 at 3:57