search

fuzziqersoftware / phosg

C++ helpers for some common tasks
18 stars 17 forks source link

JSONPickleTest - UnicodeDecodeError with python3.9 #9

Closed einstein95 closed 3 years ago

einstein95 commented 3 years ago 
$ ./JSONPickleTest
-- null
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse serialized representation)
-- null (python2 deserialize; unsorted)
terminate called after throwing an instance of 'std::runtime_error'
  what():  command returned code 256
stdout:

stderr:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Aborted

Full build log:

$ make
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Concurrency.o Concurrency.ccg++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o ConsistentHashRing.o ConsistentHashRing.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Encoding.o Encoding.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o FileCache.o FileCache.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Filesystem.o Filesystem.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Hash.o Hash.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Image.o Image.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o JSONPickle.o JSONPickle.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o JSON.o JSON.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Network.o Network.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Process.o Process.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Random.o Random.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Strings.o Strings.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o Time.o Time.cc
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o UnitTest.o UnitTest.cc
rm -f libphosg.a
ar rcs libphosg.a Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o JSONFormat.o JSONFormat.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread JSON.o JSONPickle.o Strings.o Filesystem.o Process.o Time.o JSONFormat.o  -o jsonformat
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o ConsistentHashRingTest.o ConsistentHashRingTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread ConsistentHashRingTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o ConsistentHashRingTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o EncodingTest.o EncodingTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread EncodingTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o EncodingTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o FileCacheTest.o FileCacheTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread FileCacheTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o FileCacheTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o FilesystemTest.o FilesystemTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread FilesystemTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o FilesystemTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o HashTest.o HashTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread HashTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o HashTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o ImageTest.o ImageTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread ImageTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o ImageTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o JSONPickleTest.o JSONPickleTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread JSONPickleTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o JSONPickleTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o JSONTest.o JSONTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread JSONTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o JSONTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o KDTreeTest.o KDTreeTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread KDTreeTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o KDTreeTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o LRUSetTest.o LRUSetTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread LRUSetTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o LRUSetTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o ProcessTest.o ProcessTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread ProcessTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o ProcessTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o StringsTest.o StringsTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread StringsTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o StringsTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o TimeTest.o TimeTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread TimeTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o TimeTest
g++ -fPIC -std=c++14 -g -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -Werror -DLINUX   -c -o UnitTestTest.o UnitTestTest.cc
g++ -fPIC -g -std=c++14 -lstdc++ -pthread UnitTestTest.o Concurrency.o ConsistentHashRing.o Encoding.o FileCache.o Filesystem.o Hash.o Image.o JSONPickle.o JSON.o Network.o Process.o Random.o Strings.o Time.o UnitTest.o  -o UnitTestTest
./ConsistentHashRingTest
-- create ring with one host
-- create ring with three hosts; make sure all points are set
-- create ring with two hosts; check point balance
-- check host removal affecting other hosts
-- check that host order doesn't matter
all tests passed
./EncodingTest
./EncodingTest: all tests passed
./FileCacheTest
./FileCacheTest: all tests passed
./FilesystemTest
./FilesystemTest: all tests passed
./HashTest
-- fnv1a64
-- md5
-- sha1
./HashTest: all tests passed
./ImageTest
-- [8-bit] metadata
-- [8-bit] clear
-- [8-bit] axis-aligned lines
-- [8-bit] rectangles
-- [8-bit] non-axis-aligned lines
-- [8-bit] blit
-- [8-bit] mask_blit with color mask
-- [8-bit] copy
-- [8-bit/ppm] save as ppm in memory
-- [8-bit/ppm] save as ppm to disk
-- [8-bit/ppm] compare with reference
-- [8-bit/bmp] save as ppm in memory
-- [8-bit/bmp] save as ppm to disk
-- [8-bit/bmp] compare with reference
-- [16-bit] metadata
-- [16-bit] clear
-- [16-bit] axis-aligned lines
-- [16-bit] rectangles
-- [16-bit] non-axis-aligned lines
-- [16-bit] blit
-- [16-bit] mask_blit with color mask
-- [16-bit] copy
-- [16-bit/ppm] save as ppm in memory
-- [16-bit/ppm] save as ppm to disk
-- [16-bit/ppm] compare with reference
-- [32-bit] metadata
-- [32-bit] clear
-- [32-bit] axis-aligned lines
-- [32-bit] rectangles
-- [32-bit] non-axis-aligned lines
-- [32-bit] blit
-- [32-bit] mask_blit with color mask
-- [32-bit] copy
-- [32-bit/ppm] save as ppm in memory
-- [32-bit/ppm] save as ppm to disk
-- [32-bit/ppm] compare with reference
-- [64-bit] metadata
-- [64-bit] clear
-- [64-bit] axis-aligned lines
-- [64-bit] rectangles
-- [64-bit] non-axis-aligned lines
-- [64-bit] blit
-- [64-bit] mask_blit with color mask
-- [64-bit] copy
-- [64-bit/ppm] save as ppm in memory
-- [64-bit/ppm] save as ppm to disk
-- [64-bit/ppm] compare with reference
./ImageTest: all tests passed
./JSONPickleTest
-- null
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse serialized representation)
-- null (python2 deserialize; unsorted)
terminate called after throwing an instance of 'std::runtime_error'
  what():  command returned code 256
stdout:

stderr:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

make: *** [Makefile:50: test] Aborted
rm HashTest.o EncodingTest.o ImageTest.o LRUSetTest.o TimeTest.o KDTreeTest.o ConsistentHashRingTest.o ProcessTest.o JSONPickleTest.o FilesystemTest.o FileCacheTest.o StringsTest.o UnitTestTest.o JSONTest.o
fuzziqersoftware commented 3 years ago

Do you have python aliased to Python 3? This test runs python as a subprocess assuming it's Python 2.

This reminds me, though, that I should drop support for Python 2 since I've done so in most of my other projects. I'll do so here soon.

fuzziqersoftware commented 3 years ago

I dropped support for Python 2 in 43ce20c2, which should fix the build for you since it will no longer attempt these test cases. Feel free to reopen this issue if you run into other problems.

einstein95 commented 3 years ago

Another issue, same test

./JSONPickleTest
-- null
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse pickled representation)
-- null (parse serialized representation)
-- null (python3 deserialize; unsorted)
-- null (python3 serialize; protocol 0)
terminate called after throwing an instance of 'std::runtime_error'
  what():  command returned code 256
stdout:

stderr:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: pickle protocol must be <= 5

make: *** [Makefile:50: test] Aborted