Upgrade to Python3

[rldleblanc]

Upgraded the code to work with Python3 (also works with Pypy3). UPnP was removed for now. The /web/pings is not working, I need more time to think about that. Everything else should be working. The code works with Python2 peers so you can use it with existing peers.

Please test the code and help shake out any other bugs. I don't recommend running this on your mainnet coins until more people have tested it.

Some performance numbers (17294 Bitcoin shares load time from disk)

	Python2	Python2 w/ jemalloc	Pypy2	Pypy2 w/ jemalloc	Python3	Python3 w/ jemalloc	Pypy3	Pypy3 w/ jemalloc
Run #1	57.852	56.044	15.383	15.097	52.064	52.06	19.674	18.851
Run #2	57.45	57.956	15.473	15.094	51.919	53.439	19.406	19.555
Run #3	57.505	57.346	15.169	15.902	52.185	52.897	19.074	18.592
Average	57.602	57.115	15.342	15.364	52.056	52.799	19.385	18.999
% reduction	0.00%	0.85%	73.37%	73.33%	9.63%	8.34%	66.35%	67.02%

Pyhon3 is almost 10% faster than Python2, but Pypy3 (6.0) is a little bit slower. I did not spend any time optimizing anything for Python3, just getting the codebase to run on Python3 right now. This code includes the code to strip out transactions with share version 34.

[rldleblanc]

I cleaned up the build process in preparation for the work on the share components. I've changed the setup.py so that it builds the ltc_scrypt module and makes sure dependencies are installed (Twisted for now). I've updated the code so that it works in the directory, or it can be installed system wide, in the user home directory or in a virtualenv to make things easier and more flexible. We can build distribution tarballs now with the sdist command as well.

I added a few other fixes to the website so it isn't crashing when there are less than 10 shares, it may not be 100% accurate, but better than nothing. I also added serving up the old P2Pool web interface at /orig when --web-static is specified.

I'm leaning towards ProtoBuff and gRPC for the peer communication and disk storage for the shares. This was the impitus for updating the build environment, so that the dependencies are automatically resolved and the protoc files are built automatically. I've got a few more tests, but I think we can still support all the transactions in the block by just passing the txids around or maybe even an abbreviated txid. That will make peer communications light and fast, but still allow peers to build the share/block and verify it against bitcoind.

Please let me know if anyone finds bugs along the way.

[rldleblanc]

Also, I removed the ability to build a Windows exe from the project. It wasn't documented and I don't know that it was used. I don't use Windows, but if it's a big deal then I can fire up a VM to make it work, but I'd rather focus on the shares.

With P2Pool being able to be installed now, we may want to think about the default location for the data directory. Instead of being at pwd maybe we want the default to be ~/p2pool-data/<network>/ or ~/.p2pool-data/<network>/ instead. What are your thoughts?

[komato3huk]

Sory but i not install to test builtins.ImportError: No module named 'p2pool.pow.ltc_scrypt' Error while requesting best block header:

ImportError: No module named 'p2pool.pow.ltc_scrypt' Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult _inlineCallbacks(r, g, deferred) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) --- --- File "/home/pool/p2pool_dev/p2pool/p2pool/util/deferral.py", line 44, in f result = yield func(*args, **kwargs) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/home/pool/p2pool_dev/p2pool/p2pool/node.py", line 296, in poll_header self.bitcoind_work.value['previous_block']))) File "/home/pool/p2pool_dev/p2pool/p2pool/node.py", line 272, in handle_header bitcoin_data.block_header_type.pack(new_header)) \ File "/home/pool/p2pool_dev/p2pool/p2pool/bitcoin/networks/litecoin.py", line 24, in 'p2pool.pow.ltc_scrypt').getPoWHash(data)) File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 986, in _gcd_import File "", line 969, in _find_and_load File "", line 956, in _find_and_load_unlocked builtins.ImportError: No module named 'p2pool.pow.ltc_scrypt'

[rldleblanc]

Did you run 'python3 setup.py build' and then 'sudo python3 setup.py install'? Or did you install it into your user space or virtual environment? It looks like the Scrypt module was not built and installed. These are new instructions with the move to Python3. Be sure to check the readme file for more info.

Sent from a mobile device, please excuse any typos.

On Sat, Apr 20, 2019, 1:17 AM komato3huk notifications@github.com wrote:

Sory but i not install to test builtins.ImportError: No module named 'p2pool.pow.ltc_scrypt' Error while requesting best block header:

ImportError: No module named 'p2pool.pow.ltc_scrypt' Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult _inlineCallbacks(r, g, deferred) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) --- --- File "/home/pool/p2pool_dev/p2pool/p2pool/util/deferral.py", line 44, in f result = yield func(*args, **kwargs) File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/home/pool/p2pool_dev/p2pool/p2pool/node.py", line 296, in poll_header self.bitcoind_work.value['previous_block']))) File "/home/pool/p2pool_dev/p2pool/p2pool/node.py", line 272, in handle_header bitcoin_data.block_header_type.pack(new_header)) File "/home/pool/p2pool_dev/p2pool/p2pool/bitcoin/networks/litecoin.py", line 24, in 'p2pool.pow.ltc_scrypt').getPoWHash(data)) File "/usr/lib/python3.5/importlib/*init*.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 986, in _gcd_import File "", line 969, in _find_and_load File "", line 956, in _find_and_load_unlocked builtins.ImportError: No module named 'p2pool.pow.ltc_scrypt' — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[komato3huk]

Yes of course I did it.

this is after python setup.py install

[rldleblanc]

Hmm... Let's see if we can figure out where things went bad.

Please run the two commands again (pass -v, I think there is a verbose or debug flag for setup.py) and post the output and see what shows up there.

Sent from a mobile device, please excuse any typos.

On Sat, Apr 20, 2019, 8:36 AM komato3huk notifications@github.com wrote:

Yes of course I did it.

this is after python setup.py install

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jtoomim/p2pool/pull/25#issuecomment-485131294, or mute the thread https://github.com/notifications/unsubscribe-auth/ACW6TV2OAOBYXF434XK4GYDPRMS5JANCNFSM4GS4ODFA .

[komato3huk]

pool@pool:~/p2pool_dev/p2pool$ sudo python3 setup.py install -v
/usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
/usr/lib/python3/dist-packages/setuptools/dist.py:294: UserWarning: The version specified ('16.0-145-g94e6457') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running install
Checking .pth file support in /usr/local/lib/python3.5/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.5/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing P2Pool.egg-info/PKG-INFO
writing entry points to P2Pool.egg-info/entry_points.txt
writing top-level names to P2Pool.egg-info/top_level.txt
writing dependency_links to P2Pool.egg-info/dependency_links.txt
writing requirements to P2Pool.egg-info/requires.txt
reading manifest file 'P2Pool.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'P2Pool.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/p2pool
creating build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/pow/scrypt.c -> build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/pow/ltc_scrypt.cpython-35m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/pow/scryptmodule.c -> build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/pow/__init__.py -> build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/pow/scrypt.h -> build/bdist.linux-x86_64/egg/p2pool/pow
copying build/lib.linux-x86_64-3.5/p2pool/work.py -> build/bdist.linux-x86_64/egg/p2pool
copying build/lib.linux-x86_64-3.5/p2pool/p2p.py -> build/bdist.linux-x86_64/egg/p2pool

[rldleblanc]

I think I see what the problem is, I think there is a bug in the code where it is trying to open a file in the system area read/write to update the version information and the regular user can't do that.

Try this to see if it works okay, and I'll try to fix this bug.

python3 -m virtualenv ~/testvirtenv
cd <p2pool_source_dir>
~/testvirtenv/bin/python3 setup.py install
~/testvirtenv/bin/python3 run_p2pool.py ...

This will work around the bug by installing everything needed in your home directory with permissions that won't fail because it will not install things into the system directories.

Sorry about the bug, this is my first time working with setuptools.

[rldleblanc]

Okay, I think I fixed this bug if you want to try it out, just checkout again, build and install.

[komato3huk]

Okay, I think I fixed this bug if you want to try it out, just checkout again, build and install.

Nice. Updated repository. Error with the LTC module is preserved. building 'p2pool.pow.ltc_scrypt' extension creating build/temp.linux-x86_64-3.5 creating build/temp.linux-x86_64-3.5/p2pool creating build/temp.linux-x86_64-3.5/p2pool/pow x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c p2pool/pow/scryptmodule.c -o build/temp.linux-x86_64-3.5/p2pool/pow/scryptmodule.o p2pool/pow/scryptmodule.c: In function ‘scrypt_getpowhash’: p2pool/pow/scryptmodule.c:15:5: warning: implicit declaration of function ‘scrypt_1024_1_1_256’ [-Wimplicit-function-declaration] scrypt_1024_1_1_256((char *)PyBytes_AsString((PyObject*) input), output); ^ x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c p2pool/pow/scrypt.c -o build/temp.linux-x86_64-3.5/p2pool/pow/scrypt.o x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/p2pool/pow/scryptmodule.o build/temp.linux-x86_64-3.5/p2pool/pow/scrypt.o -o build/lib.linux-x86_64-3.5/p2pool/pow/ltc_scrypt.cpython-35m-x86_64-linux-gnu.so Script make modeule ltc_scrypt.cpython-35m-x86_64-linux-gnu.so. Need builtins.ImportError: No module named 'p2pool.pow.ltc_scrypt'