search

HelloZeroNet / ZeroNet

ZeroNet - Decentralized websites using Bitcoin crypto and BitTorrent network
https://zeronet.io
Other
18.33k stars 2.27k forks source link

ZeroNet fails on PyPy #351

Open kseistrup opened 8 years ago

kseistrup commented 8 years ago

TL;DR: It's almost there, I believe (PyPy 4.0.1 on Linux):

$ # Get a fresh copy of ZeroNet
$ git clone https://github.com/HelloZeroNet/ZeroNet
$ cd ZeroNet
$ # Install the msgpack-python and gevent modules
$ # (We already had a working copy of pip and virtualenv)
$ /opt/pypy/bin/pip install -r requirements.txt
$ # Create a virtual environment
$ /opt/pypy/bin/virtualenv venv
$ source venv/bin/activate
(venv) $ python zeronet.py
- Starting ZeroNet...
[18:36:04] - OpenSSL loaded, version: 01000207F
[18:36:05] - Version: 0.3.6 r989, Python 2.7.10 (5f8302b8bf9f, Mar 04 2016, 08:29:09)
[PyPy 4.0.1 with GCC 5.3.0], Gevent: 1.1.0
[18:36:05] - Creating FileServer....
[18:36:05] TorManager Tor controller connect error: error: [Errno 111] Connection refused in TorManager.py line 151 > _socket2.py line 228
[18:36:05] - Creating UiServer....
[18:36:05] Site:1Name2..hM9F Content.json not exist: data/1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F/content.json
[18:36:05] - Removing old SSL certs...
[18:36:05] - Starting servers....
[18:36:05] Ui.UiServer --------------------------------------
[18:36:05] Ui.UiServer Web interface: http://127.0.0.1:43110/
[18:36:05] Ui.UiServer --------------------------------------
[18:36:05] Site:1Name2..hM9F Content.json not exist: data/1Name2NXVi1RDPDgf5617UoW7xA6YrhM9F/content.json
[18:36:06] FileServer Checking port 15441 using portchecker.co...
[18:36:07] FileServer [OK :)] Port open: Port 15441 is open.
[18:36:10] Ui.UiServer Wrapper key not found: 2948bfa27342bcc0fdabed5de2b53b9eafbce2a2ae5e7db650ab4d8c5bde9c28
[18:36:22] Db:TestDb Table keyvalue outdated...version: False need: 1, rebuilding...
[18:36:22] Db:TestDb Table json outdated...version: 0 need: 1, rebuilding...
[18:36:22] Db:TestDb Table test outdated...version: 0 need: 1426195822, rebuilding...
[18:36:27] - UiWSGIHandler error: OperationalError: database is locked in UiServer.py line 39 > pywsgi.py line 871 > pywsgi.py line 860 > StatsPlugin.py line 547 > Db.py line 119 > Db.py line 89 > Db.py line 60 > DbCursor.py line 53 > _sqlite3.py line 700 > _sqlite3.py line 857 > _sqlite3.py line 795 > _sqlite3.py line 181 > _sqlite3.py line 1026

In addition to the locked database as seen above, I also got an msgpack exception that I didn't catch when I ran the benchmark.

Will try again when PyPy 5.0 reaches ArchLinux.

kseistrup commented 8 years ago

Update: ZeroNet also fails on PyPy 5.0. However the benchmarks on PyPy 5.0 (providing Python 2.7.10) compared with those from vanilla Python 2.7.11 on the same machine are interesting:

PyPy 5.0

Benchmarking ZeroNet 0.3.6 (rev989) Python 2.7.10 (246c9cf22037, Mar 11 2016, 12:05:36) [PyPy 5.0.0 with GCC 5.3.0] on: linux2...

CryptBitcoin:
- hdPrivatekey x 10..........0.225s [x3.11: WOW]
- sign x 10..........0.085s [x4.10: Insane!!]
- openssl verify x 100..........0.359s [x1.03: OK]
- pure-python verify x 10..........0.301s [x5.32: Insane!!]

CryptHash:
- sha256 5M x 10..........0.305s [x1.96: Fast]
- sha512 5M x 10..........0.247s [x2.43: Fast]
- os.urandom(256) x 100 000..........1.913s [x0.34: Ehh]

Msgpack:
- pack 5K x 10 000..........
! Error: '\x84\xa3int\xce@\x00\x00\x00\xa5float\xcb@\xc8\x1c\xd6\xe61\xf8\xa1\xa4text\xda\x14\x00hellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohello\xa6binary\xda\x01\x00fqv\xf0\x1a"e\x10,\xbe\x9cT\x9e(\xa5]u\x072C\x8c\x15\xa2\xa8\x93Sw)\x19\x02\xdd\t\xfb\xf67\x88\xd9\xee\x86\xa1\xe4\xb6,\xc6\x14\xbb\xd7$z\x1d\xb2\xda\x85\xf5\xa0\x97^\x01*\xaf\xd3\xb0!\xb7\x9d\xea\x89\xbbh8\xa1"\xa7]e(@\xa2\xa5g\xb7[\xae\x8eE\xc2\x9fL\xb6s\x19\x19\r\xc8\x04S\xd0N\xe4]?/\x01\xea\xf6\xec\xd1\xb3\xc2\x91\x86\xd7\xf4K\xdf\xc2lV\xf4\xe8\x80\xfc\x8ep\xbb\x82\xb3\x86\x98F\x1c\xecS\xc8\x15\xcf\xdc\xf1\xed\xfc\xd8\x18r\xf9\x80\x0f\xfa\x8cO\x97(\x0b]\xf1\xdd\r\xe7\xbf\xed\x06\xbd\x1b?\xc5\xa0\xd7a\x82\xf3\xa8\xe6@\xf3\ri\xa1\xb10\xf6\xd4W\xbc\x86\x1a\xbb\xfd\x94!bS\xdb\xaeM\x92\x00#\x0b\xf7\xad\xe9\xc2\x8e\x86\xbfi![%\xd31]\xc6\xfc2\xc9\xda\xc6v\x82P\xcc\xa9\xea\xb9\xff\xf6\xc8\x17iD\xcf\xf3\xeeI\x04\xe9\xa1\x19\xbb\x01\x92\xf5nn4K\xf8\xbb\xc6\x17e>\xa7 \xbbv'
!=
'\x84\xa3int\xce@\x00\x00\x00\xa4text\xda\x14\x00hellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohellohello\xa5float\xcb@\xc8\x1c\xd6\xe61\xf8\xa1\xa6binary\xda\x01\x00fqv\xf0\x1a"e\x10,\xbe\x9cT\x9e(\xa5]u\x072C\x8c\x15\xa2\xa8\x93Sw)\x19\x02\xdd\t\xfb\xf67\x88\xd9\xee\x86\xa1\xe4\xb6,\xc6\x14\xbb\xd7$z\x1d\xb2\xda\x85\xf5\xa0\x97^\x01*\xaf\xd3\xb0!\xb7\x9d\xea\x89\xbbh8\xa1"\xa7]e(@\xa2\xa5g\xb7[\xae\x8eE\xc2\x9fL\xb6s\x19\x19\r\xc8\x04S\xd0N\xe4]?/\x01\xea\xf6\xec\xd1\xb3\xc2\x91\x86\xd7\xf4K\xdf\xc2lV\xf4\xe8\x80\xfc\x8ep\xbb\x82\xb3\x86\x98F\x1c\xecS\xc8\x15\xcf\xdc\xf1\xed\xfc\xd8\x18r\xf9\x80\x0f\xfa\x8cO\x97(\x0b]\xf1\xdd\r\xe7\xbf\xed\x06\xbd\x1b?\xc5\xa0\xd7a\x82\xf3\xa8\xe6@\xf3\ri\xa1\xb10\xf6\xd4W\xbc\x86\x1a\xbb\xfd\x94!bS\xdb\xaeM\x92\x00#\x0b\xf7\xad\xe9\xc2\x8e\x86\xbfi![%\xd31]\xc6\xfc2\xc9\xda\xc6v\x82P\xcc\xa9\xea\xb9\xff\xf6\xc8\x17iD\xcf\xf3\xeeI\x04\xe9\xa1\x19\xbb\x01\x92\xf5nn4K\xf8\xbb\xc6\x17e>\xa7 \xbbv'
0.143s [x5.46: Insane!!]
- unpack 5K x 10 000..........0.161s [x7.46: Insane!!]
- streaming unpack 5K x 10 000..........0.087s [x16.16: Insane!!]

Db:
- Open x 10
! Error: cannot commit transaction - SQL statements in progress
0.012s [x10.81: Insane!!]

Python 2.7.11

Benchmarking ZeroNet 0.3.6 (rev989) Python 2.7.11 (default, Mar 3 2016, 11:00:04) [GCC 5.3.0] on: linux2...

CryptBitcoin:
- hdPrivatekey x 10..........0.267s [x2.62: WOW]
- sign x 10..........0.171s [x2.05: Fast]
- openssl verify x 100..........0.157s [x2.36: Fast]
- pure-python verify x 10..........0.676s [x2.37: Fast]

CryptHash:
- sha256 5M x 10..........0.630s [x0.95: OK]
- sha512 5M x 10..........0.372s [x1.61: Fine]
- os.urandom(256) x 100 000..........2.400s [x0.27: Sloooow]

Msgpack:
- pack 5K x 10 000..........0.569s [x1.37: Fine]
- unpack 5K x 10 000..........0.940s [x1.28: Fine]
- streaming unpack 5K x 10 000..........0.821s [x1.70: Fast]

Db:
- Open x 10..........0.137s [x0.95: OK]
- Insert x 10 x 1000..........1.476s [x0.68: Goodish]
- Buffered insert x 100 x 100..........1.295s [x1.00: OK]
- Total rows in db: 20000
- Indexed query x 1000..........0.193s [x1.29: Fine]
- Not indexed query x 100..........0.544s [x1.10: OK]
- Like query x 100..........0.956s [x1.88: Fast]

Done. Total: 9.86s
kseistrup commented 8 years ago

PS: The vanilla benchmarks have been seen to take from 7.5 seconds to ~12 seconds on the same machine, so don't read too much from the absolute numbers.

kseistrup commented 8 years ago

PyPy 5.4.0 (Python 2.7.10) is out.

ZeroNet starts up well and sites like ZeroHello, ZeroName and ZeroBoard, as well as static sites) works fine. However, visit sites like ZeroTalk, ZeroBlog and ZeroMe, and you get a ton of SQL errors.

Let's call it status quo.

kseistrup commented 7 years ago

PyPy 5.6.0 (Python 2.7.12) is out.

This is far worse than with previous versions of PyPy: ZeroNet can't even start:

[22:34:51] - OpenSSL loaded, version: 0100020AF
[22:34:51] - Version: 0.5.0 r1704, Python 2.7.12 (aff251e54385, Nov 12 2016, 22:03:47)
[PyPy 5.6.0 with GCC 6.2.1 20160830], Gevent: 1.1.2
[22:34:51] Db:ContentDb Table keyvalue outdated...version: False need: 3, rebuilding...
[22:34:51] Db:ContentDb Table json outdated...version: 0 need: 3, rebuilding...
[22:34:51] Db:ContentDb Table site outdated...version: 0 need: 1, rebuilding...
[22:34:51] Db:ContentDb Table content outdated...version: 0 need: 1, rebuilding...
[22:34:51] Db:ContentDb Table file_optional outdated...version: 0 need: 11, rebuilding...
[22:34:51] Db:ContentDb Table peer outdated...version: 0 need: 1, rebuilding...
Traceback (most recent call last):
  File "zeronet.py", line 16, in main
    main.start()
  File "src/main.py", line 429, in start
    actions.call(config.action, action_kwargs)
  File "src/main.py", line 129, in call
    func(**kwargs)
  File "src/main.py", line 134, in main
    from File import FileServer
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/File/__init__.py", line 1, in <module>
    from FileServer import FileServer
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/File/FileServer.py", line 12, in <module>
    from Site import SiteManager
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Site/__init__.py", line 1, in <module>
    from Site import Site
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Site/Site.py", line 24, in <module>
    from Content import ContentManager
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Content/__init__.py", line 1, in <module>
    from ContentManager import ContentManager
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Content/ContentManager.py", line 15, in <module>
    from ContentDbDict import ContentDbDict
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Content/ContentDbDict.py", line 4, in <module>
    import ContentDb
  File "/tmp/pypy2/site-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "src/Content/ContentDb.py", line 137, in <module>
    getContentDb()  # Pre-connect to default one
  File "src/Content/ContentDb.py", line 134, in getContentDb
    content_dbs[path] = ContentDb(path)
  File "plugins/PeerDb/PeerDbPlugin.py", line 14, in __init__
    super(ContentDbPlugin, self).__init__(*args, **kwargs)
  File "plugins/OptionalManager/ContentDbPlugin.py", line 28, in __init__
    super(ContentDbPlugin, self).__init__(*args, **kwargs)
  File "src/Content/ContentDb.py", line 14, in __init__
    self.checkTables()
  File "plugins/OptionalManager/ContentDbPlugin.py", line 67, in checkTables
    changed_tables = super(ContentDbPlugin, self).checkTables()
  File "src/Db/Db.py", line 213, in checkTables
    cur.execute("COMMIT")
  File "src/Db/DbCursor.py", line 53, in execute
    res = self.cursor.execute(query)
  File "/opt/pypy/lib_pypy/_sqlite3.py", line 700, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/pypy/lib_pypy/_sqlite3.py", line 857, in execute
    return self.__execute(False, sql, [params])
  File "/opt/pypy/lib_pypy/_sqlite3.py", line 830, in __execute
    raise self.__connection._get_exception(ret)
OperationalError: cannot commit transaction - SQL statements in progress
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/pypy/lib-python/2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "plugins/PeerDb/PeerDbPlugin.py", line 84, in saveAllPeers
    for site in self.sites.values():
AttributeError: 'ContentDb' object has no attribute 'sites'
Exception AttributeError: AttributeError("'ContentDb' object has no attribute 'sites'",) in sys.exitfunc == <function _run_exitfuncs at 0x00007f7ce3f70980> ignored
- Starting ZeroNet...
[EXIT]
HelloZeroNet commented 7 years ago

In the recent versions of zeronet is much more dependent on sqlite (fast startup, peerdb, optional files db), so if it's not working correctly, then zeronet will not work at all, this is expected

ghost commented 7 years ago

this is expected

Closing ?

kseistrup commented 7 years ago

@border0464111,

The PyPy compatibility page reads:

Please just check if it imports. If it imports, it should work.

And import sqlite3 yields no errors. Also:

Supported, but written in pure Python:

  • […], sqlite3, […]

So I vote for keeping it open.

0polar commented 5 years ago

gevent do not work well with pypy

0polar commented 5 years ago

When you refactor to Python3, please also test about PyPy compatibility if possible.

Performance is important when seeding lots of sites concurrently, especially for public proxy.

HelloZeroNet commented 5 years ago

As far as I know gevent works on pypy. Based on the error message it had problem with sqlite. I will do a run when the new db module is done. But calling external modules is slower from pypy and right now there is only 32bit binary for windows. A few quick benchmark on 1000x signature verification:

0polar commented 5 years ago

Performance is not only about foreign method calls, internal method calls and memory footprint are also factors.

HelloZeroNet commented 5 years ago

I have no solid numbers, but I would say around 70% of the CPU usage comes from crypto (openssl) + sqlite. The memory footprint is always higher for an interpreter with JIT enabled.

0polar commented 5 years ago

This is the result when run on Windows:

- Starting ZeroNet...
[14:13:38] PluginManager Plugin Chart load error: ImportError: unable to load extension module 'C:\PyPy27\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found in PluginManager.py line 50 > builtins.py line 96 > __init__.py line 1 > builtins.py line 96 > ChartPlugin.py line 9 > builtins.py line 96 > ChartDb.py line 2 > builtins.py line 96 > __init__.py line 1 > builtins.py line 96 > Db.py line 1 > builtins.py line 96 > __init__.py line 24 > builtins.py line 96 > dbapi2.py line 28 > builtins.py line 96 > _sqlite3.py line 51 > builtins.py line 96
[14:13:38] - OpenSSL loaded, version: 0100020CF
[14:13:38] PluginManager Plugin Newsfeed load error: ImportError: unable to load extension module 'C:\PyPy27\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found in PluginManager.py line 50 > builtins.py line 96 > __init__.py line 1 > builtins.py line 96 > NewsfeedPlugin.py line 5 > builtins.py line 96 > __init__.py line 1 > builtins.py line 96 > Db.py line 1 > builtins.py line 96 > __init__.py line 24 > builtins.py line 96 > dbapi2.py line 28 > builtins.py line 96 > _sqlite3.py line 51 > builtins.py line 96
[14:13:38] PluginManager Plugin PeerDb load error: ImportError: unable to load extension module 'C:\PyPy27\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found in PluginManager.py line 50 > builtins.py line 96 > __init__.py line 1 > builtins.py line 96 > PeerDbPlugin.py line 2 > builtins.py line 96 > __init__.py line 24 > builtins.py line 96 > dbapi2.py line 28 > builtins.py line 96 > _sqlite3.py line 51 > builtins.py line 96
[14:13:38] - Unhandled exception: unable to load extension module 'C:\PyPy27\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found
Traceback (most recent call last):
  File "D:\Download\ZeroNet-master\zeronet.py", line 24, in main
    import main
  File "D:\Download\ZeroNet-master\src\main.py", line 71, in <module>
    PluginManager.plugin_manager.loadPlugins()
  File "D:\Download\ZeroNet-master\src\Plugin\PluginManager.py", line 58, in loadPlugins
    func()
  File "D:\Download\ZeroNet-master\plugins\AnnounceZero\AnnounceZeroPlugin.py", line 18, in importHostClasses
    from Site.SiteAnnouncer import AnnounceError
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Site\__init__.py", line 1, in <module>
    from Site import Site
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Site\Site.py", line 19, in <module>
    from Content import ContentManager
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Content\__init__.py", line 1, in <module>
    from ContentManager import ContentManager
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Content\ContentManager.py", line 16, in <module>
    from ContentDbDict import ContentDbDict
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Content\ContentDbDict.py", line 4, in <module>
    import ContentDb
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Content\ContentDb.py", line 4, in <module>
    from Db import Db
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Db\__init__.py", line 1, in <module>
    from Db import Db
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "D:\Download\ZeroNet-master\src\Db\Db.py", line 1, in <module>
    import sqlite3
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "C:\PyPy27\lib-python\2.7\sqlite3\__init__.py", line 24, in <module>
    from dbapi2 import *
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "C:\PyPy27\lib-python\2.7\sqlite3\dbapi2.py", line 28, in <module>
    from _sqlite3 import *
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
  File "C:\PyPy27\lib_pypy\_sqlite3.py", line 51, in <module>
    from _sqlite3_cffi import ffi as _ffi, lib as _lib
  File "C:\PyPy27\site-packages\gevent\builtins.py", line 96, in __import__
    result = _import(*args, **kwargs)
ImportError: unable to load extension module 'C:\PyPy27\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found

D:\Download>

I guess the monkey patch of gevent breaks sqlite3.

HelloZeroNet commented 5 years ago

Probably this is it: https://github.com/gevent/gevent/issues/1356

There is a new Python3 version avaliable here: https://github.com/HelloZeroNet/ZeroNet/tree/py3

You can try this one. I don't have C++compiler on my machine that is required to install gevent.

0polar commented 5 years ago

The following result is tested under clean install PyPy + manually fixed https://github.com/gevent/gevent/issues/1356.

PyPy2.7 on Linux

https://github.com/gevent/gevent/issues/1356 is not related to this issue. The error of this issue still exists.

PyPy2.7 on Windows

Once gevent monkey patch, permanently break sqlite3 (unless clean re-install). It is always broken, regardless of gevent.

Python 2.7.13 (9112c8071614, Feb 06 2019, 23:10:08)
[PyPy 7.0.0 with MSC v.1500 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\pypy2.7-v7.0.0-win32\lib-python\2.7\sqlite3\__init__.py", line 24, in <module>
    from dbapi2 import *
  File "C:\pypy2.7-v7.0.0-win32\lib-python\2.7\sqlite3\dbapi2.py", line 28, in <module>
    from _sqlite3 import *
  File "C:\pypy2.7-v7.0.0-win32\lib_pypy\_sqlite3.py", line 51, in <module>
    from _sqlite3_cffi import ffi as _ffi, lib as _lib
ImportError: unable to load extension module 'C:\pypy2.7-v7.0.0-win32\lib_pypy\_sqlite3_cffi.pypy-41.pyd': The specified module could not be found
>>>>

PyPy3.5 on Linux

https://github.com/tiran/pysha3/issues/22

PyPy3.5 on Windows

Once gevent monkey patch, permanently break sqlite3 (unless clean re-install). It is always broken, regardless of gevent.

Python 3.5.3 (928a4f70d3de, Feb 08 2019, 12:56:35)
[PyPy 7.0.0 with MSC v.1910 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\pypy3.5-v7.0.0-win32\lib-python\3\sqlite3\__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "C:\pypy3.5-v7.0.0-win32\lib-python\3\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
  File "C:\pypy3.5-v7.0.0-win32\lib_pypy\_sqlite3.py", line 52, in <module>
    from _sqlite3_cffi import ffi as _ffi, lib as _lib
ImportError: The specified module could not be found
>>>>

PyPy3.5 don't recognize Microsoft Visual C++, although I have installed.

  Failed building wheel for gevent
...
  Failed building wheel for pysha3
...
    error: Microsoft Visual C++ 14.1 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/
0polar commented 5 years ago

With manually install newer pysha3, the result:

root@vultr:~/ZeroNet-py3# pypy3 zeronet.py
- Starting ZeroNet...
[08:49:32] PluginManager Plugin Bigfile load error: ImportError: No module named 'merkletools' in PluginManager.py line 51 > Bigfile/__init__.py line 1 > Bigfile/BigfilePlugin.py line 19
[08:49:33] - Libsecpk256k1 loaded
[08:49:33] - Version: 0.7.0 r4017, Python 3.5.3 (928a4f70d3de, Feb 08 2019, 10:42:58)
[PyPy 7.0.0 with GCC 6.2.0 20160901], Gevent: 1.4.0
[08:49:33] - Creating FileServer....
[08:49:33] - Creating UiServer....
[08:49:34] - Removing old SSL certs...
[08:49:34] - Starting servers....
[08:49:34] Ui.UiServer --------------------------------------
[08:49:34] Ui.UiServer Web interface: http://*:8080/
[08:49:34] Ui.UiServer --------------------------------------
[08:49:35] TorManager Connecting to Tor Controller 127.0.0.1:9051
[08:49:35] TorManager Tor controller connect error: ConnectionRefusedError: [Errno 111] Connection refused in TorManager.py line 143 > _socket3.py line 335
[08:49:35] TorManager Disabling Tor, because error while accessing Tor proxy at port 127.0.0.1:9050: No connection
[08:49:35] PeerPortchecker Checking port 15634 (ipv4) using checkPortchecker result: {'ip': '45.77.185.175', 'opened': True} in 0.034s
[08:49:35] ConnServer Server port opened ipv4: True, ipv6: False
[08:49:38] Db:ContentDb Commit error: cannot commit transaction - SQL statements in progress
[08:49:41] Site:1Q9LrW..X7z1 Rebuilding db...
[08:49:41] Site:1Q9LrW..X7z1 Deleting ./data/1Q9LrWv27gyToaqDXA2GF2ZSA6PhznX7z1/data/users/zhihu.db
[08:49:41] Site:1Q9LrW..X7z1 Creating tables...
[08:49:41] Db:ZhiHu Table keyvalue outdated...version: False need: 2, rebuilding...
[08:49:41] Db:ZhiHu Table json outdated...version: 0 need: 2, rebuilding...
[08:49:41] - Unhandled exception
Traceback (most recent call last):
  File "/root/pypy3.5-v7.0.0-linux64/site-packages/gevent/greenlet.py", line 766, in run
    result = self._run(*self.args, **self.kwargs)
  File "/root/ZeroNet-py3/src/Site/SiteStorage.py", line 141, in rebuildDb
    self.db.checkTables()
  File "/root/ZeroNet-py3/src/Db/Db.py", line 243, in checkTables
    ], version=self.schema["version"])
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 161, in needTable
    self.createTable(table, cols)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 138, in createTable
    self.execute("DROP TABLE IF EXISTS %s" % table)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 101, in execute
    res = self.cursor.execute(query)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 770, in wrapper
    return func(self, *args, **kwargs)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 949, in execute
    return self.__execute(False, sql, [params])
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 890, in __execute
    self.__connection.commit()
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 475, in commit
    raise self._get_exception(ret)
_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
Traceback (most recent call last):
  File "/root/pypy3.5-v7.0.0-linux64/site-packages/gevent/greenlet.py", line 766, in run
    result = self._run(*self.args, **self.kwargs)
  File "/root/ZeroNet-py3/src/Site/SiteStorage.py", line 141, in rebuildDb
    self.db.checkTables()
  File "/root/ZeroNet-py3/src/Db/Db.py", line 243, in checkTables
    ], version=self.schema["version"])
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 161, in needTable
    self.createTable(table, cols)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 138, in createTable
    self.execute("DROP TABLE IF EXISTS %s" % table)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 101, in execute
    res = self.cursor.execute(query)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 770, in wrapper
    return func(self, *args, **kwargs)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 949, in execute
    return self.__execute(False, sql, [params])
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 890, in __execute
    self.__connection.commit()
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 475, in commit
    raise self._get_exception(ret)
_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
[08:49:41] Db:ZhiHu Commit error: cannot commit transaction - SQL statements in progress
[08:49:41] Db:ZhiHu Table keyvalue outdated...version: False need: 2, rebuilding...
[08:49:41] Db:ZhiHu Table json outdated...version: 0 need: 2, rebuilding...
[08:49:41] WorkerManager:1Q9LrW..X7z1 107.172.237.134:38840: Error writing: dbschema.json (cannot commit transaction - SQL statements in progress)
[08:49:42] Ui.UiServer Added 45.77.185.175:8080 as allowed host
[08:49:43] Db:ContentDb Commit error: cannot commit transaction - SQL statements in progress
[08:49:43] Db:ZhiHu Commit error: cannot commit transaction - SQL statements in progress
[08:49:48] Db:ContentDb Commit error: cannot commit transaction - SQL statements in progress
[08:49:48] Db:ZhiHu Commit error: cannot commit transaction - SQL statements in progress
[08:49:50] Site:1TaLkF..jipT Rebuilding db...
[08:49:50] Site:1TaLkF..jipT Deleting ./data/1TaLkFrMwvbNsooF4ioKAY9EuxTBTjipT/data/users/zerotalk.db
[08:49:50] Site:1TaLkF..jipT Creating tables...
[08:49:50] Db:ZeroTalk Table keyvalue outdated...version: False need: 2, rebuilding...
[08:49:50] Db:ZeroTalk Table json outdated...version: 0 need: 2, rebuilding...
[08:49:50] - Unhandled exception
Traceback (most recent call last):
  File "/root/pypy3.5-v7.0.0-linux64/site-packages/gevent/greenlet.py", line 766, in run
    result = self._run(*self.args, **self.kwargs)
  File "/root/ZeroNet-py3/src/Site/SiteStorage.py", line 141, in rebuildDb
    self.db.checkTables()
  File "/root/ZeroNet-py3/src/Db/Db.py", line 243, in checkTables
    ], version=self.schema["version"])
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 161, in needTable
    self.createTable(table, cols)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 138, in createTable
    self.execute("DROP TABLE IF EXISTS %s" % table)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 101, in execute
    res = self.cursor.execute(query)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 770, in wrapper
    return func(self, *args, **kwargs)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 949, in execute
    return self.__execute(False, sql, [params])
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 890, in __execute
    self.__connection.commit()
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 475, in commit
    raise self._get_exception(ret)
_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
Traceback (most recent call last):
  File "/root/pypy3.5-v7.0.0-linux64/site-packages/gevent/greenlet.py", line 766, in run
    result = self._run(*self.args, **self.kwargs)
  File "/root/ZeroNet-py3/src/Site/SiteStorage.py", line 141, in rebuildDb
    self.db.checkTables()
  File "/root/ZeroNet-py3/src/Db/Db.py", line 243, in checkTables
    ], version=self.schema["version"])
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 161, in needTable
    self.createTable(table, cols)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 138, in createTable
    self.execute("DROP TABLE IF EXISTS %s" % table)
  File "/root/ZeroNet-py3/src/Db/DbCursor.py", line 101, in execute
    res = self.cursor.execute(query)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 770, in wrapper
    return func(self, *args, **kwargs)
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 949, in execute
    return self.__execute(False, sql, [params])
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 890, in __execute
    self.__connection.commit()
  File "/root/pypy3.5-v7.0.0-linux64/lib_pypy/_sqlite3.py", line 475, in commit
    raise self._get_exception(ret)
_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
[08:49:50] Db:ZeroTalk Commit error: cannot commit transaction - SQL statements in progress
[08:49:50] Db:ZeroTalk Table keyvalue outdated...version: False need: 2, rebuilding...
[08:49:50] Db:ZeroTalk Table json outdated...version: 0 need: 2, rebuilding...
[08:49:50] WorkerManager:1TaLkF..jipT 107.172.237.134:38840: Error writing: dbschema.json (cannot commit transaction - SQL statements in progress)
[08:49:53] Db:ContentDb Commit error: cannot commit transaction - SQL statements in progress
[08:49:53] Db:ZhiHu Commit error: cannot commit transaction - SQL statements in progress
[08:49:54] Db:ZeroTalk Commit error: cannot commit transaction - SQL statements in progress
^CShutting down...
[08:49:56] - Unhandled exception
NoneType: None
Traceback (most recent call last):
  File "/root/pypy3.5-v7.0.0-linux64/site-packages/gevent/libev/corecffi.py", line 55, in python_check_callback
    pass
KeyboardInterrupt
[08:49:56] - All server stopped
[08:49:56] Site:1HeLLo..Tf3D Save peer error: cannot commit transaction - SQL statements in progress
[08:49:56] Site:1Name2..hM9F Save peer error: cannot commit transaction - SQL statements in progress
[08:49:56] Site:1Q9LrW..X7z1 Save peer error: cannot commit transaction - SQL statements in progress
[08:49:56] Site:1TaLkF..jipT Save peer error: cannot commit transaction - SQL statements in progress
[08:49:56] Db:ContentDb Commit error: cannot commit transaction - SQL statements in progress
[08:49:56] Db:ZhiHu Commit error: cannot commit transaction - SQL statements in progress
[08:49:56] Db:ZeroTalk Commit error: cannot commit transaction - SQL statements in progress
root@vultr:~/ZeroNet-py3#
purplesyringa commented 4 years ago

Blocked by pypy#3182 if we can't remove interrupt() call (I think we can't).

purplesyringa commented 4 years ago

Another problem is PRAGMA journal_mode = WAL which we use for... what? (@HelloZeroNet). The following issue tracks the problem: pypy#3183. A hotfix is to disable PRAGMA journal_mode = WAL on PyPy.

HelloZeroNet commented 4 years ago

The WAL journal mode usually provide better performance/data security, than the default mode, but the default mode also should be fine.

interrupt() is also can be optional.

purplesyringa commented 4 years ago

Nice. That's gonna make porting easier.

purplesyringa commented 4 years ago

Blocked by https://github.com/ionelmc/python-tblib/issues/56 and https://github.com/ionelmc/python-tblib/issues/41.

purplesyringa commented 4 years ago

https://github.com/ionelmc/python-tblib/pull/57 will fix https://github.com/ionelmc/python-tblib/issues/56 but we need to get it merged to gevent. I don't want to make a PR to gevent yet because of https://github.com/ionelmc/python-tblib/issues/41.

purplesyringa commented 4 years ago

Getting https://github.com/ionelmc/python-tblib/pull/58 merged and then making a PR for https://github.com/imachug/pytest/commit/aa27407513801639422be8aeeb4cb77b55ef281f should work.

purplesyringa commented 4 years ago

Pytest PR is not required, yay!

purplesyringa commented 4 years ago

Waiting for https://github.com/gevent/gevent/pull/1541 to get merged.

purplesyringa commented 4 years ago

@HelloZeroNet Gevent guys don't seem active, do you think we can use a fork?

HelloZeroNet commented 4 years ago

I would avoid it if possible.

I think if the change only affects one file, then adding an import hook for pypy platform before loading gevent module would be better. https://stackoverflow.com/questions/39418845/python-3-import-hooks

Or maybe it can be simple as sys.modules["gevent._tblib"] = __import__("lib.gevent_patch._tblib") after loading (need some testing)

purplesyringa commented 4 years ago

Yay, https://github.com/gevent/gevent/pull/1541 got merged. I'm not sure if gevent's test failure is caused by it, so I'd wait a bit before it's resolved and then update the required gevent version in requirements.txt.

HelloZeroNet commented 4 years ago

Great, does the tests passing?

btw you can define special requirements for gevent using gevent==1.1.0; implementation_name == 'pypy', so it won't fail for users who installed from apt/package managers.

purplesyringa commented 4 years ago

I wasn't sure if there's something like implementation_name, I've only tried implementation. Thanks!

ZeroNet tests pass indeed.