Windows .exe/.msi releases #733

Open HelloZeroNet opened 7 years ago

It would be nice to have a more regular, .exe releases for windows that also includes the source code, so no need to download it from github on first startup.

What liblary should we use? PyInstaller, py2exe, etc.
Single executable (unpacked to temp directory on startup) or normal, 6-7 file distribution?
Signing?
Automatization?
Source code packed into .exe or separately in .zip/directory?
What about updates?
Plugins?
Is it possible to submit it to Windows Store?
Alternative .msi or .exe installer with autostart option?
Where to store data/config file/third-party plugins?

I was able to create a .exe with PyInstaller. It's less than 10MB (zipped) with everything packed in: Python, ZeroNet source code and dependecies, Coffeescript, Openssl, msvc*dlls

Currently it's looks like this:

So pretty messy....It would be nice to move the supporter files to separate directory eg. Lib.

Need help

Unfortunately PyInstaller did not support this (https://github.com/pyinstaller/pyinstaller/issues/1048), so it would be nice if anyone could add this feature or modify the bootloader: https://github.com/pyinstaller/pyinstaller/tree/develop/bootloader

Other todo:

[X] Higher resolution icon
[ ] Data/Log dir outside of Include
[ ] Test update from https/zeronet

~~I don't like single executable.~~ I don't think single executable is good. I suggest release an installer and a zip with source code bundled. Use NSIS to create installers.

Signing: Windows's digital signature requires a certificate, which cost a lot. So I suggest using GunPG and sha1/sha256 hash.

I think it's possible to run Python on WinRT. (http://stackoverflow.com/questions/14211949/python-in-windows-store-apps)

I'm not talking about single file exe, but to move the .dll and .pyd files to a sub-directory to make it look more clean and more easy to find the executable. The keep it updateable from zeronet network the full source code is still there, it's in the Include directory. (I'm not sure about this name, maybe we should name it something different, ideas?)

I found a company that give cert for opensrouce applications for 14EUR: https://en.sklep.certum.pl/data-safety/code-signing-certificates/open-source-code-signing.html

Already signed up, but it requires Java and the interface is half polish, so I was unable to complete the registration, will try again later.

Your idea (move the .dll and .pyd files to a sub-directory) is same as mine. As for certificate, you can sign the installer exe. But you can't sign a zip file.

At least we can sign the zeronet.exe in the zip to avoid the notification about unsigned application.

I found that it's possible to move the directly imported pyd files to different directory, so it's looks better now:

Any ideas on the directory name where the .py files (zeronet source code) will be stored? Some ideas:

Source (we already have an src directory, so in this case it will be Source/src/ which is a little bit odd)
Include
Application
Program FIles (just kidding)
Core
System
Support
Runtime

What tools are you using to build these files?

At least we can sign the zeronet.exe in the zip to avoid the notification about unsigned application.

You should sign all .dll files includes .pyd files.

It's generated by pyinstaller

You should sign all .dll files includes .pyd files.

I don't think os or python checks the signatures.

New problem: I want to create a zip that have everything to go (python, openssl, sourcecode), but this means the .zip has to be updated pretty often. Which will make the ZeroBundle repository huge. Sure, I could distribute the files from zeronet.io, but it would make it easy to block. Is there any way to to replace binary files in git repo without keeping the history? (or any trick to keep the .git directory smaller)

You can(should) upload .zip files to GitHub's releases. IIRC, GitHub provides API to upload releases. You can use curl to upload it automatically.

Yeah, but it makes the mirroring harder, since it does not get propogated automatically to:

New idea: Instead of uploading the zip file, create separate repository for ZeroNet-win / ZeroNet-mac, add the exe/app/ddl/pyd and the source code as normal files, then let the provider to pack it, eg.: https://github.com/HelloZeroNet/ZeroNet/archive/master.zip

Is there any way to mirror the commits? So any commit that is submitted to ZeroNet repository have to be also added to ZeroNet-win/core directory

signed, .exe version: https://github.com/HelloZeroNet/ZeroNet-win/archive/dist.zip

You should include the ".manifest" file into ZeroNet.exe's resources.

what do you mean by that?

There's a ZeroNet.exe.manifest, which can be included in a exe file.

I think we had windows xp compatibility issues with that

What?

I think when I first published zerobundle it refused to work under windows xp without a .manifest file, but if someone could try it that would be nice.

Wait me have a test in VM.

I juist tried to embed the manifest, but then just drops this error (win10): Cannot open self f:\Work\ZeroNet-git\ZeroBundle\PyInstaller\dist\ZeroNet\ZeroNet.exe or archive f:\Work\ZeroNet-git\ZeroBundle\PyInstaller\dist\ZeroNet\ZeroNet.pkg

The command was: ..\..\tools\mt.exe -nologo -manifest "dist\ZeroNet\ZeroNet.exe.manifest" -outputresource:"dist\ZeroNet\ZeroNet.exe;1"

There's no issue after I deleted ZeroNet.exe.manifest. But there's a issue if I have not installed Visual C++ 2008 Redistributable it can't load some library. Copy Microsoft.VC90.CRT.manifest msvcm90.dll msvcp90.dll msvcr90.dll to lib can solve this.

I used PE Explorer to add manifest into exe. It works fine.

So the dlls should be in both directory? (lib and beside zeronet.exe)

Yes. It's good to put all dlls in lib dir.

It's a little annoying that Windows includes a msvcrt.dll. But only VC++ 6.0 programs could use this library.

Fuck, then have to find a way to move everything to the lib dir

maybe stick with .cmd and just convert it somehow to an exe

~~Exe is just a loader that loads python27.dll. It's possible not to make it require CRT dll.~~ I saw ZeroNet.exe dependency and found it requires kernel32.dll msvcrt.dll ws2_32.dll, which available in system.

Other option could be modifying the bootload to load the dlls from lib directory: https://github.com/pyinstaller/pyinstaller/blob/c14333fc7498d261e1ce47d2354ecaecc88037d2/bootloader/src/pyi_utils.c#L736

Unfortunetly i have no experience with C, so maybe it's better to pack as an exe that unpack to temp dir every time on startup. (im not fan of this solution, but probably the best we have) or simply drop the portable version and distribute it as an a exe/msi installer.

~~Or you can try py2exe...~~

py2exe has not been updated since 9 years, so i would avoid it. (and I think it's also does not support moving dll-s to separate directory)

As a workaround it will copy the dll-s to the lib dir on startup. (this way it does not makes the .zip larger)

Also included .manifest using ResourceHacker.exe and added dual (SHA1 & SHA256) signature. https://github.com/HelloZeroNet/ZeroBundle/commit/f544a9c2e708390134d4c9b0e42db195a89a903c Please test if you can: https://github.com/HelloZeroNet/ZeroNet-win/archive/dist.zip

Since this exe version is released, will you continue support the old ZeroBundle?

Exe is not released yet, it just testing, but after that the zerobundle will not be supported.

nope nope and dope.... try this: half-assed. FREEZE the python app. The only parts for OSX that are dodgy are Tkinter/UI elements. The rest, with path seperator and old school resources, should otherwise work, assuming you do OS checks for specific functions for machOS. Otherwise, if its all python, FREEZE it into pyc/pyd and a self-extracting exe. under wine, this can also be ran, but we dont need to go there on unices. Mac, well...can u just zip it? If you GIT the folder the project is in, then revisions,etc are a cinch(you dont know GIT). freezing on unices is tricky but ive done it some time ago.

MY C is SHIT. Dont ask, im a pascalian.

Another "easy" problem: Zeroner runs 3 times slower and using 10 times more memory in .exe from:

Benchmarking ZeroNet 0.5.1 (rev1830) Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on: win32...

CryptBitcoin:
- hdPrivatekey x 10..........0.192s [x3.65: Insane!!]
- sign x 10..........0.104s [x3.37: WOW]
- openssl verify x 100..........0.464s [x0.80: Goodish]
- pure-python verify x 10..........0.532s [x3.01: WOW]

CryptHash:
- sha256 5M x 10..........0.197s [x3.05: WOW]
- sha512 5M x 10..........0.244s [x2.46: Fast]
- os.urandom(256) x 100 000..........0.085s [x7.65: Insane!!]

Msgpack:
- pack 5K x 10 000..........1.427s [x0.55: Goodish]
- unpack 5K x 10 000..........0.705s [x1.70: Fast]
- streaming unpack 5K x 10 000..........0.636s [x2.20: Fast]

Db:
- Open x 10..........0.054s [x2.41: Fast]
- Insert x 10 x 1000..........2.237s [x0.45: Ehh]
- Buffered insert x 100 x 100..........2.483s [x0.52: Goodish]
- Total rows in db: 20000
- Indexed query x 1000..........0.690s [x0.36: Ehh]
- Not indexed query x 100..........0.251s [x2.39: Fast]
- Like query x 100..........0.422s [x4.27: Insane!!]

Done. Total: 10.92s

vs run using python.exe zeronet.py:

Benchmarking ZeroNet 0.5.1 (rev1830) Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on: win32...

CryptBitcoin:
- hdPrivatekey x 10..........0.185s [x3.78: Insane!!]
- sign x 10..........0.094s [x3.72: Insane!!]
- openssl verify x 100..........0.117s [x3.16: WOW]
- pure-python verify x 10..........0.507s [x3.16: WOW]

CryptHash:
- sha256 5M x 10..........0.150s [x4.00: Insane!!]
- sha512 5M x 10..........0.206s [x2.91: WOW]
- os.urandom(256) x 100 000..........0.071s [x9.15: Insane!!]

Msgpack:
- pack 5K x 10 000..........0.214s [x3.64: Insane!!]
- unpack 5K x 10 000..........0.319s [x3.76: Insane!!]
- streaming unpack 5K x 10 000..........0.342s [x4.09: Insane!!]

Db:
- Open x 10..........0.012s [x10.83: Insane!!]
- Insert x 10 x 1000..........0.346s [x2.89: WOW]
- Buffered insert x 100 x 100..........0.497s [x2.62: WOW]
- Total rows in db: 20000
- Indexed query x 1000..........0.084s [x2.98: WOW]
- Not indexed query x 100..........0.158s [x3.80: Insane!!]
- Like query x 100..........0.289s [x6.23: Insane!!]

Done. Total: 3.78s

Hmm...well...hmm any idea?

The frozen script:

import sqlite3, time

db = sqlite3.connect("test.db")
s = time.time()
db.execute("CREATE TEMP TABLE t(x INTEGER PRIMARY KEY ASC, y);")
for i in range(100000):
    db.execute("INSERT INTO t VALUES (%s, 'test')" % i)
print time.time() - s
raw_input(">")

Off topic: Could you complain what's the meaning of The frozen script?

It's not possible to sign .cmd, firewall/os restrictions can be problematic with python.exe, it also means less files

but this problem seems like not related to frozen mode. if I rename python.exe to zeronet.exe and run the included script with it, then it's also 18 times slower and using 10 times more memory. restarting windows does not fixes it.

can anyone check the same?

deleting "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\ZeroNet.exe" looks like fixed it...

I have not saved the value of it before the delete, probably somehow windows put it into special debug mode or similar....

Using Cython with type-annotation as a replacement of Python will speed-up zeronet about 3x-times maybe.

The CPU-hungry parts already written in C, so Cython would not help there

HelloZeroNet / ZeroNet

Windows .exe/.msi releases #733

Need help

Other todo: