search

HelloZeroNet / ZeroNet

ZeroNet - Decentralized websites using Bitcoin crypto and BitTorrent network
https://zeronet.io
Other
18.38k stars 2.27k forks source link

Mac .app/.dmg releases #734

Open HelloZeroNet opened 7 years ago

HelloZeroNet commented 7 years ago

I'm not really familiar with mac ecosystem, whats the most regular way to install/distribute apps there?

iShift commented 7 years ago

You can chose DMG or App Store, dmg also good way

  1. You should sign app (app with sign much easy to install)
  2. ?
  3. all resource should be in .app folder
  4. you can update app like now (check on start, download new content to app directory)
  5. yes, but you may have a couple of issues with the license (or may not)
HelloZeroNet commented 7 years ago

If every resource is in the .app, then if I update the source code, then it will make the signature invalid, isn't it? Also where should I put the data dir/config/log/third-party plugins?

iShift commented 7 years ago

hmm, ok, you can act like google - use Application Support folder ${HOME}/Library/Application Support

kaero commented 7 years ago

hmm, ok, you can act like google - use Application Support folder ${HOME}/Library/Application Support

Why "like Google"? It's designed to be used for application specific need. Just use /Library/Application Support/ZeroNet for system-wide things and ~/Library/Application Support/ZeroNet for OS user-related.

About the signing: yes, whole .app bundle must be signed, signing only executable binaries will solve request to allow listening on public network interfaces, but doesn't allow users to launch the application using double-click without the right click -> Cmd+Open -> Open sequence.

I think the best way for zeronet will be distrubtion only launcher as an .app in the zip or dmg. Launcher can be signed and barely changes. All the application (python, libs, modules, sources, ...) can be stored as "data" in the /Library/Application Support/ZeroNet and kept up-to-date by the launcher. Btw, any distributed this way executables must be signed for the good.. and to suppress macos firewall messages of cause ;)

HelloZeroNet commented 7 years ago

I was able to create .app pretty easily with PyInstaller, but unfortunately looks like it's not possible to execute in terminal, is there any standard procedure for that?

HelloZeroNet commented 7 years ago

Also which one you prefer? Distributing a .zip-ed .app or .dmg packed .app? (with Application folder symlink and similar)

iShift commented 7 years ago

DMG, other apps on mac use DMG or App Store

HelloZeroNet commented 7 years ago

OK, here is the plan then:

What about permissions? I suppose you need to be admin to drag anything to Application dir or write to /Library/Application Support/

Maybe we should check on startup if we are in the /Application directory and if not, then switch to portable mode: Create "ZeroNet files" directory and save everything there.

kaero commented 7 years ago

DMG, other apps on mac use DMG or App Store

or zip. Many apps use zip. And, in the actions to install it's more effective, than dmg image. Lets compare!

Dmg:

  1. Double-click to mount
  2. Wait while macOS check image before mounting (popup with progressbar)
  3. Popup with image contents opened and you can drag'n'drop .app bundle to Applications folder.
  4. Now you must unmount image in the Finder sidebar
  5. Remove image (optionally).

Zip:

  1. Double-click to extract .app bundle
  2. Drag'n'drop to Applications folder on the Finder sidebar
  3. Remove archive (optionally).

I don't know any reason to distribute applications via images instead of zip archives if App Store is not a way.

iShift commented 7 years ago

user files (keys) or all data folder should be in ~/Library/Application Support/ZeroNet

What about permissions? I suppose you need to be admin to drag anything to Application dir or write to /Library/Application Support/

standard user on os x - admin (not root) And standard users don't see password alert (sudo alert) when move apps to /Applications

Maybe we should check on startup if we are in the /Application directory and if not, then switch to portable mode: Create "ZeroNet files" directory and save everything there.

some apps ask about move it self to Application but i don't think that it is good way, if someone want portable mode - it can download source codes.

@kaero your are wrong: Dmg:

Double-click to mount

  1. All macs with SSD mount image near-instant
  2. Popup with image contents opened and you can drag'n'drop .app bundle to Applications folder.
  3. Unmount image - drag from desktop to trash
  4. Remove image (optionally).

Main feature DMG - you can have custom installer, splash screen with link to app folder.

And @kaero give me example about apps in zip Chrome use dmg, vlc use dmg, GitHub also use dmg

kaero commented 7 years ago

@iShift

Reasons to use dmg:

linkerlin commented 7 years ago

What about Nuitka?

ysc3839 commented 7 years ago

Actually macOS's .app is a directory. So we can store files in it.

HelloZeroNet commented 7 years ago

I tried to sign it with the .pfs i got from https://en.sklep.certum.pl/data-safety/code-signing-certificates.html, but the result is:

$ codesign --verbose --force --deep --sign "Open Source Developer, Tamas Kocsis" ZeroNet.app
ZeroNet.app: signed app bundle with Mach-O universal (i386 x86_64) [com.apple.ScriptEditor.id.ZeroNet]

$ codesign -v ZeroNet.app/
ZeroNet.app/: CSSMERR_TP_NOT_TRUSTED
In architecture: x86_64

Am I doing something bad or it's not possible to sign mac .app with third-party cert?

Update: according to this, third-party certificates does not work on osx, so I need to pay 99USD/year to Apple :(

ysc3839 commented 7 years ago

Actually macOS's .app is a directory. So we can store files in it.

It seems that store files in app bundle will broke signature (http://stackoverflow.com/a/22694704). I suggest store in app's parent directory.

HelloZeroNet commented 7 years ago

The source code will be stored in .app, but it will store updates in ~/Library/Application Support/ZeroNet

ysc3839 commented 7 years ago

I don't think this is good. What if I have two ZeroNet.app?

HelloZeroNet commented 7 years ago

Then it will use see same data. On mac the normal install method is dragging the .app to /Applications directory. The application has no rights to write there, so it should write to ~/Library/Application Support/ZeroNet

Alternative it can be based on if the .app is in the /Application or not

HelloZeroNet commented 7 years ago

Horray! I was able to produce .app that accepted by the gatekeeper, but if I download it using safari it runs differently: on startup it got moved to /private/.../AppTranslocation/ read-only directory, so I'm not sure how can we make it portable more info on that: http://lapcatsoftware.com/articles/app-translocation.html

iShift commented 7 years ago

Good :) No easy-portatable version is ok, if someone want portatable it can download it like now

HelloZeroNet commented 7 years ago

Strangely if you move the .app to anywhere (via Finder), then it's no longer got moved to /private... directory on startup.

So it can work like that: If the startup path is in /private or /Applications, then save data to ~/Library/Application Support/ZeroNet otherwise to local directory (where .app is).

My new concern is release-signing: My cert and private key are stored in MacOS (VMWare), so every time I make any new modification I have to boot up the VM, re-create the .app, sign it (takes minutes with --deep), then upload the new zip.

Is there any way to make it easier?

Update: This could work: https://www.bitrise.io/

HelloZeroNet commented 7 years ago

I got it running (the .app is ~9MB zipped with full python and source code included), supports portable and installed deployment. The only significant problem left is after starting it the application icon got placed to the dock, but it does not do anything. I will try to find a way to handle at the click (opening browser window) or hiding it.

ysc3839 commented 7 years ago

Add LSUIElement = true in Info.plist to hide it. I don't think it should be hid until we added a menu bar. #339

HelloZeroNet commented 7 years ago

This works as standalone (will try to implement as a plugin tomorrow):

from Tkinter import Tk
root = Tk()
root.iconify()
def click():
    import webbrowser
    webbrowser.open("http://127.0.0.1:43110")
root.createcommand('tk::mac::ReopenApplication', click)
root.mainloop()

This way we got an easy way to access ZeroNet, Exit and Open at login possibility. (as drawback it adds +8MB mem usage)

ysc3839 commented 7 years ago

You should know macOS doesn't allow GUI on non-main thread. I think this will fail. You have to start a new process.

HelloZeroNet commented 7 years ago

thanks for the suggestion, i was able to make it work:

import sys
import os
import time
import zeronet

def gui():
    global root
    print "Gui started"
    time.sleep(5)
    sys.path.append("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk")
    sys.path.append("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/")
    from Tkinter import Tk
    root = Tk()
    root.iconify()
    def click():
        "Click"
        import webbrowser
        webbrowser.open("http://127.0.0.1:43110")

    def quit():
        print "Quit"
        sys.exit(0)

    root.createcommand('tk::mac::ReopenApplication', click)
    root.createcommand('tk::mac::ShowHelp', click)
    root.createcommand('tk::mac::ShowPreferences', click)
    root.createcommand('tk::mac::standardAboutPanel', click)
    root.createcommand('tk::mac::Quit', quit)
    try:
        root.mainloop()
    except Exception, err:
        print "Gui error: %s" % err
    print "Gui ended"

def main():
    sys.argv = [sys.argv[0]] + ["--open_browser", "default_browser"] + sys.argv[1:]
    zeronet.main()
    print "Ended"
    root.destroy()

if __name__ == '__main__':
    from threading import Thread
    t = Thread(target=main)
    t.daemon = True
    t.start()

    gui()

On shutdown it triggers keyboarderror, and I had to add check_same_thread=False to database connects, but it seems working ok.

ysc3839 commented 7 years ago

What's this?def click(): "Click"

And you said

i'm not a big fan of tkinter (looks bad, slow startup) and including it to zerobundle would significantly increase the size of it.

Why you use it now?

HelloZeroNet commented 7 years ago

The code is just an experiment, I will remove the prints and the unnecessary parts.

I'm still not fan of the tkinter, but that's the best option we have now. I also tried PyObjC, but even the simpliest application takes 40MB of ram. I not going to include it to the bundle, but use the preinstalled one (if avaliable) + delay the load (time.sleep(5), but will try to figurate out something smarter) to not slow down the startup time.

HelloZeroNet commented 7 years ago

I have added the files here, but unfortunetly for some unknown reasons the signature become invalid:

ZeroNet-mac $ find ZeroNet.app -type f -print0 | sort -z | xargs -0 shasum > sum.sha | shasum
da39a3ee5e6b4b0d3255bfef95601890afd80709  -
ZeroNet-mac $ codesign -v --verify ZeroNet.app
ZeroNet.app: valid on disk
ZeroNet.app: satisfies its Designated Requirement
ZeroNet-mac-dist $ find ZeroNet.app -type f -print0 | sort -z | xargs -0 shasum > sum.sha | shasum
da39a3ee5e6b4b0d3255bfef95601890afd80709  
ZeroNet-mac-dist $ codesign -v --verify ZeroNet.app
ZeroNet.app: code object is not signed at all
In subcomponent: /Users/sumo/Downloads/ZeroNet-mac-dist/ZeroNet.app/Contents/MacOS/core/CHANGELOG.md

Any idea why?

HelloZeroNet commented 7 years ago

Ah I took 7 hours of googling, trying and dumping, I almost gave up on it, but then got it:

The codesign adding separate signature to every file in the MacOS directory. Usually it puts these signatures into the binary files content, but since we had .py files, it couldn't. In this cases it's using extended attributes (xattr) to store the signatures. Git don't sync these attributes, so they lost after the commit and the verification failed after the download. The two directories seemed to be the exact same: Every file had the same shasum, same permissions, only the xattr was different. (That I never heard before)

The solution was simple: move the .py files from the MacOS directory to the Resources. (It also fixed the 6 minute signature time)

HelloZeroNet commented 7 years ago

I was able to reduce the added data to .git directory from 2.5MB to 50kb, by moving the Python standard lib from the executable to external file. So I think the hard part is done, what is left:

HelloZeroNet commented 7 years ago

I have been fighting on an older macOS (10.9) with ImportError: dlopen(/Users/user900818/Downloads/ZeroNet2.app/Contents/MacOS/pyexpat.so, 2): Symbol not found: _XML_SetHashSalt error since yesterday. I tried to pack the required dylibs, but without any luck (python still loaded the system default one)

Switching to conda looks like fixed it and the zipped .app also become smaller: 8.7MB -> 7.9MB

just for the record, commands I used to install conda:

bash Miniconda2-latest-MacOSX-x86_64.sh
conda install gevent
conda install msgpack-python
conda config --add channels conda-forge
conda install pyinstaller

Update Of course it's not this simple: The generated .app no longer accepted on 10.12 (Identity of the developer cannot be confirmed.) regardless

$ spctl -a -t exec -vvvv ZeroNet.app
ZeroNet.app: accepted
source=Developer ID
origin=Developer ID Application: Tamas Kocsis (4977YF9Q3Z)
$ codesign -v --verify ZeroNet.app/
ZeroNet.app/: valid on disk
ZeroNet.app/: satisfies its Designated Requirement

sigh

Update 2 Reverted back to normal (non-conda) version, with conda's libexpat.so. Looks like it working on both 10.12 and 10.9. Horray! (for now)

HelloZeroNet commented 7 years ago

After some testing it's looks like working well in 10.9 (using http://www.macincloud.com/), 10.11 and 10.12

The macincloud.com stored the applications in /Users/userXX/Library/Managed Items/My Applications so I also added ~/Library/* to non-portable mode (store data in ~/Library/Application Support instead of same directory as the .app)

I will re-create the ZeroNet-mac repo (to keep it small), change the link on zeronet.io, then start experimenting with bitrise.

frabrunelle commented 7 years ago

seems to work fine for me too on macOS 10.12. nice job!! 💯

iShift commented 7 years ago

where is link ? (i want to test it too..)

frabrunelle commented 7 years ago

@iShift https://github.com/HelloZeroNet/ZeroNet-mac/tree/dist

HelloZeroNet commented 7 years ago

Well yeah, it happened: It's working on 10.9, 10.11, 10.12, but not on 10.10.

$ ZeroNet.app/Contents/MacOS/ZeroNet
Error loading Python lib '/Users/sumo/Downloads/ZeroNet-mac-dist/ZeroNet.app/Contents/MacOS/.Python': dlopen(/Users/sumo/Downloads/ZeroNet-mac-dist/ZeroNet.app/Contents/MacOS/.Python, 10): no suitable image found.  Did find:
    /Users/sumo/Downloads/ZeroNet-mac-dist/ZeroNet.app/Contents/MacOS/.Python: code signature invalid for '/Users/sumo/Downloads/ZeroNet-mac-dist/ZeroNet.app/Contents/MacOS/.Python'

Regardless

$ codesign -vvvv ZeroNet.app/Contents/MacOS/.Python 
ZeroNet.app/Contents/MacOS/.Python: valid on disk
ZeroNet.app/Contents/MacOS/.Python: satisfies its Designated Requirement
$ codesign -vvvv ZeroNet.app/
ZeroNet.app/: valid on disk
ZeroNet.app/: satisfies its Designated Requirement

Tried to google the error, but no solution found, so currently experimenting with Py2app instead of PyInstaller...

radfish commented 5 years ago

Consider using Homebrew. You write one recipe (which will be very similar to recipes for Linux distributions), and users gets to install and run with:

brew install zeronet
zeronet

Why bother with this whole DMG thing?

HelloZeroNet commented 5 years ago

Because most of the users does not have brew/xcode installed

radfish commented 5 years ago

On Sun, Apr 07, 2019 at 03:02:04AM -0700, ZeroNet wrote:

Because most of the users does not have brew/xcode installed

Agreed, many don't, but installing brew is just one paste from the homepage away and is well supported. It's a similar situation with PIP. Many users do not have PIP installed, but because installation of PIP is trivial, the strategy ZeroNet already takes is not to avoid using PIP, but to tell the users to install PIP in order to install ZeroNet. Homebrew would be no different.

Btw, brew itself is independent of Xcode or other compilers. Brew can distribute built binaries in addition to source. But ZeroNet is even easier since it's not compiled anyway, but mentioning in case this is relevant to some of ZeroNet's dependencies with native components.

My only point is just that distribution via brew might be much easier. DMG installer is independent of recipe for brew; I would just skip the installer and only package for brew (to save effort), but obviously having both works too.

Btw, once the setuptools packaging is merged, installation via pip should work on all platforms too. That said, this is orthogonal to distribution packaging recipes (brew, PKGBIULDs, Debian, etc), which just benefit by getting to use setuptools in implementing the recipe. I could contribute by testing on a Mac once setuptools is merged to see what happens with PIP.

rllola commented 5 years ago

Is this still working ? The install doc doesn't show link to a .dmg release (https://github.com/HelloZeroNet/ZeroNet#how-to-join)

I think it is a good thing to have.