Special folder tree

[JPFrancoia]

Hi,

These last days, I've been trying to deploy one of my programs. I wanted it to be runnable on OSX, Linux and Windows. I wanted it to be able to update itself. And I also wanted it to be easily installable by the final user.

Of course, I used Esky to freeze it and to bundle it. But not only: I used Inno setup on Windows, and Packages on OSX, to create an installer for the user.

While it was kind of easy to create an installer with Inno setup, it was not really straightforward. On OSX, it was a real pain in the....

The main problem I encountered was the special folder tree Esky uses:

my_program/
    exe_not_exe
    one_lib
    libpython3.4
    my_program/
        exe_really_exe
        esky_files/

If I understand correctly, exe_not_exe is here to make sure the program doesn't crash because of a partial update. But this architecture app-in-app is hard to deal with when packaging. And also, leads to unsolvable issues (like a double icon on the dock in OSX, due to the numerous tweaks I had to apply).

I wonder, and it is just a suggestion, if it would be possible to put these files needed by esky somewhere else, something like that:

my_program/
    one_lib
    libpython3.4
    exe_really_exe
    esky_files/
        exe_not_exe

I think it would simplify a lot the process of building a package, or an app directly runnable on OSX.

[timeyyy]

What is the reason switching formats would make this easier? Have you tried getting help from the packages people?

[JPFrancoia]

Yes I did, but my issue cannot be solved, ever. I'm not here to try to solve it, I'm just suggesting a simplification of the directory architecture.

Let's imagine you want to create a simple MyProgram.app that will run if you double-click it on OSX.

The normal structure of an app is like this:

MyProgram.app/
    Info.plist
    Contents/
        MacOS/
            exe_really_exe

If you double-click MyProgram.app, exe_really_exe is called, and the app starts. You can give this app to whoever you want, it will work by simply running it.

Now if the app is bundled with Esky:

MyProgram.app/
    Info.plist
    Contents/
        MacOS/
            exe_not_really_exe --> dummy file, mandatory
    MyProgram-mac-os-x86_64/
        MyProgram.app/
            Contents/
                MacOS/
                    exe_really_exe --> This file runs

If you double-click MyProgramm.app, it will run exe_not_really_exe, which is not the real main file of your app. If your app needs resources, it won't find them.

You have to do something like this:

MyProgram.app/
    Info.plist
    Contents/
        MacOS/
            launcher -> will launch the real main file
            exe_not_really_exe --> dummy file, mandatory
    MyProgram-mac-os-x86_64/
        MyProgram.app/
            Contents/
                MacOS/
                    exe_really_exe --> This file runs

You basically need to create a 'launcher' file that will start exe_really_exe. Then tell the app (by modifying Info.plist) to start launcher, which itself will start exe_really_exe. You can not, of course, tell directly Info.plist to start exe_really_exe (I tried, it didn't work).

My point is: it is doable, but complicated (but maybe I did something wrong and missed something). On Windows, when you create the installer with Inno Setup, you also need to modify the path of the real exe file. It is not complicated, but necessitate a manual intervention.

The two-levels architecture of the directories complicates (a lot) the building process. I'm just suggesting it could be easier.

[timeyyy]

mm as far as i understand it esky SHOULD be run from the top level bootstrap exe, what you are calling 'exe_not_really_exe', I'm using inno setup and it works fine.. The bootstrap file will open the corresponding exe in the subfolder. I was even under the impression that updating wouldn't work if you don't open your program using the bootstrap file. @rfk is what i said correct?

[JPFrancoia]

Maybe I'm doing something wrong then. I'm waiting @rfk to confirm, just to be sure. I haven't find anything in the wiki or elsewhere indicating which exe to run, actually.

What led me to run the second level exe:

• On linux, if somewhere in my code I do something like that: for company in os.listdir("./journals"): I get (after bundling and running the first level exe):

File "/tmp/tmp4wbgvh3n/scripts/gui.py", line 356, in getJournalsToCare
FileNotFoundError: [Errno 2] Aucun fichier ou dossier de ce type: './journals'

Which basically means the main exe does not find the resources files. Because the resources files are not in the top level folder. Running the second level exe works.

• On windows, with the exact same code, I don't have this issue. However, if I run the first level exe, I encounter another problem: I'm using PyQt, and my program can't access its sqlite database. I think it's because the sqldrivers cannot be found. They are in the second level folder. But here again, running the seconde level exe works.

I also have to mention that running the second level exe doesn't break the update process, it's totally smooth.

I would be glad if you could correct me if I did something wrong.

For what I see, I would say the external resources cause the problem. @timeyyy, do you use them in your program ?

[rfk]

The idea of this file layout is that the user should always run the top-level exe, the one you call "exe_not_really_exe". It's a safey feature, its job is to look at all the files on disk and figure out which version of your app to bootstrap into. For example, suppose you have some crash or other problem when you're half-way through an update, and you wind up with a folder structure like this:

MyProgram.app/
    Contents/
        MacOS/
           bootstrap_exe
    MyProgram-mac-os-x86_64.version1/
        MyProgram.app/
            Contents/
                MacOS/
                    real_exe
    MyProgram-mac-os-x86_64.version2/
        MyProgram.app/
            Contents/
                MacOS/
                    real_exe

Is the new version properly installed? Is the old version partially deleted? Which of the two versions should the user run? The job of the bootstrap_exe is to figure that out and make the right choice.

On linux, if somewhere in my code I do something like that: for company in os.listdir("./journals"):

FWIW, I recommend against using relative paths to locate resource files, you never really know how the user is going to invoke your app and there are a variety of ways it could be made to run in a different directory than you expect. I suggest locating everything explicitly relative to the executable, like:

  resource_dir = os.path.dirname(os.path.dirname(sys.executable))
  for company in os.listdir(os.path.join(resource_dir, 'journals')):
    ...do stuff...

All that said, part of the job of the "bootstrap exe" is to make it look like you're running the "real exe" as much as possible. So when you say things like:

a double icon on the dock in OSX

and

I'm using PyQt, and my program can't access its sqlite database. I think it's because the sqldrivers cannot be found. They are in the second level folder.

These are almost certainly bugs in esky, that sort of thing should just work when you run the bootstrap exe. If you can file testcases that reproduce these issues, that would be awesome.

[JPFrancoia]

Ok, thank you very much for telling me that, and for your precious advices. I was indeed doing something wrong.

I changed the way to locate files. Something like this works great:

if getattr(sys, "frozen", False):
    # http://stackoverflow.com/questions/10293808/how-to-get-the-path-of-the-executing-frozen-script
    self.resource_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
else:
    self.DATA_PATH = "."
    self.resource_dir = self.DATA_PATH

Whenever I need to access resources files, I use this snippet.

I'm closing this issue, it was irrelevant. Don't change anything, Esky is great !

I tried to add a few words to the wiki, to really insist on the fact that the first level executable should be the one to run. Is that ok for you guys ?

I will also open other issues, about the other points mentioned.

Many thanks again !

[timeyyy]

try to make the wiki docs more clear i guess...