docchaoswizard commented 3 years ago

Environment

Hydrus v417 Platform: Release on macOS OS specifics: Its complicated (Big Sur and Catalina)

What happens

If you try to start the macOS App on Big Sur it hangs indefinitely while "Booting DB". If you download the Source-Code and run the client.pyw with Python everything works like a charm. The built App works fine on Catalina.

What can I do to help Hydrus get Big Sur Compatibility? Are there Build Instructions to Build the Hydrus-App? I am more than happy to help the Development.

Greetings, Chaoswizard

Zweibach commented 3 years ago

From another user in the Discord support channel:

[12:25] asdfghjk: Hi all, Recently upgraded my mac to macos 11.0.1 and hydrus will no longer startup. Process just hangs, last line in the log is 'booting db…'. Moved my db and files on to a Win10 machine it still works fine so not a corrupt/damaged db issue. Starting the client with a fresh db and same issue. I have also tried going back a couple of version of Hydrus and still same issue so seems to be the update to macos 11 has caused hydrus to hang. Any other logs etc i can find to track down the issue?

hydrusnetwork commented 3 years ago

Thank you for this report. This seems to be a new thing with Big Sur, likely because the macOS laptop I use to build hydrus is now getting a bit old. I am exploring other build solutions (using github workflows rather than my old manual build scripts on local hardware) with some users to try to figure this out.

I am sorry to say I have no good solutions here for now--hydrus client does not seem to work on Big Sur.

Ah, I should say: if you can run from source, please do! There is no basic difference. This is just an issue with my build environment being old (Sierra, 10.12, and no longer updatable).

docchaoswizard commented 3 years ago

Thank you for getting in touch.

Thank you for this report. This seems to be a new thing with Big Sur, likely because the macOS laptop I use to build hydrus is now getting a bit old. I am exploring other build solutions (using github workflows rather than my old manual build scripts on local hardware) with some users to try to figure this out.

If you have a set of build instructions i can more than happily run them (Either on Catalina or Big Sur) and we can share the binaries. Hydrus is an awesome tool and i would be proud to help it be more compatible on different OSses.

skj-dev commented 3 years ago

I just cloned this repo, ran pip install -r requirements.txt and then symlinked client.pyw to /usr/local/bin/hydrus. When I ran hydrus -d="/path/to/my/hydrus/db" it came right up. 🎉

I'm also using Python from Homebrew, FWIW.

Suika commented 3 years ago

@docchaoswizard Can you check if this runs on big-sur https://github.com/Suika/hydrus-build-test/actions/runs/376559330 ? Basically pyinstaller from github workflow.

Will slowly work towards automated builds and push to hydev.

docchaoswizard commented 3 years ago

@docchaoswizard Can you check if this runs on big-sur https://github.com/Suika/hydrus-build-test/actions/runs/376559330 ? Basically pyinstaller from github workflow.

Will slowly work towards automated builds and push to hydev.

Of course. Could you please tell me how exactly? Copying client.spec to a source-code-release of Hydrus and running pyinstaller client.spec doesn't work, even after adjusting the paths (Lots of complaints of missing libraries).

Regards, Chaoswizard

Suika commented 3 years ago

@docchaoswizard Can you check if this runs on big-sur https://github.com/Suika/hydrus-build-test/actions/runs/376559330 ? Basically pyinstaller from github workflow. Will slowly work towards automated builds and push to hydev.

Of course. Could you please tell me how exactly? Copying client.spec to a source-code-release of Hydrus and running pyinstaller client.spec doesn't work, even after adjusting the paths (Lots of complaints of missing libraries).

Regards, Chaoswizard

In the linked action you should be able to download the "specdata-sur". It's pyinstalled hydrus without the whole app and other stuff. Extract it, if "client" is not executable, make it executable and start "client".

If it works, I will work on packaging it as a MacOS App. If you can't download from my action, here is the other link: https://files.catbox.moe/uvg9vm.zip

grafik

docchaoswizard commented 3 years ago

In the linked action you should be able to download the "specdata-sur". It's pyinstalled hydrus without the whole app and other stuff. Extract it, if "client" is not executable, make it executable and start "client".

If it works, I will work on packaging it as a MacOS App. If you can't download from my action, here is the other link: https://files.catbox.moe/uvg9vm.zip

Ah, understood. Download from your action worked. Sadly it doesn't start:

2020/11/27 17:22:23: hydrus client started
2020/11/27 17:22:35: booting controller…
2020/11/27 17:22:37: booting db…
2020/11/27 17:22:37: preparing db caches
2020/11/27 17:22:38: initialising managers

It hangs there, seemingly indefinitely using 100% of one Core of CPU.

Suika commented 3 years ago

@docchaoswizard jump in discord and poke me in support.

Suika commented 3 years ago

In the linked action you should be able to download the "specdata-sur". It's pyinstalled hydrus without the whole app and other stuff. Extract it, if "client" is not executable, make it executable and start "client". If it works, I will work on packaging it as a MacOS App. If you can't download from my action, here is the other link: https://files.catbox.moe/uvg9vm.zip

Ah, understood. Download from your action worked. Sadly it doesn't start:
2020/11/27 17:22:23: hydrus client started
2020/11/27 17:22:35: booting controller…
2020/11/27 17:22:37: booting db…
2020/11/27 17:22:37: preparing db caches
2020/11/27 17:22:38: initialising managers
It hangs there, seemingly indefinitely using 100% of one Core of CPU.

Oh, I remember why it probably won't work for you. Please try https://github.com/Suika/hydrus-build-test/actions/runs/387628245 The version I gave you before was 411, not 419. This time it is 419 @docchaoswizard

docchaoswizard commented 3 years ago

@Suika

Oh, I remember why it probably won't work for you. Please try https://github.com/Suika/hydrus-build-test/actions/runs/387628245 The version I gave you before was 411, not 419. This time it is 419 @docchaoswizard

Still the same issues. Hangs while initialising managers and uses 100% of one CPU-Core. Poking in Discord will commence tomorrow

Cheers, ChaosWizard

regentzu commented 3 years ago

I built a usable version of hydrus using pyoxidizer and this pyoxidizer.bzl config file

def make_dist():
    return default_python_distribution()

def make_exe(dist):
    policy = dist.make_python_packaging_policy()
    policy.resources_location = "filesystem-relative:library"
    python_config = dist.make_python_interpreter_config()
    python_config.filesystem_importer = True
    python_config.sys_frozen = True
    python_config.run_filename = "client.py"
    exe = dist.to_python_executable(
        name="hydrus_client",
        packaging_policy=policy,
        config=python_config,
    )
    exe.add_python_resources(exe.pip_install(["-r", "requirements.txt"]))
    exe.add_python_resources(exe.read_package_root(
        path=".",
        packages=["hydrus"],
    ))
    return exe

def make_embedded_resources(exe):
    return exe.to_embedded_resources()

def make_install(exe):
    files = FileManifest()
    files.add_python_resource(".", exe)
    return files

# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_dist)
register_target("exe", make_exe, depends=["dist"])
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()

# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.

PYOXIDIZER_VERSION = "0.10.3"
PYOXIDIZER_COMMIT = "UNKNOWN"

the resulting distributable needs to have the client.py copied alongside it along with the static and help directories. But it's running on BigSur 11.0.1 and successfully loaded my DB, downloaded public tag repository updates, and applied them.

ReAnzu commented 3 years ago

Using the above as a template, here's a actions workflow to produce a tarball https://github.com/ReAnzu/hydrus/runs/1508759086

The produced tarball works for me on Big Sur. It looks like some tweaks are required for a windows and linux build to complete.

Suika commented 3 years ago

PyInstaller for Windows and Linux/Ubuntu does work properly, but there seems to be a problem with MacOS + PyInstaller from the few tests that we conducted. I'll take a look at the workflow and see if that can be used for MacOS in case that there is no helping PyInstaller.

Suika commented 3 years ago

@ReAnzu Can you add https://github.com/actions/upload-artifact to your workflow, so the package can be downloaded and tested.

ReAnzu commented 3 years ago

@Suika Yup, I've also cleaned up the startup for the packaged app and created a ".app" mac app structure as well so it acts as more of a drop in replacement for the Hydrus Network.app from the pyinstaller builds. The only thing missing from my pyoxidizer setup is that I'm not generating an executable for server yet, just for the client path. Adding that step should be trivial but I don't host a server on Mac, so I've never used it and excluded it from my minimal viable patch.

Here is the latest run https://github.com/ReAnzu/hydrus/actions/runs/417946324

I've tested this on Big Sur 11.0.1 but have not tested it on Catalina or Mojave. PyOxidizer claims to use 10_9 as the base target for the pip requests so it should be compatible.

Suika commented 3 years ago

@ReAnzu, @docchaoswizard tested your release a short while ago and it worked splendedly. Looking at your changes, it seems that you have removed any references to BUILD TARGETs, does that mean it will now compile the same way on every platfrom without the need for extra configuration?

If so, using pyoxidizer for all builds would make more sense.

ReAnzu commented 3 years ago

@Suika I removed it as I was only testing MacOS builds. I can go back and test Linux and Windows as well but I think there are still some customization required in the pyoxidizer config for those versions because of the underlying libraries needed and Windows in particular needing a specific pyqt library from PyPi

Suika commented 3 years ago

Hm, ok.

ReAnzu commented 3 years ago

Linux builds just fine, I'll test Windows later and then try to run both. But it looks like customization might not be needed at all.

Suika commented 3 years ago

@ReAnzu check https://github.com/Suika/hydrus-build-test/blob/master/.github/workflows/build.yml for linux and windows builds.

lehitoskin commented 3 years ago

Does this need any additional testers? I have Big Sur that I can use to test, if necessary.

Suika commented 3 years ago

Seems to be working properly. @ReAnzu the build is now part of the latest version.

Suika commented 3 years ago

Guesss not, gotta keep it open for a few more weeks.

roachcord3 commented 3 years ago

Rather than opening up whole new issues for these relatively minor things I figured I'd just note them here. I tried the new big-sur-compatible client and ran into these issues so far:

it's not detecting the platform it's running on properly or something, so it creates the db inside the .app folder. (dev knows about this already)
right-click menus in dialogs can't expand to lower levels (e.g. can't select anything from the copy menu)
subtle UI style changes, not a big deal but it messes with the spacing on the tab bars, adding an extra gap between the tabs and the top menu bar:

roachcord3 commented 3 years ago

Minor UX bug: the new dmg for 436 looks like this, at least on Catalina. Notice the Applications folder shortcut does not look like a folder shortcut, just a nondescript file:

Dragging the .app onto it does not work, because Finder doesn't recognize it as a shortcut I guess. Not a big deal though.

hydrusnetwork commented 3 years ago

Notice the Applications folder shortcut does not look like a folder shortcut, just a nondescript file:

Thank you for this report. I just copied that shortcut file from my old situation to the new build. It must need a special permission bit on the file or something to turn it into a shortcut, I will look it up!

roachcord3 commented 3 years ago

@hydrusnetwork looks like your change to the Applications shortcut fixed it (at least on Catalina.)

Given that the latest release didn't have fixes for the right click menus not working in Catalina I may try upgrading to Big Sur to bypass the problem altogether... hopefully someone has tested right click menus on Big Sur since 435

docchaoswizard commented 3 years ago

Given that the latest release didn't have fixes for the right click menus not working in Catalina I may try upgrading to Big Sur to bypass the problem altogether... hopefully someone has tested right click menus on Big Sur since 435

Right click menus are still somewhat iffy on Big Sur. Some work perfectly (Like right click on a file), some can only be controlled with a keyboard (Like the logs from a downloader).

roachcord3 commented 3 years ago

For those not aware, I filed the right click menu bug as its own issue 2 weeks ago, since it seems like a large enough problem to warrant being tracked separately: #858

hydrusnetwork commented 2 years ago

I believe this is essentially fixed now, so closing. If you have a specific issue that came up in the thread, please raise it as a new issue.

hydrusnetwork / hydrus

Big Sur Compatibility #729

Environment

What happens