search

saalfeldlab / paintera

GNU General Public License v2.0
94 stars 17 forks source link

Native Released to System Repositories/Stores #494

Open cmhulbert opened 11 months ago

cmhulbert commented 11 months ago

Native releases should be available in platform specific ways:L

Needs dependency management:

mkitti commented 11 months ago

I understand that @hubbardp has an active Apple developer key

We may qualify for a fee waiver for an Apple membership fee waiver since HHMI is a non-profit. I have not been able to clarify if applying for the waiver would violate a HHMi policy or not. https://developer.apple.com/support/membership-fee-waiver/

mkitti commented 11 months ago

Blosc1 availability summary

The latest release of c-blosc is 1.21.4. Importantly, this upgraded Zstd from 1.5.2 to 1.5.5 due to a known rare compression bug in Zstandard pre-1.5.5.

Ubuntu LTS and Debian stable contain out dated versions of libblosc and zstandard. The latest release of libblosc1 in Debian unstable is 1.21.3+ds-1.

Homebrew and MacPorts have c-blosc 1.21.4. The dependency graph is not clear to me at the moment. Homebrew and MacPorts also ship an up-to-date zstandard 1.5.5

I have not found a "native" Windows based distribution of a modular built blosc on either Chocholatey, win-get, or the Microsoft Store. conda-forge contains an update to date recipe of blosc 1.21.4.

conda-forge contains blosc 1.21.4 for all platforms. https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/meta.yaml https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/bld.bat https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/build.sh

Ubuntu libblosc1 version 1.21.1 in Ubuntu 22.04 LTS

https://packages.ubuntu.com/jammy/libblosc1 liblz4-1 (>= 0.0~r130) libsnappy1v5 (>= 1.1.7) [not riscv64] libzstd1 (>= 1.3.2) zlib1g (>= 1:1.1.4)

Debian libblosc1 version 1.21.3 in Debian bookworm stable

https://packages.debian.org/bookworm/libblosc1 liblz4-1 (>= 0.0~r130) libsnappy1v5 (>= 1.1.9) libzstd1 (>= 1.5.2) dep: zlib1g (>= 1:1.1.4)

Homebrew c-blosc 1.21.4

https://formulae.brew.sh/formula/c-blosc

MacPorts 1.21.4

https://ports.macports.org/port/blosc/

Chocholatey

https://community.chocolatey.org/packages?q=blosc (none exists)

win-get

https://winget.run/search?query=blosc (none exists)

conda-forge

https://anaconda.org/conda-forge/blosc osx-arm64 1.21.4 osx-arm64 blosc 1.21.4 linux-64 1.21.4 linux-aarch64 1.21.4 osx-64 1.21.4 (Intel) win-64 1.21.4

mkitti commented 11 months ago

For Windows, I recommend obtaining the DLLs in this fashion via a conda enabled PowerShell:

conda create -n blosc_env -c conda-forge blosc
conda activate blosc_env
cd $ENV:CONDA_PREFIX\Library\bin
dir

Copy and paste all the DLLs into the Paintera bin directory. This is the Windows way.

    Directory: C:\conda\3\envs\blosc_env\Library\bin

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----          5/6/2022     11:21          21984 api-ms-win-core-console-l1-1-0.dll
-a----          5/6/2022     11:29          21976 api-ms-win-core-console-l1-2-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-datetime-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-debug-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-errorhandling-l1-1-0.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-fibers-l1-1-0.dll
-a----          5/6/2022     11:21          26088 api-ms-win-core-file-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-file-l1-2-0.dll
-a----          5/6/2022     11:22          21976 api-ms-win-core-file-l2-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-handle-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-heap-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-interlocked-l1-1-0.dll
-a----          5/6/2022     11:22          22008 api-ms-win-core-libraryloader-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-localization-l1-2-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-memory-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-namedpipe-l1-1-0.dll
-a----          5/6/2022     11:21          22008 api-ms-win-core-processenvironment-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-processthreads-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-processthreads-l1-1-1.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-profile-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-rtlsupport-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-string-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-synch-l1-1-0.dll
-a----          5/6/2022     11:22          21976 api-ms-win-core-synch-l1-2-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-core-sysinfo-l1-1-0.dll
-a----          5/6/2022     11:21          21976 api-ms-win-core-timezone-l1-1-0.dll
-a----          5/6/2022     11:21          21992 api-ms-win-core-util-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-conio-l1-1-0.dll
-a----          5/6/2022     11:21          26080 api-ms-win-crt-convert-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-environment-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-filesystem-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-heap-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-locale-l1-1-0.dll
-a----          5/6/2022     11:21          30184 api-ms-win-crt-math-l1-1-0.dll
-a----          5/6/2022     11:21          30176 api-ms-win-crt-multibyte-l1-1-0.dll
-a----          5/6/2022     11:21          75232 api-ms-win-crt-private-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-process-l1-1-0.dll
-a----          5/6/2022     11:21          26104 api-ms-win-crt-runtime-l1-1-0.dll
-a----          5/6/2022     11:21          26080 api-ms-win-crt-stdio-l1-1-0.dll
-a----          5/6/2022     11:22          26072 api-ms-win-crt-string-l1-1-0.dll
-a----          5/6/2022     11:21          21984 api-ms-win-crt-time-l1-1-0.dll
-a----          5/6/2022     11:21          22000 api-ms-win-crt-utility-l1-1-0.dll
-a----         5/16/2023     07:09          69632 blosc.dll
-a----         5/10/2023     03:01         327576 concrt140.dll
-a----         1/26/2023     05:00         118784 liblz4.dll
-a----          7/7/2023     05:26         525312 libzstd.dll
-a----         1/26/2023     05:01         161792 lz4.exe
-a----         5/10/2023     03:01         578384 msvcp140.dll
-a----         5/10/2023     03:01          35704 msvcp140_1.dll
-a----         5/10/2023     03:01         267160 msvcp140_2.dll
-a----         5/10/2023     03:01          50072 msvcp140_atomic_wait.dll
-a----         5/10/2023     03:01          31640 msvcp140_codecvt_ids.dll
-a----         3/11/2023     06:39          75264 snappy.dll
-a----          5/6/2022     11:22        1123808 ucrtbase.dll
-a----         5/10/2023     03:01         414104 vcamp140.dll
-a----         5/10/2023     03:01         346008 vccorlib140.dll
-a----         5/10/2023     03:01         191864 vcomp140.dll
-a----         5/10/2023     03:01         109440 vcruntime140.dll
-a----         5/10/2023     03:01          49560 vcruntime140_1.dll
-a----         6/12/2023     09:09          89088 zlib.dll
-a----          7/7/2023     05:26         525312 zstd.dll
-a----          7/7/2023     05:26         129024 zstd.exe
mkitti commented 11 months ago

Wider Linux Distribution Scan for blosc packages: https://pkgs.org/search/?q=blosc https://pkgs.org/search/?q=libblosc1 https://pkgs.org/download/blosc https://pkgs.org/download/c-blosc

mkitti commented 11 months ago

I was also finally note that we really need to move to Blosc2 for better upstream support from the authors as well as the second generation frame format. Blosc1 is becoming increasingly difficult to support.

cmhulbert commented 11 months ago

Thanks @mkitti for the info. I think what we ideally will use is the platform specific standard auto-updating sites, to make using Paintera as straight-forward from the end user. For example, if we package Paintera on windows, in a way that requires we get the dlls from conda first, It would still require that the user has conda installed (unless we can have it as a dependency?). Perhaps we can do this at package time on the windows runner, and just provide the dlls to the installer?

As for blosc2, I understand it's backwards compatible; Is it easier to package and move to blosc2 say, than maintain blosc1?

mkitti commented 11 months ago

No, the user would not need conda. We would just use conda to obtain the binaries and include them in the Windows package. The expectation on Windows is for us to vendor such dependencies into the package.

cmhulbert commented 11 months ago

Thanks @mkitti! We had a productive chat to discuss this more, a brief summary as follows:

So next steps as far as I have is to generate the deb package with the blosc dependency, and release to our ppa. That should be fairly straight forward. After, can discuss more specifics around Apple/Windows stores. In the meantime, I can also work on publishing the SNAPSHOT releases on github like we currently are, but with the bundled bloscl library. Likely the best way to do this is as @mkitti mentioned above, by pulling them from the conda recipes in the github action runner, and package them.

Some other notes, we will want to setup Paintera to publish SNAPSHOT releases on an M1 mac runner as welll.

cmhulbert commented 8 months ago

Just an update for now, but the deb package built and released here ubuntu-latest should now depend on libblosc1