Closed jaimergp closed 2 years ago
Merging #3555 (43d5b8f) into main (2fe7f19) will increase coverage by
0.12%
. The diff coverage is50.00%
.
@@ Coverage Diff @@
## main napari/napari#3555 +/- ##
==========================================
+ Coverage 83.38% 83.51% +0.12%
==========================================
Files 589 591 +2
Lines 48879 49347 +468
==========================================
+ Hits 40757 41210 +453
- Misses 8122 8137 +15
Impacted Files | Coverage Δ | |
---|---|---|
napari/__main__.py | 54.50% <50.00%> (ø) |
|
napari/_qt/qt_resources/_tests/test_icons.py | 75.00% <0.00%> (-25.00%) |
:arrow_down: |
napari/layers/image/_image_key_bindings.py | 78.43% <0.00%> (-16.57%) |
:arrow_down: |
napari/layers/_source.py | 96.00% <0.00%> (-4.00%) |
:arrow_down: |
napari/utils/geometry.py | 99.29% <0.00%> (-0.71%) |
:arrow_down: |
napari/layers/tracks/_track_utils.py | 91.47% <0.00%> (-0.57%) |
:arrow_down: |
napari/layers/image/image.py | 94.57% <0.00%> (-0.18%) |
:arrow_down: |
napari/_qt/layer_controls/qt_image_controls.py | 97.59% <0.00%> (-0.14%) |
:arrow_down: |
napari/_vispy/layers/image.py | 96.96% <0.00%> (-0.07%) |
:arrow_down: |
napari/layers/utils/color_manager.py | 98.98% <0.00%> (-0.01%) |
:arrow_down: |
... and 28 more |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 2fe7f19...43d5b8f. Read the comment docs.
How do channels work with this conda bundle approach?
Could the bundle use @andfoy channel to get pyqt
and "work" on macOS arm64 too?
Technically yes, but ideally we would get that working on conda-forge first. That's not going to happen soon so I guess that at first we will add that channel to the list.
Technically yes, but ideally we would get that working on conda-forge first. That's not going to happen soon so I guess that at first we will add that channel to the list.
Maybe sooner than later? @andfoy has been working on the official qt too: https://github.com/conda-forge/qt-feedstock/pull/206
@jaimergp Was not able to install new napari version, I already had installed v04.13.de77 and got the "Installation Failed message" so i thought that was the reason so I uninstalled it and tried again but no luck.
Also, for testing Windows version, (using workspace) links are not clickable so i am not able to download the install package
@lauramarcos
On MacOS, the installer logs are available through the topbar menu. We'll need those (with all details) to know what's going on. Click Save
and upload the txt file.
On Windows, the links might not be clickable because you need to log in with your GitHub account first.
@jaimergp This is looking really good! Really nice work! 😄
I noticed on the mac bundle, when I have the dark theme enabled, the napari logo image is missing and the logs say "Failed to load specified background image". The image appears as expected when I have the light theme enabled.
@jaimergp I've rebased this branch on main and removed all the merge commits. Please check that your work is intact, and if so, please delete your local branch and fetch this one in its place. If not, could you please push it to a fork so that I can try again?
Incidentally, I think it's still WIP but I don't think we should have napari.packaging
as part of a public API? Sorry if I peeked behind the curtain a bit early! 😂
Thanks! I'll basically delete and clone again.
Yes, don't worry about napari.packaging
for now; I don't think it's being used and I need to do some cleanup here. I have even thought of making this a separate repo, but we'll start that discussion once I am happy with the code and the bundles. There are so many moving pieces here that it will require a good writeup too.
Yay, PKG installers are now signed!!
$ pkgutil --check-signature napari-0.4.13.dev203-macOS-x86_64.pkg
Package "napari-0.4.13.dev203-macOS-x86_64.pkg":
Status: signed by a developer certificate issued by Apple for distribution
Signed with a trusted timestamp on: 2021-12-24 11:34:16 +0000
Certificate Chain:
1. Developer ID Installer: Chan Zuckerberg Initiative, LLC (7WAT66VT96)
Expires: 2026-12-23 22:33:48 +0000
SHA256 Fingerprint:
D5 5F B8 66 83 AF 3B 71 F1 DB BB 91 3B 2A 9D 16 9F 97 60 33 B3 BA
82 12 40 B8 CE 28 93 0C 26 70
------------------------------------------------------------------------
2. Developer ID Certification Authority
Expires: 2027-02-01 22:12:15 +0000
SHA256 Fingerprint:
7A FC 9D 01 A6 2F 03 A2 DE 96 37 93 6D 4A FE 68 09 0D 2D E1 8D 03
F2 9C 88 CF B0 B1 BA 63 58 7F
------------------------------------------------------------------------
3. Apple Root CA
Expires: 2035-02-09 21:40:36 +0000
SHA256 Fingerprint:
B0 B1 73 0E CB C7 FF 45 05 14 2C 49 F1 29 5E 6E DA 6B CA ED 7E 2C
68 C5 BE 91 B5 A1 10 01 F0 24
Now we "only" need to notarize the PKG...
Fantastic 😍
More info (taken from here):
$ spctl -a -v --type install napari-0.4.13.dev203-macOS-x86_64.pkg
napari-0.4.13.dev203-macOS-x86_64.pkg: rejected
source=Unnotarized Developer ID
We already had trouble with notarization in the app bundle, as written by @potating-potato some weeks ago, but maybe this PKG approach is easier in that regard. We still need some extra credentials to make this happen.
There's a GH Action we can use for that, but I think we can also implement it ourselves in the CI:
The workflow is:
response=$(xcrun altool --output-format json --notarize-app -f "$PKG_PATH" --primary-bundle-id "org.napari.pkg" -u $username -p $password --verbose)
uuid=$(echo "$response" | jq -r '.notarization-upload.RequestUUID')
Poll for result
# loop this until successful
xcrun altool --output-format json --notarization-info $uuid -u $username -p $password
If successful, staple the result
xcrun stapler staple "$PKG_PATH"
I'd argue we only want to do notarization (assuming we pass it :D) on final releases because it can be time consuming, but for now we need to do it in this PR at least. Good thing is that you can now open the PKG (even if unnotarized) by right-clicking on it and choosing Open> Installer
, instead of going to Preferences > Security.
More info (taken from here):
$ spctl -a -v --type install napari-0.4.13.dev203-macOS-x86_64.pkg
napari-0.4.13.dev203-macOS-x86_64.pkg: rejected
source=Unnotarized Developer ID
We already had trouble with notarization in the app bundle, as written by @potating-potato some weeks ago, but maybe this PKG approach is easier in that regard. We still need some extra credentials to make this happen.
There's a GH Action we can use for that, but I think we can also implement it ourselves in the CI:
The workflow is:
response=$(xcrun altool --output-format json --notarize-app -f "$PKG_PATH" --primary-bundle-id "org.napari.pkg" -u $username -p $password --verbose)
uuid=$(echo "$response" | jq -r '.notarization-upload.RequestUUID')
Poll for result
# loop this until successful
xcrun altool --output-format json --notarization-info $uuid -u $username -p $password
If successful, staple the result
xcrun stapler staple "$PKG_PATH"
I'd argue we only want to do notarization (assuming we pass it :D) on final releases because it can be time consuming, but for now we need to do it in this PR at least. Good thing is that you can now open the PKG (even if unnotarized) by right-clicking on it and choosing Open> Installer
, instead of going to Preferences > Security.
Hi @jaimergp I see:
add osx-arm64 installer (WIP)
https://github.com/napari/napari/pull/3555/commits/3fea6f9250391f763125c1a6b207a6f45e782858
❤️ ❤️ ❤️
Please let me know how I can help with testing on my M1 MBP!
Hi @psobolewskiPhD, yes I am working on it! The infra is there but there are some solver issues right now due to the non-native platform stuff we are doing. I'll ping you once the new installers are ready!
Well, there we go, a non-aesthetic update this week:
napari*=*pyside*
or napari*=*pyqt*
. napari-menu
is noarch, which means that the Bundle CI is greatly simplified (we only need to build on Linux, and works everywhere).And I am kidding, some aesthetics too:
Get them here.
I can confirm the arm64 version works very nicely! 🎉 Gotta use right-click to launch the .pkg due to security, but then it works. One minor gotcha: on my macOS 12 I had to change default location to single-user, home dir. The default location gave an error, which was actually not helpful: "package is incompatible with this version of macOS", but it does mention something about the system volume, so it got me to try the other install location and that worked fine.
The App in Applications is nice, but I was a bit confused at first about the other folder: napari-0.4.13.dev224
I almost nuked it, because I thought it was some install leftover, but then @jaimergp clued me in that this is the conda env! I wonder if this folder should be a dot folder to be hidden by default? or maybe have a different location like ~/Library rather than ~ or maybe name that gives a hint that it's the python env? What's nice is that an advanced user could in theory fix something using the CLI (see: https://github.com/napari/napari/discussions/3899 ), but poking in there I don't see an env, so I'm not sure how/where what would happen and how easy it would be to roll back. Where would using ./pip in the bin dir actually install? Anyhow, it may be worth some advanced instructions, because the potential here is really really high!!
The first launch is a bit slow, without much user feedback—perhaps this could be mentioned in the installer or on the webpage where you download the installer. Later launches are fine, bit slower than using terminal I think.
I really like how the app menu says napari
and not python. Likewise, it shows Hide napari and Quit napari. And in the Activity Monitor it shows up as napari. These small details are nice!
RTF licenses work in both PKG and Windows. It looks nicer on PKG but I guess it's a start? :)
RTF licenses work in both PKG and Windows. It looks nicer on PKG but I guess it's a start? :)
definitely! very nice ❤️
Maybe the link should contain a version in it? Changing of release will change the list of dependencies and the list of licenses.
Maybe the link should contain a version in it?
Absolutely! For now I just added a link to the napari license as a placeholder just to show it works. The real deal will link to the bundle licenses document. I have two ideas to make this release-dependent:
bundle_license.md
document in the repo for each tag and link to its GitHub location (with tag)First is easier to implement (less moving pieces), but maybe less "pretty" and definitely less searchable.
Answering to @psobolewskiPhD (I somehow missed it til now!):
The App in Applications is nice, but I was a bit confused at first about the other folder: napari-0.4.13.dev224 I almost nuked it, because I thought it was some install leftover, but then @jaimergp clued me in that this is the conda env! I wonder if this folder should be a dot folder to be hidden by default? or maybe have a different location like ~/Library rather than ~ or maybe name that gives a hint that it's the python env?
This is super valid feedback and definitely something to consider. Right now we are following Anaconda's choices here, which install to HOME, but no reason to do that. PKGs default is actually ~/Applications
but we don't want that, so maybe ~/Library
is a better choice in this platform. We need to establish which ones would be equivalent in Linux and Windows to make it consistent across platforms.
poking in there I don't see an env, so I'm not sure how/where what would happen and how easy it would be to roll back. Where would using ./pip in the bin dir actually install
The whole directory is the conda environment. You can activate it with your usual conda
if you are already using it (just use the abs path: conda activate ~/napari-x.y.z
) or use the bundled conda
under ~/napari-x.y.z/condabin
. This one will complain about not being "activated" (we don't do that on purpose), but you can use the actual conda
executable (not the shell function) on it just fine. Same with pip
in the the bin
directory.
I really like how the app menu says napari and not python!
Thanks for noticing! Debugging this was a rabbit hole 😂
This is super valid feedback and definitely something to consider. Right now we are following Anaconda's choices here, which install to HOME, but no reason to do that. PKGs default is actually
~/Applications
but we don't want that, so maybe~/Library
is a better choice in this platform. We need to establish which ones would be equivalent in Linux and Windows to make it consistent across platforms.
I would definitely vote to make the conda folder less obvious. For the conda env, I think ~/Library is a reasonable spot where it's out of view...
...but available!
Because, I think it really could be worth writing up some advanced bundle aspects/options that would be linked somewhere—but not part of the default how to napari bundle
.
I get that with the bundle if your env gets screwed you can just reinstall.
And I get that someone who really wants to get into the python stuff should really use conda/mamba-forge and start over.
But, speaking as an end-user, there is inertia—and in my case naivety?—that can make fixing something you have feel better than nuking it and starting with a different thing.
So I think the location and ability to use the bundle conda env, as @jaimergp just explained above, plus maybe the magics
that seem to work:
https://github.com/napari/napari/discussions/3899#discussioncomment-1894365
These are things that could be offered, with a strong disclaimer warning of peril. 🐉
It just feels like as long as plugins and depends are mostly on pip there will always be issues for GUI napari users 🤣
The plugin work will be ironed out with our next work. I have an idea to take pip installs and condafy them on the fly, but it will be veeery experimental.
Regarding docs, I started a packaging
doc in the dev guide. Check here and let me know which aspects you'd like to see covered, I have some spare hours this week!
Latest builds implement the following changes in default installation paths:
~/Library/napari-x.y.z
-- this was sooo tricky to implement T_T~/AppData/Local/napari-x.y.z
(this is %LOCALAPPDATA%
)~/.local/napari-x.y.z
The macOS installer also has a new notification reporting where the shortcuts were added to, as well as the installation contents path.
Regarding advanced users and how they could modify the conda environment, I think we could add a README file with simple docs there, ultimately pointing to the docs website for more info.
Can you remind me how to download the latest version?
Of course! Once the Conda / Create packages
CI job is finished and green, you can go to their Details
link. Then click on the Summary
link in the top left corner and that will take you where that workflow publishes the artifacts. As the platform-specific jobs finish, they will appear there. It takes roughly 20-30min from commit to artifact-ready.
In this case, you will see them here: https://github.com/napari/napari/actions/runs/1672188221
Using right click to launch the pkg, the default install worked now! (I did need to authorize as admin.) Love the new notification, regarding ~/Applications Not finding a napari folder in Library though... Oh, er...not good: it's not in ~/Library but in /Library !! Now I understand the admin requirement. I don't think this is a good idea—typo perhaps?
Edit: reading the notification more carefully it also said /Library/napari
Either way, this is could cause issues with admin vs. non admin users. I think best to put it in user space.
It should default to ~/Library
(and it does on my machine). I wonder if there's a permissions issue on your system that forces the installer to fallback to "all users" mode? I remember your first attempt also tried to use the main volume instead of the "Just me" mode. What macOS version are you running?
Can you upload the full logs of the installer @psobolewskiPhD ? See here for details.
Results from the first notarization:
"issues": [
{
"severity": "error",
"code": null,
"path": "napari-0.4.13.dev241-macOS-x86_64.pkg/main.pkg Contents/Payload/Library/napari-0.4.13.dev241/_conda.exe",
"message": "The binary is not signed with a valid Developer ID certificate.",
"docUrl": null,
"architecture": "x86_64"
},
{
"severity": "error",
"code": null,
"path": "napari-0.4.13.dev241-macOS-x86_64.pkg/main.pkg Contents/Payload/Library/napari-0.4.13.dev241/_conda.exe",
"message": "The signature does not include a secure timestamp.",
"docUrl": null,
"architecture": "x86_64"
},
{
"severity": "error",
"code": null,
"path": "napari-0.4.13.dev241-macOS-x86_64.pkg/main.pkg Contents/Payload/Library/napari-0.4.13.dev241/_conda.exe",
"message": "The executable does not have the hardened runtime enabled.",
"docUrl": null,
"architecture": "x86_64"
}
]
I'm on macOS 12.1 (may have been 12.0.1 when i tested...). I'm re-downloading and will re-run the install and save the log.
Step by step: Double click yields:
So right click to open, yields:
So I click "open" and get the installer. Here's the Installation window, I will just do the default:
Not changing anything, just hitting Install. Password prompt:
Getting the notifications... And done. But the conda dir is in /Library again. Oh FFS, dismissed the window and lost the log! Edit: I will delete and run it again. God I'm useless.
Here's the log from the reinstall and again it's in /Library .
Edit: Also have a wierd () symbol over the icon in Finder:
I don't recall that before. But it launched fine—first launch slow, but then speedy.
Edit: FYI I'm back on EU time now.
Here's the log from the reinstall and again it's in /Library .
Is that the full log or the errors only? I might need all of them :/
Um...I'm not sure? Full I think, because it shows more then errors? 🤣
Hm, that's intriguing. Usually it shows some conda installation details too 🤔
Lemme delete and try again 🤣 deleting takes about as long as installing 🤣 Edit: I think it was "Errors and Progress" -- that's what it showed when I reopened installer. Will set for Full now.
Hmm:
./postinstall: PREFIX=/Library/napari-0.4.13.dev241
Edit: in conda_bundle.py, line 152:
definitions["default_location_pkg"] = "Library"
Should this be "~/Library" perhaps?
Should this be "~/Library" perhaps?
I wish it was that simple :( The PKG installer is actually a bunch of mini PKGs. The parent PKG allows you to set a root location. Then each subPKG can change their install path within that root location. On top of that root + subpkg location, the installer name is added (napari-x.y.z
).
If you choose "Just me", the root is ~
. If you choose "All Users", the root is /
. Then we add Library
to that (this is the line you are pointing to), and then the napari-xyz
name.
This ^ does not apply if you choose a custom location. What I don't get is that you can end up with a /
root location because we are disabling that here.
This might not apply in macOS 12 so for now all I can say is 🤷 ... Let's see if I can upgrade in one of my machines and debug more carefully, but for now I can't think of anything else. Sorry!
No worries! Before—zulip chat—I had to select the Install Only For Me. The default install wouldn't run. Now just clicking through the defaults returns this hybrid where the app is in ~/Apps but the conda is in /Lib.
What about this line: https://github.com/jaimergp/constructor/blob/2369f103f0fee2c5d6b1d10519a6944cf1dea07d/constructor/osxpkg.py#L219
enable_anywhere='true',
Edit: to me that doesn't sound like fully disabled. when combined with: https://github.com/jaimergp/constructor/blob/2369f103f0fee2c5d6b1d10519a6944cf1dea07d/constructor/osxpkg.py#L157
options.set('customLocation', '/')
If there's some way for me to help with tests changing these lines, let me know!
That custom location setting should allow you to select (manually) a custom location anywhere. What we are disabling is an additional entry not currently visible that says "Installation for all users". This was already present in constructor before we started our work so I am guessing it's quite stable, but yes, we'll need to debug a bit.
I'll try and find a minimal constructor example to try and test those options locally on macOS 12.
If you comment out these lines the installer will be minimal.
Unfortunately most of the path manipulations for PKG are pseudo-hardcoded in constructor so you'll need a dev installation of this branch for that to work. constructor/osxpkg.py
is the file you need.
Thank you so much for your help!
PS: We have notarization working as of today! I just need to add the secrets and push my local changes!
By the way @psobolewskiPhD, what kind of content would you like to see in the $NAPARI_BUNDLE_INSTALL/README.txt
file? Some questions in particular?
By the way @psobolewskiPhD, what kind of content would you like to see in the
$NAPARI_BUNDLE_INSTALL/README.txt
file? Some questions in particular?
Where/When does this file get displayed? Sorry, I'm the sort of person that just clicks and hits enter a bunch times until the thing is installed and I can run it!
PS: We have notarization working as of today! I just need to add the secrets and push my local changes!
Let me know when there's a new bundle to test!
sigh
looks like I need conda-standalone
to get anywhere and that's not available on conda-forge for arm64.
Edit: I'm an idiot. Reading the message -> figured it out: --conda-exe (which conda)
Edit2: I don't get it, I cloned and installed your branch of constructor and napari. But, i get a bunch errors regarding [definitions]: Such as:
definitions["welcome_file"] = str(welcome_file)
definitions["conclusion_text"] = ""
definitions["readme_text"] = ""
The worst being:
Error: unknown key 'default_location_pkg' in construct.yaml
because this is what I'm trying to test. If i comment it out, then I can get a .pkg now and it installs in ~ like before. PBCAK I'm sure. 🤷
Where/When does this file get displayed?
It won't be displayed in the installer. We are dropping a README file in the installation root so, in the event a user finds themselves there, they can read on what they can do with their pseudo-conda installation.
Comes from napari/napari#3378
Description
Implements logic discussed in napari/napari#3358 - proof of concept!
Type of change
References
Closes napari/napari#3358
How has this been tested?
menuinst
-enabled package: https://github.com/conda-forge/napari-feedstock/pull/29menuinst
. See https://github.com/users/jaimergp/projects/2Pending tasks
General
Windows
MacOS
/Users/<username>/Applications
(current) or/Applications
? User shortcut does not appear on Spotlight Search; both appear under Launchpad.napari
as the application name__main__.py
from being the app name in the app menuLinux
Other tools
Final checklist:
trans.
to make them localizable. For more information see our translations guide.