search

HDFGroup / hdf-compass

Python-based viewer for HDF5 on other file formats
Other
131 stars 26 forks source link

0.6.0 Release Target #79

Closed jreadey closed 6 years ago

jreadey commented 9 years ago

This issue is to document the set of features/bug fixes for the next (v0.6) HDF Compass Release

See issue #61 for discussion of many of these items.

Excluded from this list (but intended for the next release) are some items (e.g. code signing for Mac binaries) that are a HDFGroup build/deploy process and not a source code issue per se.

ghost commented 9 years ago

@jreadey You can add checkbox for each of the issues and track progress (how many are checked vs. total). See the first listed issue.

jreadey commented 9 years ago

Builds off the latest dev branch checkins are available here:

Please submit any problems you find as an issue (or update an existing issue). If you can run from the source tree, it will be helpful to note if this problem only shows up in install executable or is present in the source version as well.

jreadey commented 9 years ago

Builds for Windows and Mac are updated:

Windows - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-win_amd64.exe Mac OS X - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-macosx.dmg

BAG plugin in (based on my limited testing) is working with this build. I ran into an issue building on Linux - I'll get a new Linux build out later in the week.

hyoklee commented 9 years ago

Mac binary works well even for https://eosdap OPeNDAP server. How did you fix it?

giumas commented 9 years ago

@jreadey How did you freeze on Mac? Py2app or PyInstaller? I am asking in relation to https://github.com/HDFGroup/hdf-compass/issues/107

giumas commented 9 years ago

Although not listed in the milestones, the following PRs add/fix the unittest for the current models:

Some additional work could be done to implement tests for other structures (since only Store and Container are currently implemented) or to add tests for the packages that are not models (e.g. hdf_compass.utils and hdf_compass.compass_viewer). However, the current situation could be good enough for the next release.

jreadey commented 9 years ago

@giumas - I used PyInstaller to freeze the binary on Mac. With py2app I was hitting the problem you described in #107. Using PyInstaller introducing the problem that @andrewcollette mentioned - dragging files to the toolbar icon does not work. I've created an issue for this: #116.

@hyoklee - I didn't do anything special to fix the OpenDAP problem. Maybe it was a beneficial effect of the switch to PyInstaller?

hyoklee commented 9 years ago

@jreadey Thanks for explaining how you packaged. I noticed that Mac's OPeNDAP problem is related to openssl used in requests module under Pydap. If requests is built with openssl 1.0.2.d, Pydap works OK with eosdap server. If it is built with 0.9.8zg, Pydap returns SSL error.

giumas commented 9 years ago

@jreadey Here there are more details on this issue: https://github.com/pyinstaller/pyinstaller/issues/1309

So we could monitor both issues (the PyInstaller and the Py2App) to see what will be fixed first.

jreadey commented 9 years ago

Builds for Windows and Mac are updated:

Windows - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-win_amd64.exe Mac OS X - https://s3.amazonaws.com/hdfgroup.download/HDFCompass/latest/hdfcompass-latest-macosx.dmg

giumas commented 8 years ago

@jreadey I have just added some minimal changes to make optional (at runtime) the cartopy dependency (due to troubles to freeze it on OS X). Here I have uploaded frozen binaries for Mavericks, Yosemite, and Windows: https://github.com/giumas/hdf-compass/releases/tag/0.6.0-beta.1 Let me know if they work!

giumas commented 8 years ago

@jreadey did you have time to look at the binaries?

jreadey commented 8 years ago

@giumas - sorry I haven't had time to work on Compass recently.

I did try freezing the current dev branch on Windows 7 today. I get a pyinstaller error though, "ImportError: no module named cartopy". So I guess that pyinstaller is building the list of imports by scanning the source files, and thus picks up the cartopy anyway.

Do you have access to a Windows 7 box without Cartopy to try this on?

Also, are you able to freeze Compass on OS X?

giumas commented 8 years ago

It is a kind of expected behavior.. You first install cartopy and all the other dependencies and freeze the application, then if there are issues at runtime in loading the cartopy library, HDF Compass does not crash in displaying the geo-arrays, but reverts to plain matplotlib.

I did not have issues to freeze on Win7, the issue with cartopy was on OS X. Specifically, there were two different versions of geos in use by the dependencies, and pyinstaller seems to pick the oldest one. I did not have times to look more on it.. It may be that you with a different environment can get the freezing smoothly.

However, did you try these frozen applications? https://github.com/giumas/hdf-compass/releases/tag/0.6.0-beta.1

giumas commented 8 years ago

This is the trick that I have added for cartopy: https://github.com/giumas/hdf-compass/blob/b951e784b97da81660fdbd68b6f894c002d05e5a/hdf_compass/bag_model/model.py#L389

jreadey commented 8 years ago

I tried the frozen Yosemite application and get the following error. Have you tried it on any Mac that didn't have cartopy installed?

/Applications/HDFCompass.app/Contents/MacOS/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
WARNING shapely.geos.load_dll:50 > Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib)`
Traceback (most recent call last):
  File "<string>", line 36, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/__init__.py", line 14, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/viewer.py", line 34, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/__init__.py", line 14, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/frame.py", line 27, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "hdf_compass/compass_viewer/geo_surface/plot.py", line 21, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "cartopy/__init__.py", line 24, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/speedups/__init__.py", line 3, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geometry/__init__.py", line 4, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geometry/base.py", line 9, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/coords.py", line 8, in <module>
  File "/private/var/folders/82/dkl1x9gs4hzgq8srb474t0vc0000gn/T/pip-build-GiX3sv/pyinstaller/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "shapely/geos.py", line 124, in <module>
  File "shapely/geos.py", line 61, in load_dll
OSError: Could not find lib geos_c or load any of its variants ['/Applications/HDFCompass.app/Contents/MacOS/HDFCompass/libgeos_c.dylib'].
HDFCompass returned -1
giumas commented 8 years ago

Mmm.. I guess that I set the wrong exception: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_viewer/geo_surface/plot.py#L23

It should be OSError in place of ImportError (or a wider Exception). And you are surely right that it was finding the geos library in somehow on my machine.

However, the straight solution would be to fix the geos issue.. We may try with opening a pyinstaller ticket. I will try to put together a minimum example with such an issue.

@QuLogic: I saw that you are an active contributor of cartopy. Have you never successfully frozen an application with cartopy on OS X?

QuLogic commented 8 years ago

No, I don't use OS X. This geos library thing seems to come up a lot there; maybe ask @pelson.

pelson commented 8 years ago

I'm afraid I don't have any experience of freezing applications either. Looking at the traceback, it looks suspiciously like a shapely issue I have seen go by... https://github.com/Toblerity/Shapely/issues/368

Please feel free to ping me if there is anything I can do.

Thanks,

giumas commented 8 years ago

@pelson and @QuLogic : I have finally been able to solve the issue with freezing HDFCompass on OS X with the new cartopy and shapely dependencies by:

About this last PR, I am not sure if it would be the case to delete the existing pyinstaller path: https://github.com/Toblerity/Shapely/blob/master/shapely/geos.py#L88 since it might be never valid. However, I am not confident on doing this deletion, so I am pinging @tallforasmurf and @htgoebel on the subject. I believe that is in somehow related to https://github.com/pyinstaller/pyinstaller/issues/1792 and fixing this part on the shapely side would avoid to have to do it in the shapely hook.

giumas commented 8 years ago

@jreadey Let's try a beta?

giumas commented 8 years ago

@jreadey I have tested the freezing after the PR merge, and shapely seems to work without need to tweak the code.

giumas commented 8 years ago

These are some notes that I have added to the wiki in case that you have troubles to freeze on Mac: https://github.com/HDFGroup/hdf-compass/wiki/Notes-on-freezing-with-Mac-OS-X

jreadey commented 8 years ago

@giumas - ok thanks, I'll give it a try tomorrow.

jreadey commented 8 years ago

@giumas - I saw you had install xcode as the first prerequisite - is this strictly necessary? Previously I was able to do the build on my OS X 10.8 machine (no xcode). If I go to install xcode, it says the 10.11 is required. But building on Yosemite means that the binary will have have that as the minimal OS X version that will work. We seem to have many users who are on a back level OS X, so it would be nice to keep the Mountain Lion support if possible.

giumas commented 8 years ago

I have been able to install xcode on old OS X machines: https://forums.developer.apple.com/thread/15207

giumas commented 8 years ago

@jreadey However, you should be able to use the Command Line Tools in place of XCode.. but I did not try this way.

giumas commented 8 years ago

@jreadey given the issues that you are having to freeze of Windows, why don't share in somehow a VM? https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/windows/

jreadey commented 8 years ago

Preview 0.6 binaries for OS X (Mountain Lion or later) and Windows 7 (or later) are available here:

Windows: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.6.0b5/HDFCompass.exe

OS X: https://s3.amazonaws.com/hdfgroup.download/HDFCompass/0.6.0b5/HDFCompass.dmg

Please try these out and comment here if you find problems. If there are no show stoppers this will go out as the 0.6 release.

oyarzun commented 8 years ago

OS X file seems to be corrupt. screen shot 2016-05-17 at 4 06 45 pm

jreadey commented 8 years ago

@oyazun, sorry about that. Try it now.

oyarzun commented 8 years ago

@jreadey Thanks that one works.

jreadey commented 8 years ago

A new preview binary is available that includes @giumas' fix for the OpenDAP plugin:

Windows: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.exe OS X: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.dmg

Everyone, please post here if you see any problems.

giumas commented 8 years ago

It works on Win7 x64

jreadey commented 8 years ago

Did you see have any problems with the url's mentioned in #60?

giumas commented 8 years ago

https://eosdap.hdfgroup.org:8080/opendap/data/hdf5/grid_1_2d.h5 works http://test.opendap.org/dap/data/nc/coads_climatology.nc works

however, the issue that I wrote on https://github.com/HDFGroup/hdf-compass/issues/60 is still there: python_2016-05-26_18-30-12

I have looked a bit on it.. and I suspect that the issue is due to the fact that we are using wxFileHistory to store urls: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_viewer/frame.py#L87. This would agree with the fact that I have this issue only with urls.

A dirty fix could be to only have the history for files.. A better way would be to create a FileUrlHistory class, but I don't have currently time to look at it. @jreadey , do you remember who added this functionality? he/she may have more insides

giumas commented 8 years ago

I have attempted to comment out this row, and then I am not more getting the error. So it should be that!

ghost commented 8 years ago

@jreadey

Everyone, please post here if you see any problems.

My Mac says: "HDFCompass.app" is damaged or corrupted" and cannot be opened. Tried three times with downloading the dmg file.

jreadey commented 8 years ago

@ajelenak-thg - I just tried again downloading from this link: http://hdfgroup.download.s3.amazonaws.com/HDFCompass/0.6.0b6/HDFCompass.dmg. I got the security warning because it is unassigned but otherwise no problems.

Has anyone else tried the OS X version above?

giumas commented 8 years ago

It works for me on OS X Mavericks Server:

tvnviewer_2016-05-29_17-53-52

giumas commented 8 years ago

As well as on OS X Yosemite:

vmplayer_2016-05-29_18-03-55

giumas commented 8 years ago

@jreadey Incidentally, I noticed that there are some minor inconsistencies in the naming.. like the link sayes 0.6.b6, the about 0.6.b5, the mounted dmg 0.6.0b3, etc.

ghost commented 8 years ago

Nope, does not work for me. Still the same "may be damaged or incomplete" error. OS X Yosemite 10.10.5 here, too.

I have the Install apps from anywhere setting. It used to work without problem until this latest build. I found several posts about this error and all of them suggest allowing installation of apps from anywhere. I have that. This SO answer points to something else.

giumas commented 8 years ago

@ajelenak-thg : I apologize in advance for the silly question, but did you delete first any previous installation? If you freeze the app on your machine, does it work?

tedhabermann commented 8 years ago

I just downloaded the link That John used on Yosemite 10.10.5 and it appears to be working as expected.

Are there good geospatial datasets to try the mapping additions?

Ted

giumas commented 8 years ago

I usually use this small size file for testing: https://bitbucket.org/ccomjhc/hyo_bag/downloads/W00275_MB_50cm_MLLW_1of1.bag, but any of the BAG files present on the NOAA Bathymetric Data Viewer should work fine: http://maps.ngdc.noaa.gov/viewers/bathymetry/

ghost commented 8 years ago

@giumas

I apologize in advance for the silly question

There are no silly questions at this point... :-)

but did you delete first any previous installation?

Yes, tried several times, too.

If you freeze the app on your machine, does it work?

Never done that before so don't have the required toolset.

Don't want to make big deal over this. It's okay if I am the only Mac user in the world who cannot run HDF Compass. :-)

giumas commented 8 years ago

@ajelenak-thg : here you can find the step-by-step instructions: https://github.com/HDFGroup/hdf-compass/wiki/Notes-on-freezing-with-Mac-OS-X

jreadey commented 8 years ago

I'm not sure @ajelenak-thg following the step by step instructions would be very informative. If there is some missing dependency with the install image, I'd rather resolve that so everyone can run the install image.

@ajelenak-thg - Are you able to run the dmg installer? If so, can you runn HDFCompass from the command line? (In shell, cd to: /Applications/HDFCompass.app/Contents/MacOS, and run ./HDFCompass_0.6.0b6)

ghost commented 8 years ago

@ajelenak-thg - Are you able to run the dmg installer?

Yes. The verification when mounting the dmg passes.

If so, can you runn HDFCompass from the command line? (In shell, cd to: /Applications/HDFCompass.app/Contents/MacOS, and run ./HDFCompass_0.6.0b6)

Yes, I can. This is the startup trace displayed in the terminal window:

PyInstaller Bootloader 3.x
LOADER: executable is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: homepath is /Applications/HDFCompass.app/Contents/MacOS
LOADER: _MEIPASS2 is NULL
LOADER: archivename is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: Extracting binaries
LOADER: Executing self as child
LOADER: set _MEIPASS2 to /Applications/HDFCompass.app/Contents/MacOS
PyInstaller Bootloader 3.x
LOADER: executable is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: homepath is /Applications/HDFCompass.app/Contents/MacOS
LOADER: _MEIPASS2 is /Applications/HDFCompass.app/Contents/MacOS
LOADER: archivename is /Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6
LOADER: Already in the child - running user's code.
LOADER: Python library: /Applications/HDFCompass.app/Contents/MacOS/Python
LOADER: Loaded functions from Python library.
LOADER: Manipulating environment (sys.path, sys.prefix)
LOADER: sys.prefix is /Applications/HDFCompass.app/Contents/MacOS
LOADER: Setting runtime options
LOADER: Initializing python
LOADER: Overriding Python's sys.path
LOADER: Post-init sys.path is /Applications/HDFCompass.app/Contents/MacOS
LOADER: Setting sys.argv
LOADER: setting sys._MEIPASS
LOADER: importing modules from CArchive
LOADER: extracted struct
LOADER: callfunction returned...
LOADER: extracted pyimod01_os_path
LOADER: callfunction returned...
LOADER: extracted pyimod02_archive
LOADER: callfunction returned...
LOADER: extracted pyimod03_importers
LOADER: callfunction returned...
LOADER: Installing PYZ archive with Python modules.
LOADER: PYZ archive: out00-PYZ.pyz
LOADER: Running pyiboot01_bootstrap.py
LOADER: Running pyi_rth_pkgres.py
LOADER: Running pyi_rth__tkinter.py
LOADER: Running pyi_rth_mplconfig.py
LOADER: Running pyi_rth_mpldata.py
LOADER: Running HDFCompass.py
/Applications/HDFCompass.app/Contents/MacOS/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
WARNING shapely.geos.load_dll:50 > Failed `CDLL(/Applications/HDFCompass.app/Contents/MacOS/HDFCompass_0.6.0b6/libgeos_c.dylib)`
Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib)`
DEBUG   shapely.geos.load_dll:46 > Trying `CDLL(/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib)`
Library path: '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib'
DEBUG   shapely.geos.load_dll:54 > Library path: '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib'
DLL: <PyInstallerCDLL '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib', handle 7feff3b398f0 at 10b7b1650>
DEBUG   shapely.geos.load_dll:55 > DLL: <PyInstallerCDLL '/Applications/HDFCompass.app/Contents/MacOS/libgeos_c.1.dylib', handle 7feff3b398f0 at 10b7b1650>
Trying `CDLL(/usr/lib/libc.dylib)`
DEBUG   shapely.geos.load_dll:37 > Trying `CDLL(/usr/lib/libc.dylib)`
Library path: '/usr/lib/libc.dylib'
DEBUG   shapely.geos.load_dll:54 > Library path: '/usr/lib/libc.dylib'
DLL: <PyInstallerCDLL '/usr/lib/libc.dylib', handle 7fff64311830 at 10b7b1610>
DEBUG   shapely.geos.load_dll:55 > DLL: <PyInstallerCDLL '/usr/lib/libc.dylib', handle 7fff64311830 at 10b7b1610>
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:190 > Python 2.7.11 64bit on Darwin 14.5.0 (x86_64)
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:192 > numpy 1.11.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:193 > matplotlib 1.5.1
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:194 > wxPython 3.0.2.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:211 > h5py 2.6.0
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:219 > hydroffice.bag 0.2.12
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:222 > lxml 3.6.0 (libxml 2.9.0, libxslt 1.1.28)
DEBUG   hdf_compass.compass_viewer.viewer.load_plugins:235 > pydap 3.1.1 (protocol 2.15)
DEBUG   hdf_compass.compass_viewer.frame.__init__:71 > new frame -> open frames: 1