search

JuliaIO / MAT.jl

Julia module for reading MATLAB files
MIT License
279 stars 71 forks source link

matread doesn't work after HDF5 update #32

Closed babaq closed 10 years ago

babaq commented 10 years ago

capture i am running julia-0.4-dev-c6036da on windows 7 with HDF5-0.4 and MAT-0.2.6

simonster commented 10 years ago

The tests pass for me. Do they pass for you? If so, can you give more information about what the file you're reading contains?

babaq commented 10 years ago

when i tried some simple struct variable with some array or string fields , it worked. but doesn't work for my dataset which have 3+ levels of nested struct fields

simonster commented 10 years ago

Can you provide a MAT file or code that reproduces the issue?

babaq commented 10 years ago

i upload the datafile in https://drive.google.com/file/d/0B3O0bZxe7bZoUHdwUDJNVTlCRzA/edit?usp=sharing

simonster commented 10 years ago

Hmm. I can read that file fine on both OS X (with Julia 0.4.0-dev+461) and Windows (with Julia 0.3.0). Can you give the full backtrace? I'm not sure what I can do without reproducing the issue, but it might be helpful.

babaq commented 10 years ago

i tried v0.3 on windows, it doesn't work for me.

capture

simonster commented 10 years ago

Ah, okay. It seems like our new libhdf5 binary for Windows wasn't built with the deflate filter, which MATLAB uses to compress its HDF5 files. @tkelman, any idea what we should do here?

tkelman commented 10 years ago

Ah, crap. So that Arch binary I found wasn't good enough after all. I think the best option is to figure out which configure option we need and submit a spec file to the opensuse build service so we can get the binary through WinRPM. How familiar are you with HDF5 build options?

simonster commented 10 years ago

I'm not at all familiar, but here are the build options used by Homebrew, which works.

tkelman commented 10 years ago

Is the "deflate filter" the same thing as szip, or something different?

simonster commented 10 years ago

Deflate is zlib. It would be good to have szip as well in case we encounter plain HDF5 files that use it, but it seems that deflate is more important.

tkelman commented 10 years ago

Cool, zlib is easy and already available in winrpm. szip is just a little messy to formally package since it's a separate source download and the opensuse build workers don't have network access during a build. If hdf5 is the only thing in the world that uses szip I could put szip in as a second source file, but that doesn't feel as proper as making a separate package for szip first then depending on it in the hdf5 package.

tkelman commented 10 years ago

Oh dear, this is turning out to be kind of a rabbit hole. The way hdf5 builds seems explicitly designed to be hard to cross-compile, it involves running an executable at build time. The Arch package was relying on wine as a build dependency to do this, but I don't think that's going to fly for an official opensuse cross-package. The win64 build of hdf5 would have to be explicitly disabled on 32-bit opensuse build workers, since 32-bit wine can't run win64 executables. We have a few options here.

  1. Try to pre-generate the output files of the executables locally and upload them as extra source files, then skip the generation step at build time. (I'm looking into this first)
  2. Depend on wine and disable 32-bit build workers for the win64 package, and keep the hdf5 package only available from my personal project url without trying to submit it to the official repository. We would have to add the url to WinRPM.sources in the build script of HDF5.jl.
  3. Build an hdf5 binary with zlib support locally and post it to sourceforge. This is what I've done for other packages but this one feels a bit more important to get right so this is my least favorite option.

Opinions? Other ideas? Also paging @timholy, I know you don't use Windows too often but whatever the fix ends up being will come across as a PR at HDF5.jl eventually.

timholy commented 10 years ago

Wow, sorry to hear about the rabbit hole. Sounds pretty rough.

Bottom line is I would accept your decision on this matter; you know so much more about this than I do, it's not even close. That said, the second choice sounds less attractive. I don't understand the problems with the third.

Is there an analog of the OpenSUSE build service where you can build on Windows?

tkelman commented 10 years ago

I don't understand the problems with the third.

A few reasons I'm less enthusiastic lately about just manually building and posting individual binaries, especially for larger, more important, commonly-used libraries:

  1. automation, documentation, and reproducibility of build process
  2. ease of future updates
  3. bus factor
  4. compatibility with dependencies
  5. ability for other libraries to easily depend on one another

@vtjnash had a mailing list post a few weeks back where he elaborated on this here https://groups.google.com/d/msg/julia-dev/4kUu2ieWZ9Q/n10z7dppJH4J (and follow-up discussion here https://groups.google.com/forum/#!topic/julia-dev/EgmjrN3o5iE).

Is there an analog of the OpenSUSE build service where you can build on Windows?

The closest thing that I'm aware of would be AppVeyor, but that's really not AppVeyor's intended purpose and the time limit there might get restrictive for a larger library like HDF5.

I got option 1 to work - I'll have to wait a bit to verify that it builds on the stable version of opensuse in addition to unstable, the build service has been a bit busy lately rebuilding a lot of things. Once they show green in the stable distribution I'll submit them for official inclusion and see what the maintainers think about me pre-generating the configuration files that would otherwise be created at build time. In the meantime you can download the binaries at http://download.opensuse.org/repositories/home:/kelman/openSUSE_Factory/noarch/mingw64-hdf5-1.8.13-34.1.noarch.rpm (on Windows you can open these in 7zip, keep going internally until you see a bin folder which should have the hdf5 dll's in it).

vtjnash commented 10 years ago

opensuse build workers don't have network access during a build

they don't allow it during the build process, but you can create links to urls that will get additionally downloaded before the build

pre-generating the config files sounds like a good solution to me

tkelman commented 10 years ago

you can create links to urls that will get additionally downloaded before the build

Do you know how to do that? I've seen a couple of packages that don't have tarballs but they have what look to be checksum files, but I can't figure out how to use that method.

In case the opensuse maintainers reject my submission, another possible solution might be just adding the line -DHDF5_ENABLE_Z_LIB_SUPPORT=ON to the Arch package's invocation of cmake in its PKGBUILD file. AUR is basically just user-uploaded tarballs, right?

vtjnash commented 10 years ago

haha, not quite, since you still have the issue that you have the wrong / too many copies of the dependencies which is very bad.

I did it for gtk3: https://build.opensuse.org/package/show/windows:mingw:win64/mingw64-gtk3 in the link for uploading a file, there's an option for selecting url which creates the _service file. i don't actually know which style is prefered in OBS.

tkelman commented 10 years ago

Oh it looks like AUR doesn't host any binaries, just PKGBUILD instructions. So people just put Arch binaries up on sourceforge and let pacman do the dependency resolution?

Anyway my mingw64-hdf5 package was accepted (https://build.opensuse.org/package/show/windows:mingw:win64/mingw64-hdf5) with the pre-generated configuration files, so I submitted the 32 bit version too and based on the track record so far that should get approved tomorrow or shortly thereafter. Also need to wait for the package to get built in opensuse 13.1, the build service has been busy all week probably because they're in the process of splitting off the next distribution version 13.2 right now.

simonster commented 10 years ago

Fixed by timholy/HDF5.jl#149.

babaq commented 10 years ago

Thanks, you guys rock!