enable s3 vfd

[grimbough]

@mikejiang just wanted to let you know I haven't forgotten about this. The 1-10-6 branch now has that version of the library with the --enable-ros3-vfd option turned & binaries for Windows built using both toolchains.

However rhdf5 is now failing some unit tests on my machine, so I'm looking into those before pushing it to Bioc-devel. Feel free to try the branch and let me know if the S3 functionality is working for you, I don't have any tests for that at the moment.

[mikejiang]

I've tested 1-10-6, s3 works perfectly!

[vjcitn]

Is the main distribution in devel going to be s3vfd-capable? i would like to experiment with this.

[grimbough]

Yep, the devel versions of Rhdf5lib and rhdf5 should have the S3 VFD enabled.

There's a vignette detailing the enabled functionality in rhdf5 at Reading HDF5 Files In The Cloud. You'll need to look at the source and/or read the HDF5 docs if you want to use Rhdf5lib directly.

This doesn't work on Windows yet, I think due to problems linking against libssl/libcurl that I haven't figured out yet, but tests fine on Linux & OSX for me.

[vjcitn]

i compiled from source on mac. do i need to set configure flags?

%vjcair> R

R version 4.0.1 beta (2020-05-28 r78607) -- "See Things Now"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

library> library(rhdf5)
1/2 packages newly attached/loaded, see sessionInfo() for details.
> public_S3_url <- "https://rhdf5-public.s3.eu-central-1.amazonaws.com/h5ex_t_array.h5"
> h5ls(file = public_S3_url,
+      s3 = TRUE)
Error in H5Pset_fapl_ros3(fapl, s3credentials) : 
  Rhdf5lib was not compiled with support for the S3 VFD

Enter a frame number, or 0 to exit   

1: h5ls(file = public_S3_url, s3 = TRUE)
2: H5Pset_fapl_ros3(fapl, s3credentials)

Selection: 0
> sessionInfo()
R version 4.0.1 beta (2020-05-28 r78607)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14.6

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] rhdf5_2.33.7  rmarkdown_2.3

loaded via a namespace (and not attached):
 [1] compiler_4.0.1     startup_0.14.1     tools_4.0.1        htmltools_0.5.0   
 [5] rhdf5filters_1.1.2 knitr_1.29         xfun_0.15          digest_0.6.25     
 [9] rlang_0.4.7        evaluate_0.14      Rhdf5lib_1.11.3  

[grimbough]

Configure should look for libcrypto and libcurl in your library path, and if it doesn't find them it turns off support for the S3 VFD, but continues with the installation with the same features as in the release branch. That leads to the message you're seeing.

You might need to set LD_LIBRARY_PATH if those are installed but not found by R. The features section the HDF5 configure output will include something like Read only S3-vfd: yes if it's found them successfully.

I can take a look at my Mac settings & also update the vignette with some instructions when I'm back from vacation.

[vjcitn]

Thanks. Worked fine on linux box, should have tried that first.