search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
128 stars 53 forks source link

Errors Running TestAll After Install on Windows 10 (http://localhost:5101 and root at C:\hsds_data) #210

Closed ron-kuhn closed 1 year ago

ron-kuhn commented 1 year ago

I have the following errors during testall.py after installing on Windows 10 and running locally:

Unit Test

Integ Test

I either fixed or bypassed these errors to get through all the other tests. Are these expected for my environment? Next step for me is to configure for Minio.

jreadey commented 1 year ago

Running on Windows can be problematic. See: https://github.com/HDFGroup/hsds/issues/127. I'd suggest using WSL2 (Windows subsystem for LInux). What command are you using to start HSDS? runall.sh in a github shell?

Comments on the test errors...

What was the issue with testGetSelectionPagination? I don't believe this test actually allocates a large amount of memory.

For the domain_test failures - did you run this immediattely after loading the test data (https://github.com/HDFGroup/hsds/blob/master/docs/post_install.md)? The summary domain information is collected asynchronously after a domain is updated or created. It takes about a minute for the summary information to be available. Try running the tests again and let me know if you are still seeing this error. (You can ignore the sqlite comment - that refers to a previously considered approach of using sqlite to store summary info.)

The dataset_test error looks to a windows specific issue - the test is using os.path to create the h5path, but we want this to always be unix-style paths. Let me take a look at this. The other dataset_test issue with num_chunks and allocated_size is the same issue as with domain_test. Try running this test again.

ron-kuhn commented 1 year ago

I configured my HSDS (Windows 10) to use my local MinIO (WIndows 10). Now these are fixed:

But these are still bad:

jreadey commented 1 year ago

I found the cause of the first two - there's a problem with the windows-style paths that are returned as HSDS walks through the directory tree. Should have a fix soon. It's not a problem with Minio as it always returns unix style paths.

I think there's a similar root cause for the last two issues. I'll look into it.

jreadey commented 1 year ago

@ron-kuhn - Can you try out the "unboundlocal" branch? I put some updates there that I think should resolve all test failures on Windows either with or without Minio.

ron-kuhn commented 1 year ago

I sure will test your fixes. Can you tell me how to build the HDF5 Rest VOL library using windows, so that my HDF5-based library can be used to directly read from HSDS? There is a build_vol_cmake.bat file but it only contains comments.

ron-kuhn commented 1 year ago

With your latest changes, everything is working (including other changes I made to get to work like python cmd). I only tested the MinIO configuration. FYI... HSDS production environment will be linux-based but developer PC's are windows. Any help with HDF5 vol-rest build in Windows?

jhendersonHDF commented 1 year ago

Hi @ron-kuhn,

the last time that I built the REST VOL on Windows, I used the CMake GUI to generate Visual Studio .sln solution files and then let VS build the connector. I'd like to eventually flesh out a .bat file for that, but hadn't gotten around to it yet. The connector could also probably be built with MinGW or similar on Windows, but I haven't tried that.

jreadey commented 1 year ago

The HSDS changes are merged into master. I'll close this issue now.
@ron-kuhn - please open an issue here: https://github.com/HDFGroup/vol-rest/issues if you are still having trouble building the REST VOL on Windows.