search

DiamondLightSource / SuRVoS

(Su)per (R)egion (Vo)lume (S)egmentaton workbench
Apache License 2.0
25 stars 5 forks source link

Loading datasets without copying them #48

Open DimitriosBellos opened 6 years ago

DimitriosBellos commented 6 years ago

Hi, I am currently using SuRVoS with a big dataset and I was wondering if there is a way to prepare the dataset (name it data.h5, place it in a dedicated folder, name the internal dataset '/data') and on the SuRVoS menu select load the dataset instead of opening it.

I would like to do that as my dataset is 57GB and loading it to memory (RAM) in order to make a copy is not an option. I know SuRVoS make a copy of the input dataset because it normalizes it. However, I don't want to normalize my dataset or at least I would like to have the option not to.

I know SuRVoS was not designed for big datasets, but I am planning to process my dataset region by region and my only obstacle is that SuRVoS has to load it as a whole at the beginning. I hope this change is easily applicable as it would allow people to use SuRVoS for big datasets.

Aha! Link: https://dls1.aha.io/features/D-28

srikanthnagella commented 6 years ago

I am not sure how this can be achieved. I will check the code and will come back to you.

srikanthnagella commented 6 years ago

How about adding a data subset selection tool? Here along with the dataset path you select the start, end and stride in each dimension.

survos

DimitriosBellos commented 6 years ago

I agree. The addition of a feature like that would be truly helpful.