OHDSI / Andromeda

AsynchroNous Disk-based Representation of MassivE DAta: An R package aimed at replacing ff for storing large data objects.
https://ohdsi.github.io/Andromeda/
11 stars 9 forks source link

Add a function to check if Andromeda temp folder can be written to (and is big enough)? #31

Closed schuemie closed 2 years ago

schuemie commented 2 years ago

When running studies, we sometimes find the Andromeda temp folder cannot be created (e.g. because the andromedaTempFolder option is set to an illegal path, or the user doesn't have rights to write there), or that the temp folder isn't big enough. Unfortunately, this typically doesn't happen until well into the study, after some time has passed.

Could we add a function to Andromeda that a study package could call at the very beginning? This could have an optional argument of the minimum required temp space. So for example:

Andromeda::checkTempFolder(minimumSize = 10*1024^3)

would ask Andromeda to make sure that the current temp folder exists, is writeable, and has at least 10GB of space available.

ablack3 commented 2 years ago

Yes I have added that function to the develop branch.

https://github.com/OHDSI/Andromeda/blob/c4e3f24dc5df46fd469a2eb65f1774e7517f67d8/R/LoadingSaving.R#L264

It does seem like there is some ambiguity around the term gigabyte and maybe we should be using 1000^3 (decimal, Gigabyte) instead of 1024^3 (binary, Gibibyte). https://en.wikipedia.org/wiki/Gigabyte#:~:text=Hard%20disk%20capacities,GiB%20(400%20GB)

How about using a larger unit than byte (MB or GB)?

schuemie commented 2 years ago

Thanks! This looks great. I have no preferences as to the unit, and which convention we apply. I'll admit most of the time I'll think in terms of gigabytes (and not caring if that is 1000^3 or 1024^3 ;-) )

ablack3 commented 2 years ago

Great! Ok I'll change the units to gigabytes. That's one less conversion for the user to do. I think getAndromedaTempDiskSpace should use the same unit.

ablack3 commented 2 years ago

I changed all disk size units in Andromeda to gigabytes and pushed changes to develop branch.