Nice to see a modern take on datasets handling in Julia!
I've been looking at DataToolkit trying to understand how to apply it and what specific advantages would it bring. I have three different usecases in mind, and cannot really understand how to plug DataToolkit in any of them.
Briefly outlining them below, any suggestions are welcome!
Small and sporadically-updated table, like 500 rows. Currently, I just put a CSV file into a data folder of the Julia package, and provide a function that reads it into a Julia table with some minor cleanup.
What can DataToolkit improve here?
An online collection of publicly-available tables. For a specific example, astronomical catalogs at https://vizier.cds.unistra.fr. Currently (in VirtualObservatory.jl) I provide a function that's basically download_and_read_table(catalog_id::String) with some conveniences.
There are obvious issues with that: every time the dataset is downloaded anew, and one cannot access a dataset without internet/when the archive is down even if it was downloaded previously. Some transparent caching would be nice.
A large well-structured collection of files (tables, images, ...), think hundreds of GBs. Currently, I manually ensure that the collection is available on the machine I need to work at, and have an interface like MyDataset("path-to-the-directory").
Would be nice to have a per-machine config file so that the path is defined here, and then MyDataset() automatically finds it. Also, maybe some basic presence/sanity checks...
Nice to see a modern take on datasets handling in Julia! I've been looking at DataToolkit trying to understand how to apply it and what specific advantages would it bring. I have three different usecases in mind, and cannot really understand how to plug DataToolkit in any of them. Briefly outlining them below, any suggestions are welcome!
Small and sporadically-updated table, like 500 rows. Currently, I just put a CSV file into a
datafolder of the Julia package, and provide a function that reads it into a Julia table with some minor cleanup. What can DataToolkit improve here?An online collection of publicly-available tables. For a specific example, astronomical catalogs at https://vizier.cds.unistra.fr. Currently (in VirtualObservatory.jl) I provide a function that's basically
download_and_read_table(catalog_id::String)with some conveniences. There are obvious issues with that: every time the dataset is downloaded anew, and one cannot access a dataset without internet/when the archive is down even if it was downloaded previously. Some transparent caching would be nice.A large well-structured collection of files (tables, images, ...), think hundreds of GBs. Currently, I manually ensure that the collection is available on the machine I need to work at, and have an interface like
MyDataset("path-to-the-directory"). Would be nice to have a per-machine config file so that the path is defined here, and thenMyDataset()automatically finds it. Also, maybe some basic presence/sanity checks...