davidMcneil / mnist

MNIST data set parser https://crates.io/crates/mnist
20 stars 9 forks source link

Replace certain dependencies (reqwest -> curl + pbr, rulinalg -> ndarray) #8

Closed quietlychris closed 2 years ago

quietlychris commented 3 years ago

As mentioned by @bytesnake in #7, the reqwest crate is fairly heavy in its dependency requirements. To get the same functionality, I've replaced with a combination of curl and pbr (an ascii progress bar crate) for visualizing download progress.

Furthermore, since the standard BASE_URL used by the crate at http://yann.lecun.com/exdb/mnist/ has often ended up being down, I've added a new function to the Builder struct that allows passing a custom base url. I've tested this on cmoran.xyz (with the associated .gz filenames on the end), which seems to work fine. As a result, the datasets that have been traditionally included in this repository have been removed, and any data downloaded to data/* should be ignored by git.

Finally, since rulinalg is no longer maintained and the scientific computing community has seemed to center more around ndarray, I've replaced that dependency along with an example of how to convert the MNIST data into an Array3 data structure, like one might use for testing a neural network.

Happy to answer any questions or discuss changes which might make sense.

quietlychris commented 3 years ago

@davidMcneil Thank you! The most recent commit should address the comments that you and Lorenz made, but if there's anything else, just let me know!

bytesnake commented 2 years ago

ping