search

fire-eggs / Danbooru2021

Python scripts and tools for working with the Danbooru2022 data set. Note: this is a sqlite database and a viewer, not directly related to machine learning.
https://www.gwern.net/Danbooru2021
MIT License
42 stars 2 forks source link

[Question] Rsyncing the dataset with WSL #54

Closed Kelhsy closed 2 years ago

Kelhsy commented 2 years ago

Considering I have a Windows server with the Linux Subsystem, is the dataset and the files (Windows doesn't like certain characters in file names that linux supports) compatible with and will I be able to sync the dataset with rsync onto my NTFS formatted drive?

fire-eggs commented 2 years ago

Hi, I've not found any special characters in gwern's dataset. The folders are numbers from 0000 to 0999 ; the filenames are all numeric; even the raw metadata files don't use special characters!

When I first downloaded the dataset (Danbooru2018) I did so on my Windows NTFS box.

Hope that helps!

Kelhsy commented 2 years ago

Yes it has, thank you, I managed to pull the entire dataset, even if at times error 12 came up I seem to have everything!