add option to periodically cause `save`ing of the result

datalad / datalad-crawler

DataLad extension for tracking web resources as datasets

http://datalad.org

Other

5 stars 16 forks source link

add option to periodically cause `save`ing of the result #122

Open yarikoptic opened 1 year ago

yarikoptic commented 1 year ago

ATM if it is to run to fetch 200k files from S3 without needing to commit -- could take awhile and then crash for some reason ruining all work. Would be nice to have some ideally global option to cause save once in a while