dat-ecosystem-archive / svalbard

A global metadata vault [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
62 stars 6 forks source link

FTP metadata crawler #13

Open max-mapper opened 7 years ago

max-mapper commented 7 years ago

Work in progress by @clkao https://gist.github.com/clkao/cd5975e4a136269f092bb001b95627cd

Goal is to be able to continuously monitor an FTP server and produce URL lists with its contents (file names, urls, file sizes, mime types if it's available from the .listings metadata)