andykais / scrape-pages

generalized scraper using a single instruction set for any site that can be statically scraped
https://scrape-pages.js.org
MIT License
6 stars 2 forks source link

make download filenames use database ids #37

Closed andykais closed 4 years ago

andykais commented 4 years ago

currently we use node-sanitize-filename to create sanitized names and save them to a folder. The problem with this approach, is that names are not unique. We could be overriding files when writing. I had thought this was fine since urls should be unique to files, but what about different headers/body/query params? Instead were just going to use database download ids when saving files. The downside is that the data becomes less readable on its own, but thats ok, we still have downloadData attached to a download in the database.