disinfoRG / ZeroScraper

Web scraper made by 0archive.
https://0archive.tw
MIT License
10 stars 2 forks source link

use md5 for url_hash #121

Open andreawwenyi opened 4 years ago

andreawwenyi commented 4 years ago

right now we are creating url_hash by using zlib.crc32 in python. It will be less work if we use mysql md5 for url_hash, like we did for redirect_to_hash (issue #120). However this requires some major change in our current code: