search

serverless-dns / blocklists

An opinionated collection of blocklists for RethinkDNS.
https://rethinkdns.com/configure
Mozilla Public License 2.0
85 stars 26 forks source link

Delta updates #19

Closed ignoramous closed 1 year ago

ignoramous commented 2 years ago

Update more frequently (once every hour?), and see if delta updates are possible? Ship only bytes that have changed since the previous version. Like a git-patch.

https://archive.is/RiUEl

ignoramous commented 2 years ago

https://github.com/elfshaker/elfshaker

ignoramous commented 1 year ago

Used xdelta with alexkasko/delta-updater and the result of diff'ing two blocklists was around 76M for just td (trie-data). 

java -jar ./delta-diff/target/delta-diff-1.1.4-SNAPSHOT-distr/delta-diff.jar ../serverless-dns/blocklists__/2022/1668635781244/u6 ../serverless-dns/blocklists__/2022/1669339013468/u6 -o patch.zip

So, the generic delta / diff updates won't rescue us here.

ignoramous commented 1 year ago

VcDiff wasn't any better either ~48MB for trie.

It doesn't looks like off-the-shelf delta / diff algorithms are going to work for us.

This requires some more thought and possibly a custom diff / delta algorithm, for which there's not enough planning or time, right now. May visit a similar approach in the future, if needed.