search

stevejenkins / pihole-cloudsync

Syncs blocklists, blacklists, and whitelists across multiple Pi-holes using a private GitHub repo
MIT License
517 stars 104 forks source link

[Suggestion] Use PyPhTP or implement similar #16

Closed mmotti closed 3 years ago

mmotti commented 4 years ago

Hi,

I don't use cloud-sync as I have only ever had one RPI and I just live with no DNS if things break lol.

In any case, I saw your suggestion on Reddit that one of my other projects may have been suitable for keeping the file size down to allow you to sync. In actual fact, this wouldn't be appropriate as it wouldn't copy enough information across to make a full 1:1 sync.

Anyway, I made this, this morning.

It basically removes all domains from the gravity table which is usually where the bulk of the DB size is, vacuums the DB (which reduces size down to around 0.1MB), exports it to /etc/pihole/PyPhTP/ and then repopulates the gravity table. If you sync the output directory to your slave RPI, you can use the same script to 'inject', or rather overwrite the pi-hole db.

On inject, it copies the gravity.db from /etc/pihole/PyPhTP/ to /etc/pihole/ and then repopulates the gravity table.

I'm not 100% sure whether I need to stop the Pi-hole service to try to prevent locks to the DB but I am trying to avoid that.

Feel free to give it a try though. If nothing else, it may give you an idea of how you might adapt your script to achieve the same outcome.

https://github.com/mmotti/PyPhTP

Example (eject) with loads of random blocklists:

mmotti@ubuntu-server:~$ curl -sSl https://raw.githubusercontent.com/mmotti/PyPhTP/master/PyPhTP.py | sudo python3 - --eject
[i] Pi-hole directory located
[i] Write access is available to Pi-hole directory
[i] Pi-hole DB located
[i] Connection established to Pi-hole DB
[i] gravity.db size: 90.24 MB
[i] Emptying the gravity table
[i] Updating the gravity count in the info table
[i] Running Vacuum
[i] gravity.db size: 0.11 MB
[i] Restarting Pi-hole
[i] Closing connection to the Pi-hole DB
[i] Copying gravity.db to /etc/pihole/PyPhTP/gravity.db
[i] Correcting permissions
[i] Refreshing Gravity for source database

90.24MB down to 0.11MB just by removing the gravity entries.

stevejenkins commented 3 years ago

Thanks! I contemplated using this, but decided against requiring another script. It now queries the DB directly and builds the needed files from there.