1gb gravity.db due to duplicates?

dMopp / pihole5-adlist-update-cron

A pihole5 compatible cronjob to fetch Adlist(s) from URL and import them to the new gravity.db

GNU General Public License v3.0

32 stars 7 forks source link

1gb gravity.db due to duplicates? #2

Closed p1r473 closed 4 years ago

p1r473 commented 4 years ago

Hello, After running this, the size of gravity.db is 1gb

Looks like there are duplicates. I am viewing the .db file in a SQLite browser and I have 16 million hosts- this is equivalent to how many unique hosts I have. I only have 3 million unique hosts so it looks like duplicates aren't removed.

p1r473 commented 4 years ago

Here is a screenshot showing duplicates are added

p1r473 commented 4 years ago

This could be an issue on Pihole's end. I have opened issue # there. If it can be resolved on their end, I will close this issue here.

p1r473 commented 4 years ago

This is intended functionality apparently of PiHole. I don't agree with the decision so I am going to try and figure out how to delete duplicates myself out of the database https://github.com/pi-hole/pi-hole/issues/3286