search

insanity54 / futureporn

Unofficial ProjektMelody Chaturbate VOD Archive. For Adults Only.
https://futureporn.net
29 stars 3 forks source link

Cluster pinset is likely missing pins #99

Closed insanity54 closed 1 year ago

insanity54 commented 1 year ago

I spun up a new cluster instance lately, and after all but four of the hashes were pinned, the disk usage was only 251GB. Something's not right here, because expected size is 2TB. Maybe this check could be integrated into status.futureporn.net

I am going to come up with a script to verify that all the hashes on the futureporn API are contained within the cluster pinset

insanity54 commented 1 year ago

Problem is real, indicated by https://github.com/insanity54/futureporn-status/issues/6 when https://github.com/insanity54/futureporn/issues/91 was closed.

I'm going to work on the script today.

insanity54 commented 1 year ago

A quick update. I worked on the script and implemented https://sbtp.xyz/qa/v1/missing-pins

I think there is a bug in this endpoint. I think the missing pins shown are not accurate because when I added a "missing" pin to the cluster, the missing pin list became larger, not smaller.

Might have something to do with the filter arg in ipfs-cluster-ctl pin ls --filter=all. --filter=all and --filter=pins and running without --filter show the same thing.

chris@zed-zed:~$ ipfs-cluster-ctl pin ls --filter=all | wc -l
113
chris@zed-zed:~$ ipfs-cluster-ctl pin ls --filter=pins | wc -l
113
chris@zed-zed:~$ ipfs-cluster-ctl pin ls | wc -l
113

What's up with that? Not sure yet. More research tomorrow

insanity54 commented 1 year ago

So the pin size list weirdness was due to the way https://www.npmjs.com/package/diff works. Feature, not a bug.

I went with a generic diff method https://stackoverflow.com/a/33034768/1004931 and all was well

I also implemented a task that runs every 15 minutes which pins any missing pins