IPFS not respecting StorageMax

koalalorenzo commented 5 years ago

Version information:

go-ipfs version: 0.4.20-
Repo version: 7
System version: amd64/linux
Golang version: go1.12.4

Type: Bug

Description:

After a while or running 0.4.20 in production and serving content, the system seems to not respect the StorageMax rule in the configuration.

I currently have these settings in the configuration:

[...]
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "8GB"
  },
[...]

But the size of the ipfs directory is 15GB

user@eu-helsinki1:/opt/ipfs# du -h --max-depth=1
15G ./blocks
4.0K    ./keystore
61M ./datastore
15G .

user@eu-helsinki1:/opt/ipfs# ipfs repo stat
NumObjects: 72290
RepoSize:   15456415465
StorageMax: 8000000000
RepoPath:   /opt/ipfs/
Version:    fs-repo@7

The GC is enabled when starting the daemon (using the args). Is there any way to enforce the GC via the configuration? That might be an issue.

Stebalien commented 5 years ago

So, we currently run GC when (a) it's enabled and (b) we surpass the storage max. This is likely a duplicate of https://github.com/ipfs/go-ipfs/issues/5140 but I'll keep this open until we can confirm that.

koalalorenzo commented 5 years ago

The issue is that the IPFS gateway is running with an actual option to enable gc.

I run manually the gc to solve the issue, but often it fills up the disk.

A good long-term solution would be to use some more "standard" way to configure IPFS from both CLI as well as the configuration file. @Stebalien as a short solution, is there anything I can help to debug this?

Stebalien commented 5 years ago

I wonder if the daemon is restarting before it tries to GC? If you'd like to debug this, you'll probably need to dig into the code (and shorten the GC timeouts to see where this is going wrong).

ipfs / kubo