Mounted /ipns file system causes very high CPU usage for many small random reads of huge file

Checklist

[X] This is a bug report, not a question. Ask questions on discuss.ipfs.io.
[X] I have searched on the issue tracker for my bug.
[X] I am running the latest go-ipfs version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.io

Version

go-ipfs version: 0.9.1
Repo version: 11
System version: amd64/linux
Golang version: go1.16.6

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://localhost:5001",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "ShardingEnabled": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWG83oXaovkDCQyBteUi1Evts7i81sa8TMaHB6JAeFmi2S"
  },
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "EnableAutoRelay": false,
    "EnableRelayHop": false,
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

I tried to do one exotic thing - to open IPFS hosted English wikipedia (as .zim file) in Kiwix application on Linux (Kiwix was installed through sudo apt install kiwix).

And it appeared a really-really slow CPU-bound process.

I mounted file system /ipfs and /ipns using command ipfs daemon --mount.

ZIM (Kiwix) file is located here /ipns/en.wikipedia-on-ipfs.org/wikipedia_en_all_maxi_2021-02.zim and is 83 GB in size.

It is known that Kiwix reads into memory only small part of file, search index and meta data. And probably Kiwix does a lot of small random file reads when parsing metadata.

I guess probably this is the cause of very high CPU usage - I think IPFS mounted file system (/ipns) is not optimized for small random reads and causes high CPU usage for such things.

I see following behavior when I open webui on my localhost - first IPFS tries to download requested blocks and download speed is quite alright, around 1 MByte/sec, this lasts for 1 minute or so. Then follows a quite long (about 5-10 minutes) period of no download (around 0 speed) but with very high CPU usage for ipfs daemon which can be seen in htop. All 4 CPU cores (8 hardware threads) of ipfs daemon use 95-100% of core. Then again 1 minute of download and again 5-10 minutes of high CPU usage.

This switching between 1 minute of download and 5-10 minutes of high CPU usage repeats for around 6-8 times. Finally after 1 hour Kiwix opens Wikipedia successfully. At this point webui shows around 1 GByte of blocks occupied by Wikipedia cached parts.

Just to note, I didn't do ipfs pin add ... for this file, and I didn't planned to, I wanted to browse distributed remote copy in Kiwix without occupying too much space, 1 GB cache in local IPFS is alright, but not whole 83 GB file.

If I open any article like Book then again a minute of download and few minutes of high CPU usage and article is opened correctly after that with all pictures shown.

To solve similar issues I'll suggest that following can be implemented - if any program tries to read few bytes of data from file at random point then file system code should read from IPFS node 1 MByte instead (aligned) and cache this 1MB in RAM. If again program reads another few bytes within this 1MB block then RAM cache should return those few bytes immediately from cached 1MB, without even contacting any IPFS node infrastructure. If limited RAM cache is full then oldest (or least accessed) 1MB blocks should be removed from RAM.

This suggested 1MB caching tecnique of cause should be tweakable by IPFS config, one can tweak cache size limit and also block size (this 1MB in my example).

I can guess that right now when any small few bytes read is issued to mounted file system then this read request goes through heavy architecture of IPFS node, finds corresponding block, reads small part of it and returns to user, without any in-RAM fast caching. Am I right? Or something else can be causing high CPU usage?

ipfs / kubo