book_changes takes a long time querying full-history servers

Issue Description

When calling the book_changes method using WebSocket or JSON-RPC, asking for data from older ledgers leads to a long delay or even a request timeout.

Steps to Reproduce

Connect to a full-history rippled server via WebSocket
Send the following request:

{
"command": "book_changes",
"ledger_index": 72381103
}

Expected Result

Server should respond quickly with a value like this:

{
  "result": {
    "type": "bookChanges",
    "ledger_hash": "33C53CF838DEE3C1F74E0BA8BEDEE28D2527E11B1C53846A22E9DBB33EC048C2",
    "ledger_index": 72381103,
    "ledger_time": 708708711,
    "validated": true,
    "changes": [
      {
        "currency_a": "XRP_drops",
        "currency_b": "rLqUC2eCPohYvJCEBJ77eCCqVL2uEiczjA/5852646F67650000000000000000000000000000",
        "volume_a": "520687381",
        "volume_b": "8568683",
        "high": "60.76767128172012",
        "low": "60.76159531559149",
        "open": "60.76159531559149",
        "close": "60.76767128172012"
      },
      {
        "currency_a": "XRP_drops",
        "currency_b": "rhLr8bGvHvBgYXAHNPyXrQAcKGrQ2X5nU4/5854726976694100000000000000000000000000",
        "volume_a": "3852",
        "volume_b": "4950495.0495049",
        "high": "0.000778104000000008",
        "low": "0.000778104000000008",
        "open": "0.000778104000000008",
        "close": "0.000778104000000008"
      },
      {
        "currency_a": "rKWH2NPtXn636uodWJ3dLLM7wHmyG2qpz6/KWH",
        "currency_b": "rTGoNeK6vpu28U2oE5tiqhH3x8z2jBhxv/TGO",
        "volume_a": "0.8",
        "volume_b": "8",
        "high": "0.1",
        "low": "0.1",
        "open": "0.1",
        "close": "0.1"
      }
    ]
  },
  "status": "success",
  "type": "response"
}

Actual Result

Server takes a long time (>30s) to respond if it responds at all. On xrplcluster.com this results in the gateway returning a 504 error.

For more recent ledgers, the server responds quickly with a valid response, so this has something to do with fetching older data specifically. It's unclear how old a ledger you have to ask for before it becomes a problem.

Environment

Tested on xrplcluster, with servers running (I think) rippled 2.1.0 and confirmed by Wietse querying FH servers directly.

Note that this issue does not seem to happen to Clio servers, so I could not reproduce on s1.ripple.com.

XRPLF / rippled