filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.83k stars 1.25k forks source link

lotus lite node: resource limit exceeded #8752

Closed mtelka closed 2 years ago

mtelka commented 2 years ago

Checklist

Lotus component

Lotus Version

Daemon:  1.15.2+mainnet+git.518dc96.dirty+api1.5.0
Local: lotus version 1.15.2+mainnet+git.518dc96.dirty

Describe the Bug

The lotus client query-ask f0... command hangs for some miners when run at lite node.

Analysis showed that the problem is related to these messages in the lite node log:

2022-05-25T16:44:22.068+0200    DEBUG   rcmgr   go-libp2p-resource-manager@v0.2.1/scope.go:242  blocked memory reservation      {"scope": "peer:12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj", "size": 262144, "priority": 255, "stat": {"NumStreamsInbound":1,"NumStreamsOutbound":256,"NumConnsInbound":0,"NumConnsOutbound":1,"NumFD":1,"Memory":67108864}, "error": "resource limit exceeded"}
2022-05-25T16:44:22.069+0200    WARN    data_transfer_network   shared/retrystream.go:92        failed to open stream to 12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj on attempt 3 of 15, waiting 2m36.146221306s to try again, err: peer:12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj: resource limit exceeded

It is strange because the machine have plenty of free memory, so the limit must come internally from the lite node itself, but I didn't found yet where it is set.

This started to appear immediately after lite node upgrade from v1.15.0 to v1.15.2. Downgrade back to v1.15.0 seems to solve the problem.

This is full config.toml for the lite node:

[API]
ListenAddress = "/ip4/REDACTED/tcp/REDACTED/http"

[Libp2p]
ListenAddresses = []

[Client]
SimultaneousTransfersForStorage = 100

Logging Information

2022-05-25T16:44:22.068+0200    DEBUG   rcmgr   go-libp2p-resource-manager@v0.2.1/scope.go:242  blocked memory reservation      {"scope": "peer:12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj", "size": 262144, "priority": 255, "stat": {"NumStreamsInbound":1,"NumStreamsOutbound":256,"NumConnsInbound":0,"NumConnsOutbound":1,"NumFD":1,"Memory":67108864}, "error": "resource limit exceeded"}
2022-05-25T16:44:22.069+0200    WARN    data_transfer_network   shared/retrystream.go:92        failed to open stream to 12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj on attempt 3 of 15, waiting 2m36.146221306s to try again, err: peer:12D3KooWKGCcFVSAUXxe7YP62wiwsBvpCmMomnNauJCA67XbmHYj: resource limit exceeded

Repo Steps

  1. Run lotus lite node v1.15.2
  2. Try to run several lotus client query-ask f0... commands for various miners.
TippyFlitsUK commented 2 years ago

Hey @mtelka 👋

Thank you for taking the time to report this error and provide detailed logs!

I have created a new issue documenting this error. I have also experienced this issue on multiple servers. I have added your ticket details to the new issue, please continue to track the error in the new ticket.

I am closing this issue as your logs relate to a dirty Lotus release. - lotus version 1.15.2+mainnet+git.518dc96.dirty

Many thanks!!