bittorrent / go-btfs

BTFS - The First Scalable Decentralized Storage System - A Foundational Platform for Decentralized Applications
https://www.btfs.io
Other
236 stars 80 forks source link

[BUG] Renter contracts sync command stops working after 100 or more files were uploaded #460

Closed mantlik closed 1 week ago

mantlik commented 1 week ago

Describe the bug After 100 or more files are uploaded to BTFS, the command btfs storage contracts sync renter stops working.

To Reproduce Steps to reproduce the behavior:

  1. Upload 100 or more files to BTFS from a single node.
  2. Run command btfs storage contracts sync renter or btfs storage contracts sync -p renter
  3. The command will return the following error after 10 minutes of execution: Error: context deadline exceeded
  4. Contracts are not updated. When -p parameter was used, all contracts are gone:
    btfs storage contracts stat renter
    {"active_contract_num":0,"compensation_paid":0,"compensation_outstanding":0,"first_contract_start":"0001-01-01T00:00:00Z","last_contract_end":"0001-01-01T00:00:00Z","role":1}

Expected behavior The command should update status of all renter contracts.

Screenshots N/A

Desktop (please complete the following information):

Additional context My understanding is that pagination of this upstream API is not working:

        req := &guardpb.ListRenterFileInfoRequest{
            RenterPid:        n.Identity.Pretty(),
            RequesterPid:     n.Identity.Pretty(),
            RequestPageSize:  guardContractPageSize,
            RequestPageIndex: int32(i),
            LastModifyTime:   lastUpdatedTime,
            RequestTime:      &now,
        }

The above code extract is from core/commands/storage/contracts/contracts.go function GetUpdatedGuardContractsForRenter. The ListRenterFileInfo API is likely ignoring RequestPageSize and/or RequestPageIndex parameter. Because of this, once the page length is reached, the API instead of returning 100 files on page index 0 and the rest on page index 1 is always returning all files. Because of it, once 100 files are reached, the break condition

            lastPage = info.Count < info.Request.RequestPageSize
            return nil
        })
        if lastPage {
            break
        }

does not work and the function is stacked in an infinite loop until timeout is reached. Then, because of the timeout, the rest of the process is not executed and no contracts are loaded. In addition, the loop is continuing running in the background even after the command is terminated due to timeout and if the same command is issued multiple times, BTFS daemon is consuming more and more resources and needs to be killed. Changing the break condition to this:

            lastPage = info.Count != info.Request.RequestPageSize
            return nil
        })
        if lastPage {
            break
        }

is a poor workaround, not the fix of the root cause, but with exception of exactly 100 uploaded files it worked and contracts were successfully loaded with the patched code.

yu-btc commented 1 week ago

Thanks for your feedback, we will check it. If the problem is confirmed, it will be optimized in the next version.