search

filecoin-project / boost

Boost is a tool for Filecoin storage providers to manage data storage and retrievals on Filecoin.
Other
111 stars 72 forks source link

improve visibility of LID within Boost UI #1454

Closed nonsense closed 1 year ago

nonsense commented 1 year ago

Information that should be surfaced

  1. How much TiB/PiB of deal data an SP has
  2. How many unsealed sectors we have
  3. How many sealed sectors, without unsealed copies we have
  4. Piece directory stats per sector? — how much was indexed… what sectors/deals are missing indices?
  1. Split unsealed errors / sectors vs other issues
  2. Display stats around sectors with no unsealed copies vs sectors with unsealed copies (% of number of sectors)
  3. Flagged pieces — number of flagged pieces ; % of data that is flagged, vs total data held by miner
  4. Fix lifecycle of deals — i.e. false positives of flagged pieces
  5. Logs / progress around re-indexing
  6. Block count & Average block size (Inspect Piece Page)

UX Changes

The Inspect page today

image

Proposal

Change the name of the "Inspect" menu item to "Local Index Directory"

This menu item links to a LID landing page. The landing page links to a page with a list of sectors (like `lotus-miner sectors list) and a page with a list of flagged pieces (similar to what we have on the Inspect page today):

image

LexLuthr commented 1 year ago

Can we please also add lotus-miner info output to this list? We should probably have basic info like miner ID etc on our landing page and move deal list to another page. Landing page can also have all the above things as a nice summary. Maybe move settings from their own page to here as well? One stop shop for overview and config.

nonsense commented 1 year ago

@LexLuthr yes, I think this is a good idea. We should bring in a bit more details from lotus-miner into Boost as you suggest.

nonsense commented 1 year ago

In terms of implementation, I think we don't have to store any of this data in Boost, as it can change from underneath us (i.e. if a SP removes unsealed copies, or they get corrupted, or smth like that). We could have a periodic background worker that populates in-memory information about:

That would already give intuition on the health of the unsealed copies + indices on a given SPs system, prior to initialising LID.

brendalee commented 1 year ago

A few additional considerations (copying over from slack):

To make it more clear to SPs where to get more info on if they need to unseal things, or index things, we can consider adding labels - such as data index state (for SPs to see the health of their data indexes), and sector state (to see the status of how many sealed/unsealed sectors they have, etc.).

Adding this could be helpful, when I went back to review this page, I got a bit confused on what Sector Unsealed Copies and Sector Proving State were for in regards to LID.

nonsense commented 1 year ago
Screenshot 2023-06-08 at 17 44 24

The LID landing page frontend part is done. The backend part is hard to get right, considering that we need to collect stats from many components of Boost and Lotus. At the moment we have:

Note that data in DAGstore and LID is currently incorrect - we have expired deals and slashed deals, i.e. sector and deal lifecycle is not handled well within Boost - this is something we should fix somehow as a stopgap, so that we don't display a lot of incorrect data to SPs when we release LID.

Furthermore there are piececids for deals that were submitted with a PSD, but never sealed - again, incorrect metadata within Boost, that should be considered when flagging pieces, as this could be very confusing to SPs.

As an SP when I open the LID landing page, I want 95% (or more) of data on it to be correct and 95% (or more) of flagged pieces to really be repairable or at least to be clear why they are flagged and not unsealed and not retrieveable.