search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.81k stars 1.25k forks source link

Sealing Deals Stall at AddPiece #9921

Open piknikSteven2021 opened 1 year ago

piknikSteven2021 commented 1 year ago

Checklist

Lotus component

Lotus Version

$ lotus-miner version
Daemon:  1.19.0+mainnet+git.64059ca87+api1.5.0
Local: lotus-miner version 1.19.0+mainnet+git.64059ca87

Describe the Bug

After an irregular amount of time since a miner process restart, deals start getting stuck either in StorageDealStaged where it will not move out into AddPiece or the sectors will enter AddPiece state and then get stuck in "preparing" state. So far, the only resolution we've found to this is restarting the miner to allow it schedule properly again.

Talking with TippyFlits here, he believes that the reason this is happening is because there is not enough storage on 3 of the 4 LTS partitions. He has suggested that we balance the storage so that the older/used storage has more space available. This is not an acceptable solution that will not scale. If the scheduler is unable to use the empty storage properly, then that sounds like a bug that needs to be resolved rather than a workaround.

Logging Information

$ lotus-miner storage list
49dc40e7-7577-41ce-b17b-f1eb7bbe9efd:
        [################################################  ] 1.838 PiB/1.91 PiB 96%
        Unsealed: 793; Sealed: 57763; Caches: 58667; Updated: 1730; Update-caches: 1836; Reserved: 0 B
        Weight: 10; Use: Store
        Groups: lts
        Local: /mnt/nas-331f.las1.corp.piknik.com/darma2-dataset
        URL: http://10.32.32.4:2345/remote
        URL: http://10.32.45.7:4856/remote
        URL: http://10.32.32.26:50005/remote
        URL: http://10.32.32.26:50015/remote

ac32e572-dd34-11ec-8f44-4ff44963b889:
        [################################################  ] 1.295 PiB/1.34 PiB 96%
        Unsealed: 4944; Sealed: 30805; Caches: 42628; Updated: 8417; Update-caches: 8523; Reserved: 0 B
        Weight: 10; Use: Store
        Groups: lts
        Local: /mnt/nas-95da.las1.corp.piknik.com/darma2-ceph/ceph1
        URL: http://10.32.32.4:2345/remote
        URL: http://10.32.51.24:4856/remote
        URL: http://10.32.32.26:50005/remote
        URL: http://10.32.32.26:50015/remote

40be2e7a-fb14-401d-b0da-6b848479d7ba:
        [##                                                ] 45.82 TiB/1 PiB 4%
        Unsealed: 887; Sealed: 0; Caches: 0; Updated: 935; Update-caches: 935; Reserved: 0 B
        Weight: 10; Use: Store
        Local: /mnt/nas-af1f.las1.corp.piknik.com/darma2-dataset
        URL: http://10.32.32.4:2345/remote
        URL: http://10.32.51.47:4856/remote
        URL: http://10.32.32.26:50005/remote
        URL: http://10.32.32.26:50015/remote

215a7391-014d-4cc3-9514-dc63f0cf1830:
        [################################################# ] 593.5 TiB/600 TiB 98%
        Unsealed: 1205; Sealed: 16191; Caches: 16346; Updated: 1989; Update-caches: 1989; Reserved: 0 B
        Weight: 10; Use: Store
        Groups: lts
        Local: /mnt/nas-95da.las1.corp.piknik.com/darma2-dataset
        URL: http://10.32.32.4:2345/remote
        URL: http://10.32.32.26:50005/remote
        URL: http://10.32.51.24:4856/remote
        URL: http://10.32.32.26:50015/remote

03d087db-a122-5418-a5cc-f3afe0fcaf4e:
        [#######################                           ] 4.783 TiB/10.4 TiB 46%
        Unsealed: 120; Sealed: 7; Caches: 8; Updated: 10; Update-caches: 9; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-0, pc1-c8d68d
        AllowTo: lts
        URL: http://10.32.15.4:3458/remote (latency: 800µs)
        URL: http://10.32.15.4:3457/remote
        URL: http://10.32.15.24:3456/remote
        URL: http://10.32.15.4:3456/remote
        URL: http://10.32.15.4:4456/remote
        URL: http://10.32.15.4:4457/remote
        URL: http://10.32.15.24:3156/remote

992eaa2f-2210-5f5c-8d47-4c67d032aaeb:
        [#####################                             ] 4.405 TiB/10.4 TiB 42%
        Unsealed: 115; Sealed: 9; Caches: 15; Updated: 14; Update-caches: 14; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-0, pc1-c8af23
        AllowTo: lts
        URL: http://10.32.15.4:3456/remote (latency: 900µs)
        URL: http://10.32.15.26:3456/remote
        URL: http://10.32.15.4:3458/remote
        URL: http://10.32.15.4:3457/remote
        URL: http://10.32.15.4:4456/remote
        URL: http://10.32.15.4:4457/remote
        URL: http://10.32.15.26:3156/remote

c85ac62a-cb3a-575c-866e-89348b7ef62d:
        [###################                               ] 3.99 TiB/10.4 TiB 38%
        Unsealed: 99; Sealed: 12; Caches: 12; Updated: 6; Update-caches: 6; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-1, pc1-c8df6c
        AllowTo: lts
        URL: http://10.32.15.5:4456/remote (latency: 700µs)
        URL: http://10.32.15.5:4457/remote
        URL: http://10.32.15.5:3456/remote
        URL: http://10.32.15.28:3456/remote
        URL: http://10.32.15.5:3457/remote
        URL: http://10.32.15.5:3458/remote
        URL: http://10.32.15.28:3156/remote

2970f712-f514-5ede-ace6-cf9c046a3e8e:
        [###################                               ] 4.028 TiB/10.4 TiB 38%
        Unsealed: 74; Sealed: 11; Caches: 19; Updated: 35; Update-caches: 35; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-0, pc1-c8af21
        AllowTo: lts
        URL: http://10.32.15.25:3456/remote (latency: 900µs)
        URL: http://10.32.15.4:3458/remote
        URL: http://10.32.15.4:3457/remote
        URL: http://10.32.15.4:3456/remote
        URL: http://10.32.15.25:3156/remote
        URL: http://10.32.15.4:4456/remote
        URL: http://10.32.15.4:4457/remote

7ce86529-557c-51cd-9bfc-e4f1d57faf06:
        [####################                              ] 4.284 TiB/10.4 TiB 41%
        Unsealed: 97; Sealed: 12; Caches: 18; Updated: 20; Update-caches: 20; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-1, pc1-cabaaf
        AllowTo: lts
        URL: http://10.32.15.29:3456/remote (latency: 1ms)
        URL: http://10.32.15.5:4456/remote
        URL: http://10.32.15.5:4457/remote
        URL: http://10.32.15.5:3456/remote
        URL: http://10.32.15.5:3457/remote
        URL: http://10.32.15.5:3458/remote
        URL: http://10.32.15.29:3156/remote

ee3c8f42-5062-5474-b1d0-8743d83cf5a4:
        [#####################                             ] 4.468 TiB/10.4 TiB 42%
        Unsealed: 103; Sealed: 14; Caches: 24; Updated: 21; Update-caches: 21; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-workergroup-seal6-1, pc1-c881f8
        AllowTo: lts
        URL: http://10.32.15.27:3456/remote (latency: 900µs)
        URL: http://10.32.15.5:4456/remote
        URL: http://10.32.15.5:4457/remote
        URL: http://10.32.15.5:3456/remote
        URL: http://10.32.15.5:3457/remote
        URL: http://10.32.15.5:3458/remote
        URL: http://10.32.15.27:3156/remote

0c0431bd-d58b-59da-b0f6-380150dbfbca:
        [###############################                   ] 2.192 TiB/3.437 TiB 63%
        Unsealed: 20; Sealed: 13; Caches: 13; Updated: 12; Update-caches: 12; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-snap0-pod0, gpub-337167
        AllowTo: lts
        URL: http://10.32.10.23:3456/remote (latency: 900µs)
        URL: http://10.32.10.23:3458/remote
        URL: http://10.32.10.23:3457/remote
        URL: http://10.32.10.23:4457/remote
        URL: http://10.32.10.23:4456/remote
        URL: http://10.32.10.23:4458/remote
        URL: http://10.32.10.23:3156/remote

30acabc5-3cf3-53c9-b638-3b860fb74e61:
        [##################################                ] 2.399 TiB/3.437 TiB 69%
        Unsealed: 27; Sealed: 12; Caches: 13; Updated: 12; Update-caches: 12; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-snap0-pod0, gpub-337129
        AllowTo: lts
        URL: http://10.32.10.45:3456/remote (latency: 1ms)
        URL: http://10.32.10.45:3457/remote
        URL: http://10.32.10.45:3458/remote
        URL: http://10.32.10.45:4458/remote
        URL: http://10.32.10.45:4456/remote
        URL: http://10.32.10.45:4457/remote
        URL: http://10.32.10.45:3156/remote

ed04c849-fe32-5fab-a2fa-b6a7fb89bdd4:
        [###################                               ] 1.337 TiB/3.437 TiB 38%
        Unsealed: 16; Sealed: 6; Caches: 8; Updated: 7; Update-caches: 9; Reserved: 0 B
        Weight: 1; Use: Seal 
        Groups: fc-snap0-pod0, gpub-33719f
        AllowTo: lts
        URL: http://10.32.10.44:3458/remote (latency: 1ms)
        URL: http://10.32.10.44:3456/remote
        URL: http://10.32.10.44:3457/remote
        URL: http://10.32.10.44:4458/remote
        URL: http://10.32.10.44:3156/remote
        URL: http://10.32.10.44:4456/remote
        URL: http://10.32.10.44:4457/remote

$ lotus-miner sealing jobs
ID        Sector  Worker    Hostname     Task  State        Time
570b2f11  15383   f3be0b6f  gpub-33719f  RU    running      1m53.6s
1e6562b1  13494   e75ea68d  gpua-33a80f  PR2   running      1m9.7s
aef61307  15169   1446497c  gpub-33719f  PR2   running      56.5s
037243c6  15293   28d2418c  gpua-33a425  PR2   running      54.6s
931f2ad6  15324   c0739b21  gpub-337167  PR2   running      32.5s
a2ab606d  13525   46e49564  gpub-337129  PR2   running      26.5s
7f86dafa  13486   bd543081  gpub-33719f  PR2   running      8.7s
00000000  17781   198d9a5b  pc1-c881f8   AP    prepared     4h17m25.7s
00000000  17805   f814296c  pc1-cabaaf   AP    prepared     3h59m59.6s
00000000  17806   50b97586  pc1-c8af23   AP    prepared     3h54m54.5s
00000000  17799   8d098df8  pc1-c8df6c   AP    prepared     3h53m35s
00000000  17785   daa2c2f8  pc1-c8af21   AP    prepared     3h52m24.8s
00000000  17803   19dbc8bc  pc1-c8d68d   AP    prepared     3h51m11.7s
00000000  17812   f0e7a2d5  gpub-337167  AP    prepared     3h46m9.8s
00000000  17810   6f45f743  gpub-337129  AP    prepared     3h43m39.4s
00000000  17815   f0e7a2d5  gpub-337167  AP    prepared     3h42m27.2s
00000000  17813   6f45f743  gpub-337129  AP    prepared     3h41m17.9s
00000000  17796   58d4ebff  gpub-33719f  AP    prepared     3h40m3.3s
00000000  17802   6f45f743  gpub-337129  AP    prepared     3h38m45.7s
00000000  17809   f0e7a2d5  gpub-337167  AP    prepared     3h37m27.8s
00000000  17820   58d4ebff  gpub-33719f  AP    prepared     3h32m39s
00000000  17822   f0e7a2d5  gpub-337167  AP    prepared     3h31m26.3s
00000000  17824   6f45f743  gpub-337129  AP    prepared     3h30m13.9s
00000000  17829   58d4ebff  gpub-33719f  AP    prepared     3h18m54.4s
00000000  17831   58d4ebff  gpub-33719f  AP    prepared     3h14m36.1s
00000000  17826   f814296c  pc1-cabaaf   AP    assigned(1)  3h26m30.4s
00000000  17823   50b97586  pc1-c8af23   AP    assigned(1)  3h25m19.2s
00000000  17838   198d9a5b  pc1-c881f8   AP    assigned(1)  3h24m2.7s
00000000  17837   8d098df8  pc1-c8df6c   AP    assigned(1)  3h21m26.9s
00000000  17834   daa2c2f8  pc1-c8af21   AP    assigned(1)  3h20m8.4s
00000000  17839   19dbc8bc  pc1-c8d68d   AP    assigned(1)  3h17m35.3s
00000000  17817   6f45f743  gpub-337129  AP    assigned(1)  3h15m2.5s
00000000  17832   58d4ebff  gpub-33719f  AP    assigned(1)  3h13m49.1s
00000000  17830   f0e7a2d5  gpub-337167  AP    assigned(1)  3h12m38.4s
00000000  17841   f0e7a2d5  gpub-337167  AP    assigned(1)  3h11m26s
00000000  17844   6f45f743  gpub-337129  AP    assigned(1)  3h10m10.7s
00000000  17825   58d4ebff  gpub-33719f  AP    assigned(1)  3h8m59.9s
00000000  17843   f0e7a2d5  gpub-337167  AP    assigned(1)  3h7m49.1s
00000000  17567   58d4ebff  gpub-33719f  AP    assigned(1)  3h6m34s
00000000  17833   6f45f743  gpub-337129  AP    assigned(1)  3h5m23.8s
00000000  17840   58d4ebff  gpub-33719f  AP    assigned(1)  3h4m9.4s
00000000  17835   f0e7a2d5  gpub-337167  AP    assigned(1)  3h2m56.6s
00000000  17842   6f45f743  gpub-337129  AP    assigned(1)  3h1m44.6s
00000000  17836   daa2c2f8  pc1-c8af21   AP    assigned(2)  3h0m31.4s
00000000  17849   19dbc8bc  pc1-c8d68d   AP    assigned(2)  2h59m20.2s
00000000  17857   198d9a5b  pc1-c881f8   AP    assigned(2)  2h58m8s
00000000  17855   f814296c  pc1-cabaaf   AP    assigned(2)  2h56m55.8s
00000000  17853   50b97586  pc1-c8af23   AP    assigned(2)  2h55m44.8s
00000000  17858   8d098df8  pc1-c8df6c   AP    assigned(2)  2h54m30.5s
00000000  17848   6f45f743  gpub-337129  AP    assigned(2)  2h53m16.9s
00000000  17856   f0e7a2d5  gpub-337167  AP    assigned(2)  2h52m3.9s
00000000  17852   58d4ebff  gpub-33719f  AP    assigned(2)  2h50m52s
$ lotus-miner info
Enabled subsystems (from miner API): [Mining Sealing SectorStorage]
Enabled subsystems (from markets API): [Markets]
Chain: [sync ok] [basefee 228.043 fFIL]
Miner: f01851060 (32 GiB sectors)
Power: 9.31 Pi / 19.1 Ei (0.0477%)
        Raw: 3.583 PiB / 15.26 EiB (0.0229%)
        Committed: 3.583 PiB
        Proving: 3.583 PiB (32 GiB Faulty, 0.00%)
Projected average block win rate: 48.01/week (every 3h29m56s)
Projected block win with 99.9% probability every 24h8m33s
(projections DO NOT account for future network and miner growth)

Miner Balance:    159319.618 FIL
      PreCommit:  0
      Pledge:     56809.097 FIL
      Vesting:    5170.975 FIL
      Available:  97339.546 FIL
Market Balance:   202.344 FIL
       Locked:    178.001 FIL
       Available: 24.343 FIL
Worker Balance:   853.665 FIL
       Control:   7.12 FIL
Total Spendable:  98224.674 FIL

Beneficiary:    f01668004

Sectors:
        Total: 120085
        Proving: 23435
        Available: 93506
        UpdateActivating: 218
        SnapDealsAddPiece: 199
        UpdateReplica: 1
        ProveReplicaUpdate: 5
        ReplicaUpdateWait: 127
        FinalizeReplicaUpdate: 2
        TerminateWait: 6
        TerminateFinality: 9
        Removed: 2561
        ReplicaUpdateFailed: 11
        FinalizeReplicaUpdateFailed: 5

Workers: Seal(47) WdPoSt(1) WinPoSt(1)

Storage Deals: 3876, 94.84 TiB
      Active:            3522  86.09 TiB (Verified: 3522 86.09 TiB)
      AwaitingPreCommit: 25    480 GiB   (Verified: 25   480 GiB)
      Staged:            329   8.281 TiB (Verified: 329  8.281 TiB)

Retrieval Deals (complete): 1, 77 B

Repo Steps

  1. Import deals.
  2. Wait 2-3 days.
  3. Deals start getting stuck at AddPiece.
piknikSteven2021 commented 1 year ago 
$ lotus-miner sealing sched-diag
{
  "CallToWork": {
    "1851060-13451-282eff95-8454-4f3e-bbdf-fa127178c6d0": "seal/v0/provereplicaupdate/2(f9265d546e332bb9db2d189042ed000cad53b8bbd2ad90c8a86eff91c8b15516)",
    "1851060-13486-7f86dafa-7ecb-4bb9-84f8-9407b569fa80": "seal/v0/provereplicaupdate/2(4b89aa80ecacfc2b0b70193b9b388072af6842501ce71f428e2f54d50e1e3bfb)",
    "1851060-13497-6d5398f5-61d2-4fe3-a7f1-2134abc279e7": "seal/v0/provereplicaupdate/2(25e5d297d287500d2f7bb7e166674a0386892b59a6373fb552b6bfe559d0321d)",
    "1851060-13522-174fa160-064c-493b-8908-94be650c288a": "seal/v0/provereplicaupdate/2(1d554cb66b69866b574e2bcd9cfb6ea88a7ed819ef93af730fbd3678f683cfd5)",
    "1851060-13524-da762585-8b0f-4521-8a07-e660a8debf06": "seal/v0/provereplicaupdate/2(2c730705f3b620c51ff0119d8d35345e9b3264003fe6fe0ac241a73e58f6c521)",
    "1851060-13525-a2ab606d-d570-4b95-a9b8-5c1927596d8e": "seal/v0/provereplicaupdate/2(4f0cf86f6b72543ff96689e0e08185eeec013d781fdf702c9a353a9836aa5a3a)",
    "1851060-15309-35172b71-a462-4b41-a116-dc57d9904511": "seal/v0/provereplicaupdate/2(6a0a63d7e5191ab11057fc8fab2e1c2b507434f759130ac8ab2ce281bff6dde2)",
    "1851060-15324-931f2ad6-ad62-429b-8864-1bb1d017a523": "seal/v0/provereplicaupdate/2(d5c0d4bc68c52ddcaaa10289d03d77aa449721d917b874212d549e75fed755fc)",
    "1851060-15383-570b2f11-d0e9-4787-bfce-db210c17d217": "seal/v0/replicaupdate([[{\"ID\":{\"Miner\":1851060,\"Number\":15383},\"ProofType\":8},[{\"Size\":34359738368,\"PieceCID\":{\"/\":\"baga6ea4seaqicd5ejzvbyak4zq7gevcndnfgd766bzhu6lwvxem5mu6sgt536ey\"}}]]])"
  },
  "EarlyRet": null,
  "ReturnedWork": null,
  "SchedInfo": {
    "OpenWindows": [
      "581bcdcb-c85e-4b17-b89f-bdf12cad062d",
      "581bcdcb-c85e-4b17-b89f-bdf12cad062d",
      "eb867f06-8d0d-4d33-838d-25498ca15f3c",
      "9f19db6d-9abc-4b9e-b25a-497216e2118d",
      "e0bc9978-f037-4d77-b6ea-8a9aaf63cbee",
      "8071edd4-c21d-46e2-bc6e-742bbaf673a1",
      "22a93e40-ead3-49b2-99a0-847e17012b1a",
      "eb867f06-8d0d-4d33-838d-25498ca15f3c",
      "f3291e35-39c1-49d1-893b-11d173036751",
      "8071edd4-c21d-46e2-bc6e-742bbaf673a1",
      "9f19db6d-9abc-4b9e-b25a-497216e2118d",
      "e0bc9978-f037-4d77-b6ea-8a9aaf63cbee",
      "dbf45fe1-a96f-4662-8d5f-280c97b5d5f5",
      "22a93e40-ead3-49b2-99a0-847e17012b1a",
      "f3291e35-39c1-49d1-893b-11d173036751",
      "dbf45fe1-a96f-4662-8d5f-280c97b5d5f5",
      "136e08fd-aaca-416c-96e2-5d6bc41eb079",
      "5b35640f-1a44-4c13-8d2e-ad39a513e572",
      "5b35640f-1a44-4c13-8d2e-ad39a513e572",
      "ea07df71-16e8-4e81-8389-97c62535cbf5",
      "2c951330-1e57-4837-8fd4-bdbc7b02baa9",
      "9bcad06a-e394-4584-86d7-17af1ab27ece",
      "ea07df71-16e8-4e81-8389-97c62535cbf5",
      "2c951330-1e57-4837-8fd4-bdbc7b02baa9",
      "7ae8f4d6-a5fb-4d04-8cb8-288483d1d359",
      "9bcad06a-e394-4584-86d7-17af1ab27ece",
      "136e08fd-aaca-416c-96e2-5d6bc41eb079",
      "7ae8f4d6-a5fb-4d04-8cb8-288483d1d359",
      "2f4eaf6d-b9d8-444c-b156-b7ff559e61a4",
      "2f4eaf6d-b9d8-444c-b156-b7ff559e61a4",
      "17850599-a486-4c20-b4fc-f92f454e1ed3",
      "01c9f4ff-bf8a-448d-86e3-732a2c946f35",
      "01c9f4ff-bf8a-448d-86e3-732a2c946f35",
      "17850599-a486-4c20-b4fc-f92f454e1ed3",
      "79482bce-158e-4715-ab10-d9023e239dea",
      "79482bce-158e-4715-ab10-d9023e239dea",
      "da7641f2-8086-4799-ae0b-534ec89f9644",
      "dc76f2f3-8350-4e90-9e59-549d6698435d",
      "2841cb14-9141-4138-a403-28f77b9adf5f",
      "f3be0b6f-a492-4eb1-aa6f-f420f2fc33bb",
      "e75ea68d-f81b-4e85-9ec6-21c3f92a9721",
      "4e50ded0-8c2d-4565-940b-207c34a0dc0f",
      "28d2418c-5867-4868-a698-659044ad10ce",
      "c0739b21-dc67-4f1f-9fb5-d5b6e736822f",
      "2841cb14-9141-4138-a403-28f77b9adf5f",
      "f3be0b6f-a492-4eb1-aa6f-f420f2fc33bb",
      "6aee7673-d78b-4dd9-9046-031e9e9cfe1f",
      "bd543081-0f36-428e-bb06-8a02d753dc4b",
      "46e49564-dde5-4fe0-bbaf-2ead8be9aa27",
      "06d2aa4e-6e97-44dd-bd5a-e91fa85715dc",
      "421ae3e8-c765-4b52-8ece-08bfc1d2c20c",
      "dc76f2f3-8350-4e90-9e59-549d6698435d",
      "2e4a3ba5-6d8f-473a-999d-51f23e7a25a3",
      "e75ea68d-f81b-4e85-9ec6-21c3f92a9721",
      "4e50ded0-8c2d-4565-940b-207c34a0dc0f",
      "1446497c-7eec-4812-b47c-d42e03090b59",
      "28d2418c-5867-4868-a698-659044ad10ce",
      "da7641f2-8086-4799-ae0b-534ec89f9644",
      "c0739b21-dc67-4f1f-9fb5-d5b6e736822f",
      "921d65be-23a9-4e5b-81b7-f8ffe6b81faa",
      "46e49564-dde5-4fe0-bbaf-2ead8be9aa27",
      "bd543081-0f36-428e-bb06-8a02d753dc4b",
      "8589ef2e-2f45-4b58-a18c-fcfccf8d4c77",
      "3185f6a9-1f51-4951-9966-426d7aa4dde7",
      "49e86d01-563b-4afd-a98e-6a2fae048976",
      "921d65be-23a9-4e5b-81b7-f8ffe6b81faa",
      "1446497c-7eec-4812-b47c-d42e03090b59",
      "6aee7673-d78b-4dd9-9046-031e9e9cfe1f",
      "3185f6a9-1f51-4951-9966-426d7aa4dde7",
      "6226681e-8c42-47d9-8842-3501ec3d7fc9",
      "06d2aa4e-6e97-44dd-bd5a-e91fa85715dc",
      "6226681e-8c42-47d9-8842-3501ec3d7fc9",
      "421ae3e8-c765-4b52-8ece-08bfc1d2c20c",
      "49e86d01-563b-4afd-a98e-6a2fae048976",
      "2e4a3ba5-6d8f-473a-999d-51f23e7a25a3",
      "8589ef2e-2f45-4b58-a18c-fcfccf8d4c77"
    ],
    "Requests": [
      {
        "Priority": 1024,
        "SchedId": "4293596d-dacc-4bab-b3bc-aa5b1ceee190",
        "Sector": {
          "Miner": 1851060,
          "Number": 41229
        },
        "TaskType": "seal/v0/finalize/replicaupdate"
      },
      {
        "Priority": 1024,
        "SchedId": "7c2fdbb8-0305-446d-8d1a-432afe02e40d",
        "Sector": {
          "Miner": 1851060,
          "Number": 41230
        },
        "TaskType": "seal/v0/finalize/replicaupdate"
      },
      {
        "Priority": 1024,
        "SchedId": "8aa61f7b-4daa-4186-9e7b-13f09f7173aa",
        "Sector": {
          "Miner": 1851060,
          "Number": 17570
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7573431b-88e9-44ea-a7e3-c9afc829cb6f",
        "Sector": {
          "Miner": 1851060,
          "Number": 17618
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "4aa1729d-1648-4396-953e-78b3b765bc86",
        "Sector": {
          "Miner": 1851060,
          "Number": 17641
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "0e686e80-a95e-4849-8071-dfd36fb92797",
        "Sector": {
          "Miner": 1851060,
          "Number": 17659
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1ec4056a-8b43-46c2-9aae-85c0dd991e4c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17692
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "445df848-b9d4-4f70-8c45-93d5b84f5e50",
        "Sector": {
          "Miner": 1851060,
          "Number": 17693
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b35f1195-34b7-4a82-bf12-e9999a276549",
        "Sector": {
          "Miner": 1851060,
          "Number": 17845
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b40fb692-2f7e-40a9-9c90-d406c0cf8fa7",
        "Sector": {
          "Miner": 1851060,
          "Number": 17846
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "46f00e2d-0cb6-4aeb-b424-d9aaa7971aab",
        "Sector": {
          "Miner": 1851060,
          "Number": 17850
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "12fe0b63-e66f-4b86-a778-bd2684206975",
        "Sector": {
          "Miner": 1851060,
          "Number": 17851
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "2249c726-fe3f-480f-8369-c938e5d1de98",
        "Sector": {
          "Miner": 1851060,
          "Number": 17854
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7ceaedd5-25ca-4dc1-a306-7045f90aacd0",
        "Sector": {
          "Miner": 1851060,
          "Number": 17859
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "4ae74327-ecf2-47fe-ad8c-73a74b31c915",
        "Sector": {
          "Miner": 1851060,
          "Number": 17860
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8d39fd16-c82b-4bcc-b37e-1c37a0d3740a",
        "Sector": {
          "Miner": 1851060,
          "Number": 17861
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "6ca22dd6-8851-436c-acc3-999b374f8132",
        "Sector": {
          "Miner": 1851060,
          "Number": 17862
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "753f01ff-c097-4dd0-9139-712bfbc35761",
        "Sector": {
          "Miner": 1851060,
          "Number": 17863
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "75bc62fa-be49-4332-ab61-7f75f15d93c6",
        "Sector": {
          "Miner": 1851060,
          "Number": 17864
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "daa94930-b8b7-48ac-8b4c-ff0e033fced9",
        "Sector": {
          "Miner": 1851060,
          "Number": 17865
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "13816934-a850-4682-b370-f38aad3d45a0",
        "Sector": {
          "Miner": 1851060,
          "Number": 17866
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "2b04020c-ac25-40f4-9add-1ea128b79a7e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17867
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "36883809-7bdb-4ba4-9089-bfc26fbf52b4",
        "Sector": {
          "Miner": 1851060,
          "Number": 17868
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "4a7884bc-0b4a-4e42-9e77-e88377c8f45c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17869
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "0563719d-510b-43f2-84a9-0efb6164889f",
        "Sector": {
          "Miner": 1851060,
          "Number": 17870
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ff425eab-0c22-428f-96d9-46c763fb8a37",
        "Sector": {
          "Miner": 1851060,
          "Number": 17871
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e617b2bc-a7bf-4fb8-b5fe-dacf84126de0",
        "Sector": {
          "Miner": 1851060,
          "Number": 17872
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "fcf70cba-12e3-40ee-ad58-d03ad9695be0",
        "Sector": {
          "Miner": 1851060,
          "Number": 17873
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "666969d2-e0d5-4cc2-b707-87b3e3774c07",
        "Sector": {
          "Miner": 1851060,
          "Number": 17874
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "c7a041aa-0a74-437e-b65b-3d5bc9abcf8e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17875
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b6cdaa9f-5516-470f-9484-03434e3f5ba1",
        "Sector": {
          "Miner": 1851060,
          "Number": 17876
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "01b72e21-c2e8-4139-99a6-f14fe45a23e5",
        "Sector": {
          "Miner": 1851060,
          "Number": 17877
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "0f52ddd5-86a5-4e11-9613-67cde6d4cc2f",
        "Sector": {
          "Miner": 1851060,
          "Number": 17879
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "d6926412-59fa-4da6-8081-efd5ebfb8977",
        "Sector": {
          "Miner": 1851060,
          "Number": 17880
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b383ade5-0e6d-4c66-b71c-d19f0a94aeb2",
        "Sector": {
          "Miner": 1851060,
          "Number": 17881
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "74ff4a5b-1286-4290-ade8-16f088d5e9b7",
        "Sector": {
          "Miner": 1851060,
          "Number": 17882
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "2799cf48-b6c5-43bc-b2aa-21eacfebf51e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17884
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1183cfa3-72ba-455a-9b3c-f5588d749ede",
        "Sector": {
          "Miner": 1851060,
          "Number": 17885
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ffbf8692-d0d0-45cc-b01f-88c179cdbbff",
        "Sector": {
          "Miner": 1851060,
          "Number": 17886
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "08915f70-6bde-4c8c-a088-c6d2737a8ca4",
        "Sector": {
          "Miner": 1851060,
          "Number": 17887
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "be4fa965-dd83-4f97-8e2b-8efe0aa4d205",
        "Sector": {
          "Miner": 1851060,
          "Number": 17888
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "019343fc-f1ea-4717-aedd-eb864e19e68c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17889
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "253b8608-2a62-42a3-a85e-e0685ab5bd19",
        "Sector": {
          "Miner": 1851060,
          "Number": 17890
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "c6cac44c-5a32-4baa-b3cf-fe13d958f3f7",
        "Sector": {
          "Miner": 1851060,
          "Number": 17891
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "af787d4b-a26f-4f6d-b67b-4db9bc11f29e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17892
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ae193168-7fdb-4e62-a44c-75170cafd96d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17893
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "c3a99aad-8319-44b7-a3cb-e7ff41393d8b",
        "Sector": {
          "Miner": 1851060,
          "Number": 17894
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8dbfd29c-5435-46a2-8679-7d194001ee19",
        "Sector": {
          "Miner": 1851060,
          "Number": 17895
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "871d07c4-d0ed-497c-8a2a-ff14a334629b",
        "Sector": {
          "Miner": 1851060,
          "Number": 17896
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a5c5fb21-1a9a-4ed3-b701-4d87160d287f",
        "Sector": {
          "Miner": 1851060,
          "Number": 17897
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ca2b735f-f646-45aa-8f2a-6140ad7e9218",
        "Sector": {
          "Miner": 1851060,
          "Number": 17898
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "f7869ffd-36c0-4564-b55d-9ae6e4edd90d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17899
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "54de006a-ae0e-4cde-98c6-ecbb0afca4e7",
        "Sector": {
          "Miner": 1851060,
          "Number": 17900
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a9494485-d0bd-4e2f-a2fc-0e0534051745",
        "Sector": {
          "Miner": 1851060,
          "Number": 17901
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "841ec76d-4e39-499e-b040-cf2b1500bc42",
        "Sector": {
          "Miner": 1851060,
          "Number": 17902
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "0d6d5254-d75f-4d1c-babb-a3f091666edd",
        "Sector": {
          "Miner": 1851060,
          "Number": 17903
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "f80ce34f-dd70-488b-a488-025963d17415",
        "Sector": {
          "Miner": 1851060,
          "Number": 17904
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "dea1629b-9dca-4482-a6ef-1e1fdfc0fb96",
        "Sector": {
          "Miner": 1851060,
          "Number": 17905
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7b587651-e0f1-4fb7-9fdf-3cf3ddeadde7",
        "Sector": {
          "Miner": 1851060,
          "Number": 17906
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "d8b75804-26cf-46cd-9e43-e45507d539d3",
        "Sector": {
          "Miner": 1851060,
          "Number": 17907
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8e2fd1e8-1968-4efc-98a1-1b58c00ae536",
        "Sector": {
          "Miner": 1851060,
          "Number": 17908
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ae2def3f-d89f-43b0-a5da-79d60a75eeec",
        "Sector": {
          "Miner": 1851060,
          "Number": 17909
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "01ce52a8-0d6a-49b9-9ec8-ebaa6d7ba61e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17910
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8f7c9a71-a402-4dfe-bd9c-8bd1cebbfe58",
        "Sector": {
          "Miner": 1851060,
          "Number": 17911
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e938fe5f-1939-4a22-a625-3472edfeda45",
        "Sector": {
          "Miner": 1851060,
          "Number": 17912
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "16b83a82-b19a-4d27-b634-452b8a85290d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17913
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "393dfa36-e705-427f-abe3-17002b2755de",
        "Sector": {
          "Miner": 1851060,
          "Number": 17914
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b1157500-429e-4847-9c5d-6eee87ca1980",
        "Sector": {
          "Miner": 1851060,
          "Number": 17915
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b260db09-06b5-4959-8d07-20f3e9926cf5",
        "Sector": {
          "Miner": 1851060,
          "Number": 17916
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1db1a503-5808-4f91-931f-63dc5c0e005d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17917
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8754ba0d-5567-445c-910a-2a8a01cea129",
        "Sector": {
          "Miner": 1851060,
          "Number": 17918
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e4c85738-e402-4a19-9db0-f99849dde017",
        "Sector": {
          "Miner": 1851060,
          "Number": 17919
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "43b00ebc-e017-4c7d-b619-7ad4252794fd",
        "Sector": {
          "Miner": 1851060,
          "Number": 17920
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "5bcd1fce-b736-4257-ab2f-354f31f16e74",
        "Sector": {
          "Miner": 1851060,
          "Number": 17921
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9eafbf2c-8a6e-44bd-9146-0b08d6cb8d35",
        "Sector": {
          "Miner": 1851060,
          "Number": 17922
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "5e11e9de-ea03-4389-987e-87ed34ea2093",
        "Sector": {
          "Miner": 1851060,
          "Number": 17923
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "778a8c94-bc5f-4618-a558-15cd4a3830e4",
        "Sector": {
          "Miner": 1851060,
          "Number": 17924
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "94f3d295-81c1-4243-bb87-61e88464a62d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17925
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "5df3874b-9a97-4c8f-94b0-b3ae58dea261",
        "Sector": {
          "Miner": 1851060,
          "Number": 17926
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "c90e8b2f-ba7c-42b8-8a4a-56422071c2e9",
        "Sector": {
          "Miner": 1851060,
          "Number": 17927
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7d591c1b-4dbc-4779-9dbf-54e5183fc822",
        "Sector": {
          "Miner": 1851060,
          "Number": 17928
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "320cadf0-c704-4b36-9018-63afd6e1c933",
        "Sector": {
          "Miner": 1851060,
          "Number": 17929
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "2c86610c-b71d-45f3-8697-b25a57fd8b28",
        "Sector": {
          "Miner": 1851060,
          "Number": 17930
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "0096a4d9-c2ac-4622-8899-9660578281a3",
        "Sector": {
          "Miner": 1851060,
          "Number": 17931
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "3e881221-f3ba-4c62-8fe7-7e534f1d2b7f",
        "Sector": {
          "Miner": 1851060,
          "Number": 17932
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9503194f-1d1d-4f92-ab1e-5f8997dd50c3",
        "Sector": {
          "Miner": 1851060,
          "Number": 17933
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "aabe64ee-5284-4ebb-94eb-0588a263f002",
        "Sector": {
          "Miner": 1851060,
          "Number": 17934
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1d27b478-c31a-4a92-b6c3-328dc03a3d69",
        "Sector": {
          "Miner": 1851060,
          "Number": 17935
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9cc3c154-6cb8-41f4-9fde-7b772e7ad530",
        "Sector": {
          "Miner": 1851060,
          "Number": 17936
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "10272381-e8a4-4da8-9a37-5d2c7b7c1c70",
        "Sector": {
          "Miner": 1851060,
          "Number": 17938
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "bafd4f88-0e8b-47b0-80b0-03e9b5e18260",
        "Sector": {
          "Miner": 1851060,
          "Number": 17939
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a1e5ec58-927e-4850-b998-83857e96d6b1",
        "Sector": {
          "Miner": 1851060,
          "Number": 17941
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "619fde67-0b75-417a-baf4-27b8c669a8de",
        "Sector": {
          "Miner": 1851060,
          "Number": 17942
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1fcee4c9-16b5-4d1e-a385-ee75a3c762d5",
        "Sector": {
          "Miner": 1851060,
          "Number": 17943
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "06d59164-6071-4966-bf02-65675debb1ab",
        "Sector": {
          "Miner": 1851060,
          "Number": 17944
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7c914fa9-b42b-47de-8b22-4ccd67765feb",
        "Sector": {
          "Miner": 1851060,
          "Number": 17945
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "3e4c4b50-a088-46ad-b261-4c39952e07f9",
        "Sector": {
          "Miner": 1851060,
          "Number": 17946
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9c142dbb-6590-4e20-8c24-ea285ffddc00",
        "Sector": {
          "Miner": 1851060,
          "Number": 17947
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "825301d6-60d7-44c6-976d-bb5366c24dec",
        "Sector": {
          "Miner": 1851060,
          "Number": 17948
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7f791243-0bd8-4e01-8eaa-c856658fba7c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17949
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "67fd1662-1548-4646-b468-2307a9c6841a",
        "Sector": {
          "Miner": 1851060,
          "Number": 17950
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "63a4081c-c7b4-40df-bb7f-3c57e2032fd5",
        "Sector": {
          "Miner": 1851060,
          "Number": 17951
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "75556f76-9afe-4e34-a7c2-f39dbcff15ce",
        "Sector": {
          "Miner": 1851060,
          "Number": 17952
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9ebeac2f-6721-444d-afc0-90300cbd41dc",
        "Sector": {
          "Miner": 1851060,
          "Number": 17953
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "8c1064a3-8502-41ce-89cd-f33413b6b70e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17954
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "3806e4c6-5520-42cb-a39a-46fd93eca2ad",
        "Sector": {
          "Miner": 1851060,
          "Number": 17955
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "1707e370-fc75-4164-a167-da3b1ac2158a",
        "Sector": {
          "Miner": 1851060,
          "Number": 17956
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b2964047-d7f1-41a1-bd6b-aef17411a45b",
        "Sector": {
          "Miner": 1851060,
          "Number": 17957
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "04b15101-462b-4e59-98d7-6662308b63dc",
        "Sector": {
          "Miner": 1851060,
          "Number": 17958
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9604e7d5-91e1-45a4-96f8-67d7c109553d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17959
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a9098929-7bf4-4b6e-8422-c1f8b9e4c689",
        "Sector": {
          "Miner": 1851060,
          "Number": 17960
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "44776bf0-cd47-479c-9d74-8d899cbcfc9c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17961
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "028eaa4c-66f5-46ee-b9ef-e7dc95b730df",
        "Sector": {
          "Miner": 1851060,
          "Number": 17962
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e0fe7652-9d12-475b-8be8-8a37e3ae247e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17963
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "79c2c552-6b36-47f0-9d0a-a966a093db84",
        "Sector": {
          "Miner": 1851060,
          "Number": 17964
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "d16391e1-8139-49fe-8def-10ebcc958b45",
        "Sector": {
          "Miner": 1851060,
          "Number": 17965
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "de572a00-3e8f-4af9-9771-2917c94bce04",
        "Sector": {
          "Miner": 1851060,
          "Number": 17966
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "b34a95b7-f5d8-4e54-a0a0-5986f917816d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17967
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7a7ef944-5930-420b-9bf7-ec3068da8d39",
        "Sector": {
          "Miner": 1851060,
          "Number": 17968
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a8b5889f-4579-4b20-9b4d-2127db19a964",
        "Sector": {
          "Miner": 1851060,
          "Number": 17969
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e411c21e-ad6c-42cf-8b39-081e8b2f6ceb",
        "Sector": {
          "Miner": 1851060,
          "Number": 17970
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "f7f78b96-2cad-4041-b6c4-dc932d2de9d5",
        "Sector": {
          "Miner": 1851060,
          "Number": 17971
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "d20ac67b-7470-4fcb-9901-090745b6b66d",
        "Sector": {
          "Miner": 1851060,
          "Number": 17972
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "cd8be15b-0bb0-43c3-b652-6ea8f9d7b813",
        "Sector": {
          "Miner": 1851060,
          "Number": 17973
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "73d52176-65fd-4698-8e0b-4642c0bf8a0a",
        "Sector": {
          "Miner": 1851060,
          "Number": 17974
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "39c543c9-e23d-40dd-8708-6a4c99024833",
        "Sector": {
          "Miner": 1851060,
          "Number": 17975
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "61bd6b18-98a2-4125-a89a-b10b96f75915",
        "Sector": {
          "Miner": 1851060,
          "Number": 17976
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "5426bbe0-af5c-42d1-9391-6419b33e311e",
        "Sector": {
          "Miner": 1851060,
          "Number": 17977
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "9240b4a5-782e-4f2b-8f59-92aa90d334f0",
        "Sector": {
          "Miner": 1851060,
          "Number": 17978
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "6bb08f03-8f80-472b-967b-3b99d8e0b567",
        "Sector": {
          "Miner": 1851060,
          "Number": 17979
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "7af1acbe-a60f-4fb6-84e1-badc4e98abbf",
        "Sector": {
          "Miner": 1851060,
          "Number": 17981
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "f39b9e42-b626-4fbe-9c4c-e135ad239c42",
        "Sector": {
          "Miner": 1851060,
          "Number": 17982
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a6127c93-65f4-4618-925d-73ce6eeca34b",
        "Sector": {
          "Miner": 1851060,
          "Number": 17983
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "e9b0ce75-e832-49e6-87a7-c37ba7696d83",
        "Sector": {
          "Miner": 1851060,
          "Number": 17984
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "6cebd8fe-88af-4f21-9694-78f9263aec3c",
        "Sector": {
          "Miner": 1851060,
          "Number": 17985
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "ae7508ae-244b-49d6-8d1d-e6d6972b90c1",
        "Sector": {
          "Miner": 1851060,
          "Number": 17988
        },
        "TaskType": "seal/v0/addpiece"
      },
      {
        "Priority": 1024,
        "SchedId": "a330e669-7187-499b-b6aa-356383c6609a",
        "Sector": {
          "Miner": 1851060,
          "Number": 17990
        },
        "TaskType": "seal/v0/addpiece"
      }
    ]
  },
  "Waiting": [
    "seal/v0/provereplicaupdate/2(1d554cb66b69866b574e2bcd9cfb6ea88a7ed819ef93af730fbd3678f683cfd5)",
    "seal/v0/provereplicaupdate/2(d5c0d4bc68c52ddcaaa10289d03d77aa449721d917b874212d549e75fed755fc)",
    "seal/v0/provereplicaupdate/2(6a0a63d7e5191ab11057fc8fab2e1c2b507434f759130ac8ab2ce281bff6dde2)",
    "seal/v0/provereplicaupdate/2(25e5d297d287500d2f7bb7e166674a0386892b59a6373fb552b6bfe559d0321d)",
    "seal/v0/replicaupdate([[{\"ID\":{\"Miner\":1851060,\"Number\":15383},\"ProofType\":8},[{\"Size\":34359738368,\"PieceCID\":{\"/\":\"baga6ea4seaqicd5ejzvbyak4zq7gevcndnfgd766bzhu6lwvxem5mu6sgt536ey\"}}]]])",
    "seal/v0/provereplicaupdate/2(2c730705f3b620c51ff0119d8d35345e9b3264003fe6fe0ac241a73e58f6c521)",
    "seal/v0/provereplicaupdate/2(f9265d546e332bb9db2d189042ed000cad53b8bbd2ad90c8a86eff91c8b15516)",
    "seal/v0/provereplicaupdate/2(4f0cf86f6b72543ff96689e0e08185eeec013d781fdf702c9a353a9836aa5a3a)",
    "seal/v0/provereplicaupdate/2(4b89aa80ecacfc2b0b70193b9b388072af6842501ce71f428e2f54d50e1e3bfb)"
  ]
}
piknikSteven2021 commented 1 year ago 
2022-12-20T20:07:33.095Z        INFO    sectors pipeline/input.go:759   Upgrading sector        {"number": "17986", "type": "deal", "proofType": 8, "expiration": "3982589", "pledge": "0.169711977344992647 FIL"}
2022-12-20T20:07:34.685Z        ERROR   sectors pipeline/input.go:580   we are trying to create a new sector with open sectors map[{1851060 17567}:0xc1f3a00a50 {1851060 17570}:0xc312a437d0 {1851060 17618}:0xc28593fef0 {1851060 17641}:0xc37572f5f0 {1851060 17659}:0xc20de87980 {1851060 17692}:0xc21b7f2540 {1851060 17693}:0xc097c6a2d0 {1851060 17767}:0xc199db3470 {1851060 17770}:0xc1fa7619b0 {1851060 17772}:0xc47283ddd0 {1851060 17775}:0xc0c9a6f980 {1851060 17776}:0xc28f7e42d0 {1851060 17777}:0xc0e78c50e0 {1851060 17778}:0xc0a880b3b0 {1851060 17780}:0xc2eb6ec390 {1851060 17781}:0xc0e83ef1d0 {1851060 17782}:0xc212574b10 {1851060 17783}:0xc0caca2600 {1851060 17784}:0xc1860f09f0 {1851060 17785}:0xc2f0ffef00 {1851060 17787}:0xc25db97cb0 {1851060 17790}:0xc1311c8b10 {1851060 17792}:0xc1266997d0 {1851060 17796}:0xc3e4616e40 {1851060 17797}:0xc14d9f7560 {1851060 17799}:0xc3db513f80 {1851060 17802}:0xc358066660 {1851060 17803}:0xc248170660 {1851060 17805}:0xc2d98ac030 {1851060 17806}:0xc24a5a2b10 {1851060 17809}:0xc2ff99c8a0 {1851060 17810}:0xc1c28e7b00 {1851060 17811}:0xc2810f2390 {1851060 17812}:0xc2e878c780 {1851060 17813}:0xc2ea78c360 {1851060 17815}:0xc33394c0c0 {1851060 17816}:0xc31b4052f0 {1851060 17817}:0xc3a7270ae0 {1851060 17820}:0xc1cdbb7110 {1851060 17822}:0xc0025b5230 {1851060 17823}:0xc1ede4e1e0 {1851060 17824}:0xc522b9fb90 {1851060 17825}:0xc1c43adcb0 {1851060 17826}:0xc149a043f0 {1851060 17828}:0xc29b4b6a50 {1851060 17829}:0xc1f031d1d0 {1851060 17830}:0xc0fbac3f80 {1851060 17831}:0xc2f5d2f470 {1851060 17832}:0xc0de90f530 {1851060 17833}:0xc216727320 {1851060 17834}:0xc143b97da0 {1851060 17835}:0xc1646d1b90 {1851060 17836}:0xc09eb99740 {1851060 17837}:0xc2412ad7d0 {1851060 17838}:0xc2d8a2bbf0 {1851060 17839}:0xc2cb6e91a0 {1851060 17840}:0xc2272b08d0 {1851060 17841}:0xc0f363d7a0 {1851060 17842}:0xc2ca1f6a50 {1851060 17843}:0xc0cf9a64e0 {1851060 17844}:0xc276889080 {1851060 17845}:0xc2e0839a10 {1851060 17846}:0xc157360360 {1851060 17848}:0xc522b9e3c0 {1851060 17849}:0xc1c182a720 {1851060 17850}:0xc3dee0b9e0 {1851060 17851}:0xc0daff04b0 {1851060 17852}:0xc4349d6840 {1851060 17853}:0xc083ba3ef0 {1851060 17854}:0xc47d177860 {1851060 17855}:0xc195a4fbc0 {1851060 17856}:0xc3ee364db0 {1851060 17857}:0xc492370390 {1851060 17858}:0xc0d053c9f0 {1851060 17859}:0xc1f90cde60 {1851060 17860}:0xc3415a4f00 {1851060 17861}:0xc16c25dbf0 {1851060 17862}:0xc3364769f0 {1851060 17863}:0xc131ff4450 {1851060 17864}:0xc366babb90 {1851060 17865}:0xc22ca89110 {1851060 17866}:0xc1cb5fa390 {1851060 17867}:0xc42549a510 {1851060 17868}:0xc1a61bc5a0 {1851060 17869}:0xc398097b90 {1851060 17870}:0xc0e48595c0 {1851060 17871}:0xc0d15968a0 {1851060 17872}:0xc1e4339650 {1851060 17873}:0xc286153830 {1851060 17874}:0xc24db12a20 {1851060 17875}:0xc0e052a1e0 {1851060 17876}:0xc09dd92c60 {1851060 17877}:0xc0c7f9d200 {1851060 17879}:0xc000560750 {1851060 17880}:0xc2b8604c90 {1851060 17881}:0xc376620600 {1851060 17882}:0xc14a7f85d0 {1851060 17884}:0xc11687bdd0 {1851060 17885}:0xc2da2fd7d0 {1851060 17886}:0xc1e5d35f20 {1851060 17887}:0xc2073d0de0 {1851060 17888}:0xc10a7a75f0 {1851060 17889}:0xc4a0097860 {1851060 17890}:0xc1182cf3b0 {1851060 17891}:0xc146803f80 {1851060 17892}:0xc41c1ba870 {1851060 17893}:0xc0eeaf7230 {1851060 17894}:0xc361194240 {1851060 17895}:0xc1ac4497a0 {1851060 17896}:0xc214e48420 {1851060 17897}:0xc386a93bc0 {1851060 17898}:0xc2364bff20 {1851060 17899}:0xc0f0f15d40 {1851060 17900}:0xc2b1c6af60 {1851060 17901}:0xc094244240 {1851060 17902}:0xc1fb23d770 {1851060 17903}:0xc1f8f57b90 {1851060 17904}:0xc31b5a1fb0 {1851060 17905}:0xc0c54755c0 {1851060 17906}:0xc25522f200 {1851060 17907}:0xc082a5d0e0 {1851060 17908}:0xc0974a78c0 {1851060 17909}:0xc1cea2c570 {1851060 17910}:0xc21a41fe30 {1851060 17911}:0xc2bb3d3c50 {1851060 17912}:0xc1ff79f7a0 {1851060 17913}:0xc0b0e208d0 {1851060 17914}:0xc13d8bc8a0 {1851060 17915}:0xc3a7270cc0 {1851060 17916}:0xc1048db620 {1851060 17917}:0xc1cb2cc840 {1851060 17918}:0xc1929d5530 {1851060 17919}:0xc0cc51d6b0 {1851060 17920}:0xc3ac337260 {1851060 17921}:0xc23f25ae10 {1851060 17922}:0xc1415a2480 {1851060 17923}:0xc2194a2780 {1851060 17924}:0xc0c98e8810 {1851060 17925}:0xc168813710 {1851060 17926}:0xc2a0d4c2a0 {1851060 17927}:0xc234a41bc0 {1851060 17928}:0xc218fdf260 {1851060 17929}:0xc3700a3740 {1851060 17930}:0xc20df7fd40 {1851060 17931}:0xc092f05080 {1851060 17932}:0xc1f6fdcea0 {1851060 17933}:0xc2698685a0 {1851060 17934}:0xc1578beff0 {1851060 17935}:0xc21c72e9f0 {1851060 17936}:0xc15cea8a20 {1851060 17938}:0xc0851ed3b0 {1851060 17939}:0xc298a8e000 {1851060 17941}:0xc139ca87b0 {1851060 17942}:0xc13d8bc840 {1851060 17943}:0xc281290e10 {1851060 17944}:0xc27f930090 {1851060 17945}:0xc25d2fe0c0 {1851060 17946}:0xc143e7f2c0 {1851060 17947}:0xc15ab19620 {1851060 17948}:0xc168aeced0 {1851060 17949}:0xc13468ca80 {1851060 17950}:0xc1942c5800 {1851060 17951}:0xc124bd3230 {1851060 17952}:0xc1d0ad3950 {1851060 17953}:0xc31d9d8a50 {1851060 17954}:0xc0d9e69590 {1851060 17955}:0xc0da891530 {1851060 17956}:0xc36d350480 {1851060 17957}:0xc2c22f25a0 {1851060 17958}:0xc178bbcea0 {1851060 17959}:0xc128e48900 {1851060 17960}:0xc0560a7aa0 {1851060 17961}:0xc1e418d4d0 {1851060 17962}:0xc146803470 {1851060 17963}:0xc0d3ce5800 {1851060 17964}:0xc1886c3620 {1851060 17965}:0xc3c27da0f0 {1851060 17966}:0xc05dd48480 {1851060 17967}:0xc1c64dcc30 {1851060 17968}:0xc019cfccf0 {1851060 17969}:0xc31d8b1d10 {1851060 17970}:0xc279937500 {1851060 17971}:0xc16a5c8570 {1851060 17972}:0xc1d463a480 {1851060 17973}:0xc1c6f6bf50 {1851060 17974}:0xc38f28a330 {1851060 17975}:0xc3414df9e0 {1851060 17976}:0xc2e6e420f0 {1851060 17977}:0xc213878a80 {1851060 17978}:0xc23f5be660 {1851060 17979}:0xc2fa6bcdb0 {1851060 17981}:0xc45327c630 {1851060 17982}:0xc42986e870 {1851060 17983}:0xc250e068a0 {1851060 17984}:0xc2f16b27e0 {1851060 17985}:0xc1a9e5b2c0 {1851060 17986}:0xc2b6ba5bc0 {1851060 17988}:0xc31475e090 {1851060 17990}:0xc098c8ed20 {1851060 17996}:0xc1a2daae70]
piknikSteven2021 commented 1 year ago

AddPiece Logs:

$ journalctl -u lotus-worker-ap@0.service -f -o cat
2022-12-20T16:04:21.570Z        DEBUG   advmgr  sealer/worker_local.go:165      acquired sector {{1851060 17788} 8} (e:0; a:1): {{0 0} /mnt/gpub-337129/scratch/storage/unsealed/s-t01851060-17788    }
2022-12-20T16:04:22.394Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-12-20T16:04:32.395Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-12-20T16:04:42.396Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-12-20T16:04:52.398Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -12288
2022-12-20T16:05:02.398Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -32768
2022-12-20T16:05:12.399Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -32768
2022-12-20T16:05:22.400Z        WARN    stores  paths/local.go:155      negative reserved storage: p.reserved=34359738368, reserved: -32768
2022-12-20T16:05:33.722Z        DEBUG   advmgr  sealer/worker_local.go:165      acquired sector {{1851060 17794} 8} (e:1; a:0): {{0 0} /mnt/gpub-337129/scratch/storage/unsealed/s-t01851060-17794    }
2022-12-20T16:06:44.029Z        DEBUG   advmgr  sealer/worker_local.go:165      acquired sector {{1851060 17788} 8} (e:1; a:0): {{0 0} /mnt/gpub-337129/scratch/storage/unsealed/s-t01851060-17788    }
^C
$ date
Tue 20 Dec 2022 07:00:48 PM UTC
TippyFlitsUK commented 1 year ago

It should also be noted that this server setup is only processing snap deals and the issue started appearing when there were only 3 LTS storage locations at 96%, 98% and 98% capacity respectively.

The fourth 1 PiB storage location was added a couple of days ago as a result of the Slack discussion that Steven has linked.

Lotus does already provide a MaxStorage option which can be set in the LTS sectorstore.json file.

Thank you for providing such comprehensive logs, Steven!! 🙏

RobQuistNL commented 1 year ago

Are you running 1.2.0 imports with remotecommp enabled?

piknikSteven2021 commented 1 year ago

We do not run with Remote CommP.