search

pokt-network / poktroll

The official Shannon upgrade implementation of the Pocket Network Protocol implemented using Rollkit.dev
MIT License
15 stars 6 forks source link

[RelayMiner] RelayMiner Lean Client #528

Closed RawthiL closed 2 weeks ago

RawthiL commented 1 month ago

Objective

Enable a N:1 Supplier:RelayMiner relation, i.e. connect multiple Suppliers to the same RelayMiner process.

This is required for mid-large node runners operations and economical constraints.

Origin Document

This comment refers the a N:1 Supplier:RelayMiner relation as "not okay" when it is fundamental for mid-large node runners.

Without this feature the hardware costs of these operations will be at least 6 times higher due to hardware requirements.

More context at this discord thread.

Related Lean Pocket data:

Goals

Deliverables

Non-goals / Non-deliverables

General deliverables

Creator: @RawthiL Co-Owners: @jorgecuesta

Olshansk commented 1 month ago

Thanks for bringing this up @RawthiL and @jorgecuesta!

I updated some parts of the issue, but for reference, here is the implementation in Morse: https://github.com/pokt-network/pocket-core/pull/1463/files

After synching on this with @okdas, we believe this should be "as simple as" the business logic that would accompany the following configuration changes:

- signing_key_name: supplier1
+ default_signing_key_name: supplier1
smt_store_path: smt_stores
metrics:
  enabled: true
  addr: :9090
pocket_node:
  query_node_rpc_url: tcp://127.0.0.1:36657
  query_node_grpc_url: tcp://127.0.0.1:36658
  tx_node_rpc_url: tcp://127.0.0.1:36657
suppliers:
  - service_id: anvil
+ # implicitly supplier1
    listen_url: http://localhost:6942
    service_config:
      backend_url: http://localhost:8081
      publicly_exposed_endpoints:
        - localhost
  - service_id: svc2
+  signing_key_name: supplier2
    listen_url: http://localhost:6943
    service_config:
      backend_url: http://localhost:8082
      publicly_exposed_endpoints:
        - notlocalhost
pprof:
  enabled: false
  addr: localhost:6060
red-0ne commented 1 month ago

With some additional specs w.r.t. how to route the request to the right supplier, this is how I imagined it could be.

red-0ne commented 1 month ago

Given that:

  1. We can't prevent different Suppliers from staking with the same service_id.
  2. We can't prevent different Suppliers to have the same publicly_exposed_endpoint.
  3. Our current implementation allows having different services being handled by the same listen_url (which is great for lean clients).

Assuming that we have a config with 2 Suppliers in our lean client like the following:

suppliers:
  - service_id: anvil
    signing_key_name: supplier1
    listen_url: http://localhost:9999
    service_config:
      backend_url: http://localhost:8081
      publicly_exposed_endpoints:
        - leanclienthost

  - service_id: anvil
    signing_key_name: supplier2
    listen_url: http://localhost:9999
    service_config:
      backend_url: http://localhost:8082
      publicly_exposed_endpoints:
        - leanclienthost

Here, both Suppliers provide the same service_id on the same listen_url and publicly_exposed_endpoints. Which makes it ambiguous on how to select which Supplier will be signing requests.

With the idea of:

  1. Keeping this config structure (because it's simple and awesome).
  2. Not adding any in-chain checks or restrictions when staking (unique publicly_exposed_endpoints across different Suppliers).
  3. Not enforcing unique listen_urls in the config. Because we'd have to run 2500 HTTP servers if we want to collocate 2500 different Suppliers

We're left with a few solutions:

  1. Enforce in the config that different Supplierss do not provide the same service_id. If supplier1 provides svc1, then supplier2 cannot. (I personally don't like this solution and suspect lean client operators would want to have multiple Suppliers with for the same service_id)
  2. Add a supplier_address field in the RelayRequest proto like the following 
    message RelayRequest {
RelayRequestMetadata meta = 1 [(gogoproto.nullable) = false];
bytes payload = 2;
+ string supplier_address = 3;
}

    Which will be populated by the Gateway when selecting a Supplier for a RelayRequest.

This is possible because:

  1. We select the Supplier (url) before signing the RelayRequest
  2. We can omit the supplier_address field from the signable bytes if we want to sign before selecting a Supplier or keep the relay request acceptable by any Supplier from the Session.

I believe the later is the simplest solution, and the most direct path to have lean clients given our current implementation.

jorgecuesta commented 1 month ago

I believe the later is the simplest solution, and the most direct path to have lean clients given our current implementation.

okdas commented 1 month ago

@red-0ne is there any way to get information about which supplier relay is arriving for using RelayRequestMetadata? Seems like the Session has a list of Suppliers:

repeated poktroll.shared.Supplier suppliers = 6;

I think we can match/route to suppliers on RelayMiner based on that array. Or am I thinking nonsense?

Btw, if we translate the current (Morse) lean pocket client to the relay miner config it would look like this:

suppliers:
  - service_id: anvil
    signing_key_name: supplier1
    listen_url: http://localhost:9999
    service_config:
      backend_url: http://localhost:8081
      publicly_exposed_endpoints:
        - leanclienthost1

  - service_id: anvil
    signing_key_name: supplier2
    listen_url: http://localhost:9999
    service_config:
      backend_url: http://localhost:8081
      publicly_exposed_endpoints:
        - leanclienthost2

Difference between your snippet and that snippet are:

red-0ne commented 1 month ago

@okdas ,

We can use the Suppliers list from the Session but that doesn't exclude the possibility that supplier1 and supplier2 could be in that set, since the Session's Suppliers set construction is not aware of the suppliers that are being collocated. And this possibility grows with the number of Suppliers collocated.

Having distinct publicly_exposed_endpoints could also be a solution but that would involve:

  1. Have a protocol change that makes sure that, at staking time hostnames are unique across all Suppliers. The on-chain logic does not know about the ones that will be collocated so it cannot allow for some and deny for others without adding complexity.
  2. In large setups, RelayMiner operators have to setup as many hostnames as there are Suppliers they want to collocate.
Olshansk commented 1 month ago

Discussed offline. We'll add Supplier address to RelayRequestMetadata which should enable what we need. The important thing to note is that it MUST be hydrated by the Application before the relay is sent.

Have a protocol change that makes sure that, at staking time hostnames are unique across all Suppliers. The on-chain logic does not know about the ones that will be collocated so it cannot allow for some and deny for others without adding complexity.

I'd rather avoid this but can see it become a necessity due to other reasons. For now, let's just table it in our minds and it'll come up again if necessary.

Olshansk commented 1 month ago

See #532 for documentation when we pick this up.

Screenshot 2024-05-22 at 3 48 59 PM
okdas commented 2 weeks ago

567 has been merged in