CHIP-22 -- Enhanced Harvester Protocol

[danieljperry]

This CHIP provides a method for proprietary harvesters to communicate with the reference farmer. Its status is Review (Fast Track). Please leave your reviews here. We will set up a public meeting to discuss this CHIP soon.

[SlowestTimelord]

Instead of enforcing dev fees via a threshold, has the use of a split payment address been considered? Such as the royalty split puzzle used by splitxch.com.

It wouldn't be exactly the same puzzle as you'd want a way to curry in the farmer reward address in addition to the hard coded dev fee address. Also it would need a mechanism to poke (spend) the coin.

[danieljperry]

Instead of enforcing dev fees via a threshold, has the use of a split payment address been considered? Such as the royalty split puzzle used by splitxch.com.

It wouldn't be exactly the same puzzle as you'd want a way to curry in the farmer reward address in addition to the hard coded dev fee address. Also it would need a mechanism to poke (spend) the coin.

The idea so far is to keep the implementation as simple as possible, hence the threshold. However, the split payment puzzle is something that could be discussed when we set up a public call for this CHIP.

[SlowestTimelord]

With respect to the CHIP itself, can the reference farmer support multiple harvesters at the same time? For example for farmers with a mix of uncompressed, BB, and GH plots, can the farmer use a different harvester to use depending on the plot type?

If so, maybe that could allow @madMAx43v3r to create separate flat fee harvesters (CPU harvester for lower compression levels, GPU harvester for higher compression levels), while uncompressed plots can continue using the reference harvester with no fees.

[harold-b]

With respect to the CHIP itself, can the reference farmer support multiple harvesters at the same time? For example for farmers with a mix of uncompressed, BB, and GH plots, can the farmer use a different harvester to use depending on the plot type?

If I am understanding your question correctly, there shouldn't be any problem using multiple harvesters. With the current "fee convention", as it currently stands, there isn't any obstacle for harvesters using either a flat or dynamic fee, so everything should work out of the box as long as the harvester is using the protocol enhancements.

There might be some extra work to do for multiple harvesters on the UI side (as to reporting fees), but that's out of the scope of this particular CHIP.

[MakiMaki2023]

Dear CNI, would it make sense to restructure block reward to pay the farmer, pool and dev with a hard fork? Doing so would simplify the integration of dev fee from 3 parties (NoSSD and GH). Like how you join a pool through nft, but you also join the dev pool (just a proxy to dev wallet)

[danieljperry]

We are going to have a public zoom call to Discuss this CHIP on Nov 29 at 3 PM PST (11 PM UTC, Nov 30 at 7 AM in China). See our Discord for more details.

[danieljperry]

Dear CNI, would it make sense to restructure block reward to pay the farmer, pool and dev with a hard fork?

It sounds technically feasible by adding a developer singleton ID to the plot files and restructuring the pool coin. However, there are some significant downsides to this approach:

• It would require a hard fork, which would necessitate a long lead time. To use an existing example, the CHIP with the filter reduction hard fork took about 8 months to go from Draft to Final, and it will activate around 300 additional days after being added to a release. A similar approach would be required for this change.
• It would require a replot for those farmers and pools that wanted to use this approach.
• Restructuring the payout schedule could be controversial in the community, and the new schedule wouldn't be optional. Thus, this CHIP would require a very high level of approval.

CHIP-22 takes a simpler approach which does not require replotting or a hard/soft fork. It could be adopted much quicker, and those who only use open source farmers/harvesters would remain unaffected. Our goal here is to keep things as simple as possible.

All of this being said, if you want to restructure the rewards payout, you will need to open a new CHIP and try to build support for it. Meanwhile, we will continue to develop CHIP-22, and if your CHIP later gains broad adoption, the possibility would exist for us to sunset CHIP-22 in favor of your CHIP.

[MakiMaki2023]

Dear CNI, would it make sense to restructure block reward to pay the farmer, pool and dev with a hard fork?

It sounds technically feasible by adding a developer singleton ID to the plot files and restructuring the pool coin. However, there are some significant downsides to this approach:

* It would require a hard fork, which would necessitate a long lead time. To use an existing example, the CHIP with the filter reduction hard fork took about 8 months to go from `Draft` to `Final`, and it will activate around 300 additional days after being added to a release. A similar approach would be required for this change.

* It would require a replot for those farmers and pools that wanted to use this approach.

* Restructuring the payout schedule could be controversial in the community, and the new schedule wouldn't be optional. Thus, this CHIP would require a very high level of approval.

CHIP-22 takes a simpler approach which does not require replotting or a hard/soft fork. It could be adopted much quicker, and those who only use open source farmers/harvesters would remain unaffected. Our goal here is to keep things as simple as possible.

All of this being said, if you want to restructure the rewards payout, you will need to open a new CHIP and try to build support for it. Meanwhile, we will continue to develop CHIP-22, and if your CHIP later gains broad adoption, the possibility would exist for us to sunset CHIP-22 in favor of your CHIP.

Thank you very much for the quick feedback Daniel-san. It sounds like enhancing the existing plots with singleton ID is not a simple task that a replot is required.

[CryptoSlug]

I am opposed to this CHIP. Reason: The only benefit is currently for non official pool protocol users so they can see if they are having their fee handled properly by the non-official pool operator.

If this means a farmer like nossd who only appears as one farmer but with 1660 harvesters on the network will now become 1660 farmers and is able to direct a farmer to do things, anything such as sending one mojo to thousands, or is somehow able to manipulate timing of events either with a really fast timelord or with lots of timelords - I don't see how this risk is outweighed by the benefits to the few who are pooling with a non-official pool protocol operator.

[SlowestTimelord]

I am opposed to this CHIP. Reason: The only benefit is currently for non official pool protocol users so they can see if they are having their fee handled properly by the non-official pool operator.

If this means a farmer like nossd who only appears as one farmer but with 1660 harvesters on the network will now become 1660 farmers and is able to direct a farmer to do things, anything such as sending one mojo to thousands, or is somehow able to manipulate timing of events either with a really fast timelord or with lots of timelords - I don't see how this risk is outweighed by the benefits to the few who are pooling with a non-official pool protocol operator.

The benefit for this CHIP is to ensure individual farmers are the ones deciding what transactions get included in each block (using the open source Chia farmer client). The CHIP removes control from centralized pools/software like NoSSD by having them become only responsible for the harvesting functionality.

[danieljperry]

I am opposed to this CHIP. Reason: The only benefit is currently for non official pool protocol users so they can see if they are having their fee handled properly by the non-official pool operator.

If this means a farmer like nossd who only appears as one farmer but with 1660 harvesters on the network will now become 1660 farmers and is able to direct a farmer to do things, anything such as sending one mojo to thousands, or is somehow able to manipulate timing of events either with a really fast timelord or with lots of timelords - I don't see how this risk is outweighed by the benefits to the few who are pooling with a non-official pool protocol operator.

I think you are alluding to a pool building a botnet and using it to attack the network. Anyone running closed-source software from an anonymous source runs the risk of their machine(s) being compromised, regardless of whether this CHIP is in place. In addition, this CHIP doesn't enable any timelord-based attacks that weren't already present in the network.

[CryptoSlug]

The benefit for this CHIP is to ensure individual farmers are the ones deciding what transactions get included in each block (using the open source Chia farmer client). The CHIP removes control from centralized pools/software like NoSSD by having them become only responsible for the harvesting functionality.

Does this mean only the Chia software can manage farm operations or the proprietary software still can be used alongside or instead of? Will it be a choice for farmers? Using nossd for example: Chia farmer decides he wants to be part of nossd. Replots to nossd plot format using nossd software. Uses official Chia client to farm by using nossd pool nft, chia client sees the nossd plot format and includes in plot listing for farm and harvesting occurs as normal - pool manages rewards and fee deduction. Existing nossd software no longer has same interaction anymore? or does nossd software still manage harvesting and Chia software doesn't see plots but does see wallet harvesting traffic and Pool Overview status page?

[danieljperry]

Here is the recording of the call where we discuss this CHIP: https://www.youtube.com/watch?v=ul4Coa99aOE

[SlowestTimelord]

Having listened to the call recording, I will express that I have no objections to the CHIP.

As I see it, the CHIP itself is an innocuous update to the harvester protocol to allow additional information to shared between the harvester and farmer, allowing for reward redirection and logging of that data and activity. Any monitoring of a third-party harvester's behavior based on that logged data remains a responsibility for the individual farmer or as a future GUI feature.

It is important to note that implementation of this CHIP does not:

• set any expectations that third parties will integrate with the new protocol
• impact the ability for existing farmers to continue farming with their existing setup

[CryptoSlug]

Will this CHIP mean users who didn't want to have to replot to join a pool that currently has non OP plots can now do so with their existing plots and also plot any new plots with the proprietary farmer to take advantage of the tech offered by the particular pool?

[dddroptables]

Instead of enforcing dev fees via a threshold, has the use of a split payment address been considered? Such as the royalty split puzzle used by splitxch.com. It wouldn't be exactly the same puzzle as you'd want a way to curry in the farmer reward address in addition to the hard coded dev fee address. Also it would need a mechanism to poke (spend) the coin.

The idea so far is to keep the implementation as simple as possible, hence the threshold. However, the split payment puzzle is something that could be discussed when we set up a public call for this CHIP.

This actually seems like the simplest solution IMO. My reinterpretation of @SlowestTimelord 's idea is to create a singleton puzzle similar to the pooling singleton, and use that in the farmer reward portion in the plot. This singleton would need to be constructed differently than a pooling plotnft, but would allow for a hardcoded reward address plus a fixed fee (both provided by plot creator at plot time), and a dynamically changeable address for the main farmer rewards. This design would allow for the entire farming/harvesting flow of software to be open source (after plot is created, since presumably plot creation software may be proprietary).

Positives of this approach

• No forking needed for the change
• Entire farming/harvesting flow can be open source / built into the reference farmer
• Dynamic ability for farmers to change their rewards address directly on the blockchain via an NFT (also alleviating a lot of confusion about farmer reward address configs)
• Farmers can be assured fees cannot change after plots are created as they will be baked into the singleton
• Plot tool owners can be assured that fees will be applied for all block rewards fairly and users won't be able to reverse engineer closed source software to remove the fees when harvesting

Downsides of this approach

• Needs to be built into chialisp as a singleton/NFT, with the same due diligence as pooling plotnft design
• There would need to be some reward claiming functionality to claim rewards from the new singleton nft, similar to how pooling plotnft claiming works (meaning some changes to farming UX and user reeducation)

[xearl4]

@dddroptables Major downside of using a smart coin to manage dev fee distribution is that it would require all who want to use this solution to replot. The solution as currently proposed in CHIP-22 won't force a replot on Gigahorse users.

Another downside of the particular approach you mention is precisely that the fees are baked into the singleton. That would make it impossible for harvester vendors to change dev fees later. With the solution as currently proposed by CHIP-22, a change in fee structure is simply a matter of releasing a new harvester. So if, for example, a harvester vendor wants to reduce their fees to counter competition, that's easily done.

[dddroptables]

@dddroptables Major downside of using a smart coin to manage dev fee distribution is that it would require all who want to use this solution to replot. The solution as currently proposed in CHIP-22 won't force a replot on Gigahorse users.

Another downside of the particular approach you mention is precisely that the fees are baked into the singleton. That would make it impossible for harvester vendors to change dev fees later. With the solution as currently proposed by CHIP-22, a change in fee structure is simply a matter of releasing a new harvester. So if, for example, a harvester vendor wants to reduce their fees to counter competition, that's easily done.

@xchdata1 thanks for the feedback. While you are correct that farmers with proprietary/non conforming plots would need to replot, this would be the case for at least all of NoSSD pool users AFAIK since their plots are locked into the NoSSD pool permanently. At present they represent at least the second largest pool on the network so substantial replotting should be in consideration no matter the proposed change.

With regards to the proposal of using a smart coin to calculate the fees - I proposed this be a fixed fee when creating the singleton, but there is no reason why a smart coin couldn't allow fee changes. It would just be a more complex singleton puzzle which would require something like asymmetric multi signatures (one to change farmer puzzle hash by farmer, one to change vendor fee/vendor puzzle hash by vendor).

I like the idea of the smart coin structure much more than the client calculating the fees as well since they would be much more consistent for both the vendor and the farmer. Meaning, for example, 1% could be taken out of every block win to the vendor rather than having to randomly take 100% of block win X% of the time on the client side calculation.

I also happen to believe that any proposed change that does not resolve the problem of needing to run a closed source client to farm isn't solving the biggest risk with the current proprietary farmers/pools. Ideally there need not be any closed source clients required to run when farming if the protocol is well designed.

[hoffmang9]

@dddroptables Major downside of using a smart coin to manage dev fee distribution is that it would require all who want to use this solution to replot. The solution as currently proposed in CHIP-22 won't force a replot on Gigahorse users.

Another downside of the particular approach you mention is precisely that the fees are baked into the singleton. That would make it impossible for harvester vendors to change dev fees later. With the solution as currently proposed by CHIP-22, a change in fee structure is simply a matter of releasing a new harvester. So if, for example, a harvester vendor wants to reduce their fees to counter competition, that's easily done.

@xchdata1 thanks for the feedback. While you are correct that farmers with proprietary/non conforming plots would need to replot, this would be the case for at least all of NoSSD pool users AFAIK since their plots are locked into the NoSSD pool permanently. At present they represent at least the second largest pool on the network so substantial replotting should be in consideration no matter the proposed change.

With regards to the proposal of using a smart coin to calculate the fees - I proposed this be a fixed fee when creating the singleton, but there is no reason why a smart coin couldn't allow fee changes. It would just be a more complex singleton puzzle which would require something like asymmetric multi signatures (one to change farmer puzzle hash by farmer, one to change vendor fee/vendor puzzle hash by vendor).

I like the idea of the smart coin structure much more than the client calculating the fees as well since they would be much more consistent for both the vendor and the farmer. Meaning, for example, 1% could be taken out of every block win to the vendor rather than having to randomly take 100% of block win X% of the time on the client side calculation.

I also happen to believe that any proposed change that does not resolve the problem of needing to run a closed source client to farm isn't solving the biggest risk with the current proprietary farmers/pools. Ideally there need not be any closed source clients required to run when farming if the protocol is well designed.

If you have a proposed Chialisp smart contract that implements your vision you should request a CHIP for it and give an in depth outline - including a proposed initial implementation. However if you're merely speculating that such a thing could be developed, CHIPs are not where you ask someone else to do work for you.

[dddroptables]

@dddroptables Major downside of using a smart coin to manage dev fee distribution is that it would require all who want to use this solution to replot. The solution as currently proposed in CHIP-22 won't force a replot on Gigahorse users.

Another downside of the particular approach you mention is precisely that the fees are baked into the singleton. That would make it impossible for harvester vendors to change dev fees later. With the solution as currently proposed by CHIP-22, a change in fee structure is simply a matter of releasing a new harvester. So if, for example, a harvester vendor wants to reduce their fees to counter competition, that's easily done.

@xchdata1 thanks for the feedback. While you are correct that farmers with proprietary/non conforming plots would need to replot, this would be the case for at least all of NoSSD pool users AFAIK since their plots are locked into the NoSSD pool permanently. At present they represent at least the second largest pool on the network so substantial replotting should be in consideration no matter the proposed change. With regards to the proposal of using a smart coin to calculate the fees - I proposed this be a fixed fee when creating the singleton, but there is no reason why a smart coin couldn't allow fee changes. It would just be a more complex singleton puzzle which would require something like asymmetric multi signatures (one to change farmer puzzle hash by farmer, one to change vendor fee/vendor puzzle hash by vendor). I like the idea of the smart coin structure much more than the client calculating the fees as well since they would be much more consistent for both the vendor and the farmer. Meaning, for example, 1% could be taken out of every block win to the vendor rather than having to randomly take 100% of block win X% of the time on the client side calculation. I also happen to believe that any proposed change that does not resolve the problem of needing to run a closed source client to farm isn't solving the biggest risk with the current proprietary farmers/pools. Ideally there need not be any closed source clients required to run when farming if the protocol is well designed.

If you have a proposed Chialisp smart contract that implements your vision you should request a CHIP for it and give an in depth outline - including a proposed initial implementation. However if you're merely speculating that such a thing could be developed, CHIPs are not where you ask someone else to do work for you.

Thanks for the feedback @hoffmang9. My rationale for providing feedback on this CHIP is to optimistically help to highlight some of the main deficiencies with this proposal. I also noticed that this CHIP was authored by CNI and therefore likely to be implemented by CNI. My hope was that shedding some light on a potentially more effective solution could lead to CNI revisiting this plan before implementing- maybe even redirecting the planned implementation effort on this one to something more effective.

[danieljperry]

@dddroptables Major downside of using a smart coin to manage dev fee distribution is that it would require all who want to use this solution to replot. The solution as currently proposed in CHIP-22 won't force a replot on Gigahorse users.

Another downside of the particular approach you mention is precisely that the fees are baked into the singleton. That would make it impossible for harvester vendors to change dev fees later. With the solution as currently proposed by CHIP-22, a change in fee structure is simply a matter of releasing a new harvester. So if, for example, a harvester vendor wants to reduce their fees to counter competition, that's easily done.

@xchdata1 thanks for the feedback. While you are correct that farmers with proprietary/non conforming plots would need to replot, this would be the case for at least all of NoSSD pool users AFAIK since their plots are locked into the NoSSD pool permanently. At present they represent at least the second largest pool on the network so substantial replotting should be in consideration no matter the proposed change. With regards to the proposal of using a smart coin to calculate the fees - I proposed this be a fixed fee when creating the singleton, but there is no reason why a smart coin couldn't allow fee changes. It would just be a more complex singleton puzzle which would require something like asymmetric multi signatures (one to change farmer puzzle hash by farmer, one to change vendor fee/vendor puzzle hash by vendor). I like the idea of the smart coin structure much more than the client calculating the fees as well since they would be much more consistent for both the vendor and the farmer. Meaning, for example, 1% could be taken out of every block win to the vendor rather than having to randomly take 100% of block win X% of the time on the client side calculation. I also happen to believe that any proposed change that does not resolve the problem of needing to run a closed source client to farm isn't solving the biggest risk with the current proprietary farmers/pools. Ideally there need not be any closed source clients required to run when farming if the protocol is well designed.

If you have a proposed Chialisp smart contract that implements your vision you should request a CHIP for it and give an in depth outline - including a proposed initial implementation. However if you're merely speculating that such a thing could be developed, CHIPs are not where you ask someone else to do work for you.

Thanks for the feedback @hoffmang9. My rationale for providing feedback on this CHIP is to optimistically help to highlight some of the main deficiencies with this proposal. I also noticed that this CHIP was authored by CNI and therefore likely to be implemented by CNI. My hope was that shedding some light on a potentially more effective solution could lead to CNI revisiting this plan before implementing- maybe even redirecting the planned implementation effort on this one to something more effective.

For now we want to implement the simplest solution that allows proprietary harvesters to work with the reference farmer. We could also revisit your solution in the future. If you (or anyone else in the community) want to work on this, we'll be happy to look at your CHIP.

[danieljperry]

We're approaching the final testing phase this CHIP. Get your final reviews in soon. Next step is Last Call.

[danieljperry]

We are adding some minor updates in PR #17435. This is the final update we are planning to make before moving the CHIP to Last Call. Please review at your convenience.

[madMAx43v3r]

Should probably clarify that the challenge in (sha256(proof of space | challenge) mod 2^32) is the value from ProofOfSpace data structure, ie. the plot challenge, the same which is used to lookup proofs.

There are a few other challenges as well, which can be confusing.

[danieljperry]

Should probably clarify that the challenge in (sha256(proof of space | challenge) mod 2^32) is the value from ProofOfSpace data structure, ie. the plot challenge, the same which is used to lookup proofs.

There are a few other challenges as well, which can be confusing.

Added, thanks!

[danieljperry]

I added one more PR to this CHIP: 17481. Special thanks to @DaOneLuna for your contribution!

[danieljperry]

The code for this CHIP has been included in 2.2.0-rc3. All stakeholders: please test and include your final reviews here. https://github.com/Chia-Network/chia-blockchain/releases/tag/2.2.0-rc3

[madMAx43v3r]

I just farmed a block using CHIP-22 with 2.2.0-rc3: https://spacefarmers.io/farmers/4ecd5c26fb31adb4dd201530aa1de0f8f16dabb024d2d0b0ee1066008b4f88fd/blocks

Using GH fast farmer (not public yet), just to tell you it works with 2.2.0-rc3.

EDIT: It's a closed source farmer + harvester, it connects directly to the node. Official farmer is not used.

[harold-b]

Very nice, Max!

I just farmed a block using CHIP-22 with 2.2.0-rc3: https://spacefarmers.io/farmers/4ecd5c26fb31adb4dd201530aa1de0f8f16dabb024d2d0b0ee1066008b4f88fd/blocks

[danieljperry]

As a working version of the code for this CHIP is complete in version 2.2.1, I'm moving it to Last Call. If no changes are required in the next two weeks, it will become Final.

[danieljperry]

This CHIP is now Final. No further changes are allowed, other than adding errata. Thanks to all who participated!