Please re-enable getblocktemplate

[DrHaribo]

I use getblocktemplate to implement merged mining. I don't understand why it has been disabled. Please re-enable it again.

[p3yot3]

Agreed. I haven't been able to merge mine NMC for months. Each coin seems to have gone off on a tangent, implementing their own version of merge mining, (often at the request of FUpool) which has resulted in them not working at all or only on certain pools (see above). HUC done the same, only to be dumped by FUpool. SYS also - probably because it is based off NMC.

Why are coins deviating away from the standard implmentation? It makes no sense.

[JeremyRand]

@domob1812 would be the right person to answer this, but my understanding from talking to him and Luke-Jr is that GBT isn't applicable to merged mining.

The reason why the "standard implementation" is no longer being used is that merged mining as designed by Vince doesn't work at all with BIP 9, and has a lot of shortcomings in other areas. We're actively engaging with other groups (including Forrest of P2Pool, Patrick of Dogecoin, and Luke-Jr of Blockstream) to try to improve things. Unfortunately, most of the improvements we want to make are complex, require significant testing and review, and simply aren't feasible without a hardfork. Because of the urgency of fixing BIP 9 compatibility, most of that is on hold until we can do a low-complexity, high-urgency hardfork that does very little other than fixing BIP 9.

That said -- what other projects use merged mining at the moment whom we should be engaging with? I'm aware of us, Huntercoin, Dogecoin, P2Pool's sharechain, and Blockstream's sidechains. I know there are lots of others, but I don't know how many of them are actually actively developed. I'm sure there are a number of them -- so feel free to point me to some other projects that we should be engaging with, and I'll look into reaching out to them.

Also, can you explain what exactly you're using GBT for? I was unaware of its usage for merged mining, but I admit that's not an area of the code that I really touch at all.

Cheers.

[p3yot3]

Why implement BIP 9 if it breaks merge mining for everyone? What ever "shortcomings" it had, it must surely be better than no merge mining at all?

[JeremyRand]

Umm... maybe because scrapping BIP 9 means we can't merge a lot of highly useful upstream Bitcoin Core changes without massive reengineering on our part that none of us wants to do? And maybe because even if we scrap BIP 9, merged mining will still break completely when Bitcoin starts using some of the higher BIP 9 bits? And maybe because fixing merged mining to work with BIP 9 (both on Bitcoin's and Namecoin's) is totally doable with a hardfork that's not very complex?

I'm sorry, but I'm pretty sure that the amount of work involved in us trying to backport all of the Bitcoin Core changes from now until eternity to work without BIP 9, as well as the amount of work involved in trying to convince Bitcoin to eternally reserve a bunch of bits that break things for us (I've already talked to them; even getting them to reserve a subset of those bits just until we can deploy a hardfork is a stretch), is a hell of a lot more work than whatever it takes to make your mining code work with the post-hardfork Namecoin code.

We're happy to work with you on helping you fix your setup. You'll notice that we made fixes at F2Pool's request to facilitate their infrastructure -- we're happy to do that for any miner who asks, as long as the request is technically sane and it doesn't become a time sink for us. But can you please try to keep some perspective? The BIP 9 authors, F2Pool, and the Namecoin developers are not in a conspiracy against you. Bashing BIP 9, F2Pool, and us (while declining to answer any of the specific technical questions I asked) is not going to get your mining setup fixed. All it accomplishes is annoying us and making you look petty.

I have actual useful things to be doing this week (I don't consider stating the obvious about BIP 9 and F2Pool to be useful work), so I may or may not be responding further to this thread until early next week.

[p3yot3]

Please, lose the attitude.

I simply asked why, because I did not know. It was a simple question, & not deserving of such a long moany answer.

Please continue to suck up to FUpools every request & screw every other mining pool. Rebrand it to F2coin & be done with it. When they dissappear or dump you, which they will sooner or later, you'll have nobody to take up the slack.

Great work.

[JeremyRand]

@DrHaribo can you elaborate on what your mining setup is using GBT for? Sincere apologies that your thread got hijacked -- we'd love to help get things working for you. I think @domob1812 will probably see your thread this weekend, apologies for the slow reply.

Best,

-Jeremy

[DrHaribo]

I only use getauxblock once to get the chainid. After that I use getblocktemplate to get the data I need to construct namecoin blocks and I use submitblock when I have a valid block.

In the namecoin block I construct I take the version value from namecoind's getblocktemplate and bitwise-or it with (1 << 8) to indicate an auxiliary proof-of-work (merged mining).

Basically I do what getauxblock does, with my own code. This is superior to getauxblock because it gives me control of the contents of the blocks I am creating.

Disabling getblocktemplate obviously breaks the way I do merged mining, and it seems to me there is no reason to disable it. In fact it reduces the control a pool has over the blocks it creates.

Is there a mailing list, forum, or similar, to follow about the version field and possible hardfork?

[JeremyRand]

@DrHaribo thank you for the thorough explanation. Just to verify that I understand, control of the contents == control over which transactions are included? Or something else?

​I think it would be very useful to set up a mailing list or forum or something for all the stakeholders in merged mining. That would include all the cryptocurrencies that use MM, all the mining pools that do MM, and all the groups working on MM specs (in particular Luke-Jr / Blockstream, ForrestV / P2Pool, and us, but maybe there are others). It is definitely unfortunate that there's a lack of communication between the various people who use MM, and the sooner this is fixed, the better off everyone will be. I think Daniel has a lot of contacts at some of the other SHA256D MM coins, and I have some contacts at Dogecoin, Blockstream, and Eligius. I think Daniel and I are both able to reach F2Pool. Not certain about P2Pool and the other mining pools, and whatever other projects might exist. I'd be happy to inquire about the prospect of setting up an MM meeting space of some kind (whether a mailing list or forum or something else).

In the meantime, there's an open GitHub pull request to Namecoin Core; I don't have access to the link right now but I think it's the only issue in the nc0.12.1 milestone. That code isn't fully tested nor really even specced out, so almost anything is up for revision if needed. We won't be focusing on any hardforking plans until some other blocking issues are fixed (see the nc0.12.0 milestone), which we've made some progress on lately, but I don't have an ETA on that. I hope those blockers will be fixed within a month or two. Obviously feedback/planning/testing of hardfork code before those blockers is totally fine, it's just not really my focus nor many other people's right now (which is a major reason why I haven't spent more time reaching out to people already).

Hope this helps -- and sorry again for the lack of communication.

[domob1812]

My understanding of GBT is that it should allow pools to work where the individual miners are able to choose the transactions they want included. I don't see how that's applicable for merge-mining, where the PoW is "outsourced" by definition.

The explanation of your usecase sounds to me like a slight abuse - ideally, you shouldn't have to worry about the block version, for instance. That's what the core client is for; when we are going forward with the planned hardfork, your code would break anyway - and it is not the intent to have every miner fix their own setup; instead, you should rely on the core client to do that for you.

For choosing which transactions to include, why can't you just patch the core client's selection logic accordingly? It seems to me that this should be the correct way to do that.

[JeremyRand]

So I'm just curious, are most of the Bitcoin miners implementing selection logic and softfork voting in external code via GBT, or are they mostly patching Bitcoin Core. I honestly don't know how the relevant infrastructure is handled in the real world, other than what I've heard from Eligius and F2Pool (which, of course, are a small fraction of miners).

My take is that we're best off supporting whatever workflow is common among Bitcoin miners by default, except when it doesn't make sense for MM (e.g. using GBT for decentralized pool control, which isn't compatible with Namecoin's MM, but I believe Luke's MM spec is intended to support such use cases). This is, of course, a difficult task to pull off when we don't know what most miners are doing. So hopefully we can work out a system that fits all the common use cases (and hopefully as many uncommon use cases as possible) while not making things too complex on our end.

My understanding is that GBT is difficult to modify to handle the proposed hardfork's validation rules. Daniel, am I correct in inferring that?

I'm going to be dealing with GETD#4 in Berlin via video conference for the next 2 days (I get about 4.5 hours of sleep first), so I won't be talking much in this thread until Sunday or Monday.

Cheers.

[DrHaribo]

@domob1812: I'm not talking about GBT between miners and pool - very few pools do that. I'm talking about the pool using GBT to communicate with namecoind. The individual miners use stratum and cannot choose which transactions to include. I don't understand why getblocktemplate is abuse or why it needs to be disabled.

@JeremyRand: yes, by content I mean transactions, including the coinbase.

[JeremyRand]

@domob1812 While I'm not an expert on this, what @DrHaribo is doing doesn't seem like an abuse to me. What he's doing allows him to write transaction selection and coinbase construction logic in arbitrary languages, whereas patching Namecoin Core to do this would, presumably, restrict him to using (non-memory-safe) C++. For that reason alone, I think it's a valid use case. (Am I mistaken on this?)

In addition, Bitcoin Core supports GBT for that use case, and since that use case isn't made inapplicable by merged mining, I don't see any reason to deviate from Bitcoin Core's behavior on this.

I'm unaware of what impact, if any, AAA has on GBT. @domob1812 can you provide some info on what the impact is, so that we can figure out how to fix things for @DrHaribo ?

Cheers.

[domob1812]

I disabled GBT for two reasons (in combination): First, because I didn't really see the point, as explained above - but if that's something beneficial to miners, I'm fine to enable it again (I just didn't know people where doing that). Second, because it was not clear to me how that would work together with merge-mining - the pre-disabled GBT just built a non-merge-mined block, which will no longer be possible with AAA.

So we could re-enable GBT still building a non-merge-mined block, if the callers take responsibility to turn that into a proper merge-mined block themselves (like @DrHaribo seems to be doing already). If that's what we want and you don't see a risk of confusing anybody who's not aware of that and then finds GBT's blocks invalid without this modification, I can definitely re-enable GBT.

[JeremyRand]

@DrHaribo given what @domob1812 said above, does it make sense in your opinion to re-enable GBT? If so, then I concept-ACK re-enabling GBT.

Cheers.

[DrHaribo]

Yeah, I would be happy to get GBT back.

What is AAA?

[JeremyRand]

What is AAA?

AAA = AlwaysAuxpowActive, the name of the planned hardfork to fix BIP 9 compatibility.

[DrHaribo]

Ok, so because of version bits the auxpow bit will be assumed and after the hardfork I have to stop setting it like I do now. That should make using getblocktemplate even cleaner, in my opinion.

Hopefully the hardfork will be announced on the alert mailing list and other relevant places.

[JeremyRand]

Geir Harald Hansen:

Ok, so because of version bits the auxpow bit will be assumed and after the hardfork I have to stop setting it like I do now. That should make using getblocktemplate even cleaner, in my opinion.

Sounds good to me, I concept-ACK re-adding GBT. Last I heard @domob1812 does most of his Namecoin Core development on weekends (except in urgent cases), so I expect that nothing will happen on this front until this weekend at the earliest.

Hopefully the hardfork will be announced on the alert mailing list and other relevant places.

Yes, it will definitely be announced on the alerts list, and as many other available channels as feasible, with as much advance notice as we can. At the moment we're focusing on some prerequisites, but we appear to quickly be reaching the point where the prerequisites are satisfied, so that we can focus on the AAA hardfork. Thanks for your patience.

[JeremyRand]

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512

@domob1812 is there any status update on this? (Not sure if this issue is blocked by something else at the moment?) -----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJYQh8cAAoJELPy0WV4bWVwTYIQAMeDdDxuGx6dKC4ntpNO3uDb R6m7yuWVSuvvrJGhcn1Ac9MKmC1FtYFWVX7UGNwhb0uIds2q3buYhnaOkrkNoXG+ 5zaiuqdvNlPTqd+LKO+aukLDZ+kzs6Ga6vaJ+oy6gorJPVceZQadfbP+Ml4DMq// X/71+tE7bZK7uh1HaG+uNJcawmjKKf5CmaHEMCOqpd7U6StGYQDJrGiWg0erDiEF H3VW1KkcUwC8dEOGd6rj5Ky/s4+bAKZBWykSBSFCHbPIw3Gjds9NnTQkJcfYHgSD GgY61UkFUsDUipA2mZJWofj9EnoJHkRXlLCU9SyYGzJJQXtXapqk8PfAgZiEyV7D qEfJ3D5Mj0hHs/IEf5TPJ9ibk3LnQbpENRlvg3J+qrqI5uGmw905KZscf95zbFbU BViipmLXCB3qc11NuktTZkoXFNZ9aCPvuh3CLk2K6Mh8vV6S7oqhW9Yre3c4IVIE Ry9ScAXOPuKgfN4ATqE/bwjFhM75Lbug269yiDOpknbGDZiSqwqtxY/kNy+tNRkM uANKJhshTBv0ao7wSAXuqYjoGNDstgU2GCd8zGpMeW8TTrXr7SN+0is7fnyEglN2 gguyvZHJgIc87ke5AizxsoAYqCgMOL3AgmUnM5UHv0pCk3cR1kKXwvbgZkIY/6zl vmpmK8caSq1GF02fE6CA =AlKV -----END PGP SIGNATURE-----

[domob1812]

Sorry, no status update so far. But I don't think this is blocked on anything - I just didn't have time to work on it so far initially and later forgot about this. I'll try to find some time soon, this is the next Namecoin-related thing for me to work on besides the regular upstream merging.

[domob1812]

PR created: https://github.com/namecoin/namecoin-core/pull/138

Please test that this works as expected and for your purposes. If it does, I'm happy to merge this and port to all the branches.

[DrHaribo]

Looks great. Exactly the same changes I have been making to use namecoind.

[domob1812]

Fixed on dev, master and 0.13.