Getting this error as of today

[ikolubr]

Error getting work from bitcoind:
Traceback (most recent call last):
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/root/p2poolSoloNew/p2pool/util/deferral.py", line 41, in f
    result = yield func(*args, **kwargs)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/root/p2poolSoloNew/p2pool/bitcoin/helper.py", line 102, in getwork
    unpacked = bitcoin_data.tx_type.unpack(packed)
  File "/root/p2poolSoloNew/p2pool/util/pack.py", line 55, in unpack
    obj = self._unpack(data, ignore_trailing)
  File "/root/p2poolSoloNew/p2pool/util/pack.py", line 42, in _unpack
    obj = self.read(data)
  File "/root/p2poolSoloNew/p2pool/bitcoin/data.py", line 178, in read
    locktime = self._int_type.read(file)
  File "/root/p2poolSoloNew/p2pool/util/pack.py", line 175, in read
    return struct.unpack(self.desc, data)[0]
struct.error: unpack str size too short for format```
litecoin on latest version, p2pool as well.

[kr1z1s]

Oh. Need to write an additional transaction handler for MWEB. I'll see what I can do.

[jtoomim]

AFAIK p2pool does not need the parsed MWEB transaction data for anything. P2pool needs to be able to parse transactions that are in block templates because p2pool does block assembly (it needs to be able to trim off transactions to avoid exceeding the blocksize limit after adding a big coinbase tx), but IIUC the MWEB is just a hash that p2pool commits.

Previous versions of p2pool (before #22) would propagate the transactions over the p2pool p2p layer in order to do fast block/share propagation between p2pool nodes, but that functionality was removed after Compact Blocks made it effectively obsolete.

So we could probably get away with changing the handler for a tx msg over p2p to do nothing at all. This might cause issues for starting a new share chain from scratch, but we could probably solve that by removing support for older share versions (i.e. < 34). This might also improve performance, CPU load, and memory consumption.

[kr1z1s]

File "/root/p2poolSoloNew/p2pool/bitcoin/helper.py", line 102, in getwork unpacked = bitcoin_data.tx_type.unpack(packed)

File "/root/p2poolSoloNew/p2pool/bitcoin/helper.py", line 102, in getwork unpacked = bitcoin_data.tx_type.unpack(packed)

This error occurs when parsing transactions after a call getblocktemplate

[jtoomim]

Hmm, does getblocktemplate also return the pending MWEB transactions?

[kr1z1s]

Hmm, does getblocktemplate also return the pending MWEB transactions?

I think so. This issue requires more careful study.

[kr1z1s]

I have been watching for a long time when this error appears. This error appears when changing block number in network. When getblocktemplate returns 0 transactions. When transactions appear in mempool, the error disappears.

Upd. The error also appears with a full mempool. Here is a commit with changes to transaction structure. https://github.com/litecoin-project/litecoin/commit/4df88cf4b7a8d6088f414055e4edbde9a7c78fbb

[ikolubr]

I'm not a programmer... but let me know if i can help somehow. Thank you,

[Azskygod]

Watching and waiting for updates on this issue and ready to assist with testing, logs, etc. Let me know if I can help. Thanks!

[Azskygod]

"3. If your pool software modifies, reorders, or reprioritizes transactions that it receives from getblocktemplate, it must be updated to ensure it never alters the final transaction, known as the HogEx or "Integrating Transaction." The pool also must be careful to not modify or remove any transaction the HogEx depends on (txs with outputs it spends).

The reason for (3) is that the HogEx commits to the MWEB data, and moves the pegged in coins to the HogAddr (address holding all MMEB coins), and sends the pegout coins to the addresses specified by the pegout transactions in the MWEB. Any mismatch in MWEB data and the HogEx transaction would result in the block not passing consensus checks."

https://github.com/litecoin-project/litecoin/blob/v0.21.2/doc/mweb/mining-changes.md

[kr1z1s]

"3. If your pool software modifies, reorders, or reprioritizes transactions that it receives from getblocktemplate, it must be updated to ensure it never alters the final transaction, known as the HogEx or "Integrating Transaction." The pool also must be careful to not modify or remove any transaction the HogEx depends on (txs with outputs it spends).

The reason for (3) is that the HogEx commits to the MWEB data, and moves the pegged in coins to the HogAddr (address holding all MMEB coins), and sends the pegout coins to the addresses specified by the pegout transactions in the MWEB. Any mismatch in MWEB data and the HogEx transaction would result in the block not passing consensus checks."

https://github.com/litecoin-project/litecoin/blob/v0.21.2/doc/mweb/mining-changes.md

I read it.

It is necessary to write and debug a new type of transaction handler. This is not a quick job. It requires a lot of free time. You can try it too. After all, you were able to write an addition to the pool. I mean the operation of the pool with Dogecoin for all your miners. The current transaction handler is located here - https://github.com/jtoomim/p2pool/blob/master/p2pool/bitcoin/data.py#L149

Unfortunately, I could not solve the issue quickly with the help of crutches.

[Azskygod]

@kr1z1s our P2Pool development experience consists of a custom GUI only with no back end modifications. This problem is beyond our limited scope of knowledge of MWEB. We can assist with a bug bounty only.

@northern-lites This is a continuation of issue #59. Can you assist?

@jtoomim Is raising the minimum relay TX fee to keep the number of transactions below the crash threshold a possible workaround until this is fixed?

Please advise. Thanks!

[mansilladev]

@kr1z1s:

I have been watching for a long time when this error appears. This error appears when changing block number in network. When getblocktemplate returns 0 transactions.

I've not seen the error occur when there are 0 tx in mempool (ex: right after block solve). Nor have I seen this when there's just 1 tx in mempool (coinbase transaction) which you can try yourself by running litecoind with --blocksonly. NOTE: getmempoolinfo may report size 0, but getblocktemplate will show single coinbase transaction in list.

The error also appears with a full mempool.

I also believed there was a correlation between either tx count or memory kB footprint of mempool, because it seemed to throw error in these scenarios; however, I've also seen it bail with only 108 tx @ 28 kB (where it was running fine with > 200 tx and > 200kB). Thus, this might be a red herring. (not relevant)

It is necessary to write and debug a new type of transaction handler. This is not a quick job. It requires a lot of free time.

I agree that this will require some deeper inspection.

@Azskygod:

Is raising the minimum relay TX fee to keep the number of transactions below the crash threshold a possible workaround until this is fixed?

This might help reduce occurrences of errors until actual fix is applied.

[mansilladev]

Observation -- when p2pool logs throw error struct.error: unpack str size too short for format it seems to be when litecoin debug.log reflects MWEB weight greater than 0 (on CreateNewBlock log entry).

[kr1z1s]

@kr1z1s our P2Pool development experience consists of a custom GUI only with no back end modifications. This problem is beyond our limited scope of knowledge of MWEB. We can assist with a bug bounty only.

@northern-lites This is a continuation of issue #59. Can you assist?

@jtoomim Is raising the minimum relay TX fee to keep the number of transactions below the crash threshold a possible workaround until this is fixed?

Please advise. Thanks!

Suggested below. Before fixing problem with transactions, you can run litecoind with param --blocksonly After that, any transactions including MWEB will not be transmitted via getblocktemplate Of course it's wrong to do that. But as a temporary solution, this can be done. I checked on my nodes. There are no errors.

I work for a MWEB handler. But the process is not fast.

[jtoomim]

My current thinking is that p2pool does not need to decode transactions at all, and should simply handle transactions as a dict with the raw binary representation of the tx (or possibly the hex, not sure), the transaction fee, and the size and weight. Rather than adding code to handle MWEB, we should remove the code that makes handling MWEB necessary.

[Azskygod]

@mansilladev HyperDonkey.com's East US P2Pool node successfully merge mined a LTC and DOGE block today with the "--blocksonly" option enabled on Litecoind as suggested. A full log dump is attached for review.

• https://hyperdonkey.com/p2pool/logs.zip
• https://chainz.cryptoid.info/ltc/block.dws?2274836.htm
• https://dogechain.info/tx/2bfbbaf2ac318e71236f58a2c226ae8e6520d18955374e6b49afee14d512a94f

@jtoomim This is a temporary workaround only. P2Pool is still unable to mine Litecoin MWEB blocks containing transactions. There seem to be conflicting ideas on how to handle P2Pool MWEB transactions. Can you clarify and provide some direction? David Burkett has also offered to help answer P2Pool related questions on the Litecoin MWEB Development Telegram here:

• https://t.me/MWEB_Testnet

@kr1z1s HyperDonkey.com is offering a 5,000 DOGE BUG BOUNTY for a tested and merged fix for this issue!

(Bug Bounty reward address: D7qVwccq5Bmq4hoCG2kJMGsktTWAL4TnTS)

P2Pool.zip Litecoind_debug.zip Dogecoind_debug.zip

[jtoomim]

Are you guys observing an actual crash in p2pool when this bug gets triggered, or is it just an error with a traceback which p2pool reports without fully crashing? My node (http://ml.toom.im:9327) has been running this code for weeks, and while I do frequently see this error, I haven't seen my node go down at all.

[jtoomim]

The only reason why p2pool might need to change the transactions in a block template is if the p2pool coinbase transaction is large enough that it would put the block over the e.g. 1 MB limit. A workaround for this might be to hardcode p2pool to not modify block templates on LTC and also configure litecoind to limit its block templates to e.g. 970 kB to allow for up to 30 kB for the coinbase transaction. (I've seen coinbase txs of around 10 kB before on the BTC p2pool, though the LTC p2pool usually has fewer users and payout addresses than that.)

[Azskygod]

We were experiencing random P2Pool crashes and miner disconnections prior to adding "--blocksonly" to the Litecoind daemon. P2Pool seems to run without errors when no Litecoin transactions are included. There was only one transaction in the last Litecoin block with "--blocksonly" enabled. Is that the MWEB HogEx transaction?

https://chainz.cryptoid.info/ltc/block.dws?2274836.htm

I received a request from a HyperDonkey.com merge miner to donate to the existing 5,000 DOGE P2Pool Bug Bounty. All Dogecoin sent to this address (D7qVwccq5Bmq4hoCG2kJMGsktTWAL4TnTS) will be awarded to the coder(s) of a tested and merged fix for this issue. (Update: The bug bounty is now 9,200 DOGE.)

Please let me know if I can help with testing, etc. I also have error logs before/after the "--blocksonly" change. Thanks!

[jtoomim]

@AdieLaine The version numbers are no longer relevant on this branch. If I had been updating version numbers (which I simply have not bothered to do), this repository would currently be P2Pool v35, not v16. The code on p2pool.in is obsolete, and has been for about 5 years.

Your other comments are also off the mark and aren't relevant to this thread.

[kr1z1s]

I appreciate your response. I was merely pointing out other examples that presented this issue. At the root, this is what was posted from MWEB Change from the Litecoin GitHub.

https://github.com/litecoin-project/litecoin/blob/v0.21.2/doc/mweb/mining-changes.md

Required Pool Software Changes

The "mweb" rule must be supplied to the getblocktemplate "rules" array to signal to the node that MWEB is supported. getblocktemplate '{"rules": ["segwit", "mweb"]}' In the response, there will be an "mweb" field for all blocks after MWEB activation. If that field exists, then before calling submitblock with your solved block, append a byte with value 1 followed by the data from the "mweb" field, to the very end of the serialized block. NOTE: This <0x01> is just an indicator that tells the node that MWEB data follows.

<0x01> If your pool software modifies, reorders, or reprioritizes transactions that it receives from getblocktemplate, it must be updated to ensure it never alters the final transaction, known as the HogEx or "Integrating Transaction." The pool also must be careful to not modify or remove any transaction the HogEx depends on (txs with outputs it spends). The reason for (3) is that the HogEx commits to the MWEB data, and moves the pegged in coins to the HogAddr (address holding all MMEB coins), and sends the pegout coins to the addresses specified by the pegout transactions in the MWEB. Any mismatch in MWEB data and the HogEx transaction would result in the block not passing consensus checks.

Your recent update for the P2Pool code had an update to helper.py on Apr 23, and referring to line 140: mweb='01' + work['mweb'] if 'mweb' in work else '',

The number should not start by 0 because it will be in an octal escape sequence. I noticed you append the use of 01 to the submitblock code within helper.py. , But the Group number should only contain ASCII decimal digits in range [0-9]. Initial 0 is not allowed except for group number 0. This source can be found:

python/cpython#91760

Since the guidance from litecoin calls for 1, the use of 01 may be injecting errors that cannot be resolved.

It was posted here of an example that could derive from the use of 01:

#59 (comment)

_

You took discussion aside. Current problem is NOT in the block template request. Current problem is NOT sending block to network.

Current problem is IN transaction parsing!

File "/root/p2poolSoloNew/p2pool/bitcoin/helper.py", line 102, in getwork unpacked = bitcoin_data.tx_type.unpack(packed)

When transactions are disabled, block is created normally. And it is normally accepted by coin network.

Changes here - https://github.com/litecoin-project/litecoin/blob/0.21/src/primitives/transaction.h With the flag set to 8, MWEB transactions must be processed - https://github.com/litecoin-project/litecoin/blob/0.21/src/primitives/transaction.h#L310

We process these transactions as segwit - https://github.com/jtoomim/p2pool/blob/master/p2pool/bitcoin/data.py#L117

Structure of mweb transactions is different - https://github.com/litecoin-project/litecoin/blob/0.21/src/mweb/mweb_transact.h

That's the whole problem.

And stop looking at the version. It is taken from "github" files. You can put any number.
On my nodes, the version is 77, although it was possible to make 69. =)

[jtoomim]

@AdieLaine You don't know what you're talking about and are wasting our time.

Your recent update for the P2Pool code had an update to helper.py on Apr 23, and referring to line 140: mweb='01' + work['mweb'] if 'mweb' in work else '',

The number should not start by 0 because it will be in an octal escape sequence. I noticed you append the use of 01 to the submitblock code within helper.py. , But the Group number should only contain ASCII decimal digits in range [0-9]. Initial 0 is not allowed except for group number 0. This source can be found:

In mweb='01', the '01' is a string, not a number. That's what the quote marks do. Python does not interpret it as octal, it simply passes that string along in the getblocktemplate request.

Everything else you have mentioned is off base as well, and I don't want to waste time answering all of it.

[jtoomim]

@AdieLaine We already know exactly what the problem is. The bottleneck here is that no programmers have taken the time to either (a) write a parser in p2pool for the MW transaction format, or (b) change the p2pool code to no longer try to parse transactions and to treat them as simple binary blobs.

Your guesses are unnecessary and wrong.

[Azskygod]

@kr1z1s are you working on a fix?

The bug bounty for this issue is up to 9,200 DOGE. I would like to promote it on social media to facilitate a resolution, but don't want to cause havoc if you're working on something already.

Please advise.

[jtoomim]

I'm working on something.

[jtoomim]

https://github.com/jtoomim/p2pool/tree/rawtx

https://github.com/jtoomim/p2pool/commit/dbaaba535806421202e68bfad86c490ea269870d

I successfully made a few testnet blocks with this code. However, I haven't tried it on testnet with any transactions besides the default coinbase and HogEx txs. It ought to be tested with non-SegWit, SegWit spends (i.e. txid != hash), and MWEB transactions before using it on mainnet, as it's possible that this commit might have a bug in the merkle root calculation code with some of those transaction types, and therefore create invalid blocks.

Also ought to be tested with blockchains other than LTC (e.g. BCH, BTC).

Added bonus: this commit appears to reduce memory consumption by about 50% (unparsed txs are more RAM-efficient than parsed ones), and probably is faster/uses less CPU too.

[mansilladev]

@jtoomim - on the rawtx branch testing on mainnet, I'm still seeing this error pop up just as soon as an mweb tx is popped into mempool:

2022-06-09 17:05:28.316615 New work! Share diff: 1341.36 Block value: 12.53 LTC (265 tx, 432 kB)
2022-06-09 17:05:28.628661 Skipping from block cc5bef2d6e6876665bb3e2248f70b338884c78577aa56a970d8fd35e8fdc51f8 to block 81ac8bd57699e35a6fe9621426045912c99318a22dd21238d98d0c9838da7288!
2022-06-09 17:05:29.332547 Got new merged mining work!
2022-06-09 17:05:29.639189 RECV tx 020000000000b4c12200
Error handling message: (see RECV line)
Traceback (most recent call last):
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 209, in doRead
    return self._dataReceived(data)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 215, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 27, in dataReceived
    self.dataReceived2(data)
  File "/home/neil/p2pool-rawtx/p2pool/util/datachunker.py", line 40, in _DataChunker
    wants = receiver.send(buf.get(wants))
--- <exception caught here> ---
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 57, in dataReceiver
    self.packetReceived(command, type_.unpack(payload, self.ignore_trailing_payload))
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 55, in unpack
    obj = self._unpack(data, ignore_trailing)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 42, in _unpack
    obj = self.read(data)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 280, in read
    item[key] = type_.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/bitcoin/data.py", line 173, in read
    next = self._wtx_type.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 280, in read
    item[key] = type_.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 157, in read
    res = [self.type.read(file) for i in xrange(length)]
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 280, in read
    item[key] = type_.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 294, in read
    value = self.inner.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 280, in read
    item[key] = type_.read(file)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 175, in read
    return struct.unpack(self.desc, data)[0]
struct.error: unpack str size too short for format

Now, it didn't seem to go as far as saying it couldn't connect to the litecoind server, and in fact, one of my workers still got a share accepted while MWEB weight was about 78 (two mw tx I popped in there). Thus, my workers didn't disconnect from the p2pool node.

So, is that unpack error message that's still present.. was that unexpected?

[jtoomim]

@mansilladev I changed the code in getblocktemplate to not parse transactions. I didn't change the code to not parse transactions received over *coin p2p, because (as of share version 34) p2pool no longer uses txs received over p2p for anything. But apparently the code still tries to parse them (though I didn't notice this in my own testing). That should be pretty easy to fix. In the mean time, this unhandled exception and error message probably doesn't do any harm.

[mansilladev]

@jtoomim Been running this branch for most of the day with some hashpower load against it, and it appears stable when MW txs come into mempool (other than the exception/error message). That being said, there I did notice a bug when trying to run it from scratch (no existing shares in ./data/litecoin/ directory). To be clear, when running this branch with a previously populated data directory with share data, it runs fine.. but when running from scratch, this pops up the moment p2pool makes a connection with the first node it encounters, and the same with subsequent nodes, thus never connecting to any node successfully:

2022-06-10 02:23:48.106925 Outgoing connection to peer 67.199.179.49:9326 established. p2pool version: 3501 '16.0-197-g19c30d5'
2022-06-10 02:23:48.115784 RECV version ad0d00000000000000000000000000000000000000000000000000000000ffffa2dbb003a0ca000000000000000000000000000000000000ffffc0a80316246ef472d884158893c11131362e302d3139372d673139633330643501000000479ca3336585...
Error handling message: (see RECV line)
Traceback (most recent call last):
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 209, in doRead
    return self._dataReceived(data)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 215, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 27, in dataReceived
    self.dataReceived2(data)
  File "/home/neil/p2pool-rawtx/p2pool/util/datachunker.py", line 40, in _DataChunker
    wants = receiver.send(buf.get(wants))
--- <exception caught here> ---
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 57, in dataReceiver
    self.packetReceived(command, type_.unpack(payload, self.ignore_trailing_payload))
  File "/home/neil/p2pool-rawtx/p2pool/p2p.py", line 91, in packetReceived
    p2protocol.Protocol.packetReceived(self, command, payload2)
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 71, in packetReceived
    handler(**payload2)
  File "/home/neil/p2pool-rawtx/p2pool/p2p.py", line 255, in handle_version
    fragment(self.send_remember_tx, tx_hashes=[], txs=self.node.mining_txs_var.value.values())
  File "/home/neil/p2pool-rawtx/p2pool/p2p.py", line 22, in fragment
    f(**kwargs)
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 102, in <lambda>
    return lambda **payload2: self.sendPacket(command, payload2)
  File "/home/neil/p2pool-rawtx/p2pool/util/p2protocol.py", line 91, in sendPacket
    payload = type_.pack(payload2)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 67, in pack
    return self._pack(obj)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 49, in _pack
    self.write(f, obj)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 286, in write
    type_.write(file, item[key])
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 164, in write
    self.type.write(file, subitem)
  File "/home/neil/p2pool-rawtx/p2pool/bitcoin/data.py", line 194, in write
    return tx_id_type.write(file, item)
  File "/home/neil/p2pool-rawtx/p2pool/util/pack.py", line 284, in write
    assert set(item.keys()) >= self.field_names
exceptions.AssertionError: 

2022-06-10 02:23:48.124436 Lost peer 67.199.179.49:9326 - Connection was aborted locally, using.
in handle_share_hashes:
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionAborted: Connection was aborted locally, using.

[Azskygod]

Congratulations and thank you @jtoomim!

Someone on the LTC P2Pool network mined block #2280057 today with transactions and MWEB successfully included:

• https://chainz.cryptoid.info/ltc/block.dws?1adee4e24e6281a3fa9602cda412a3a9f49fd32e405aa24dddca8907a3cc0f31

HyperDonkey.com will now update all 3 of its nodes to the rawtx branch and report any remaining issues here.

[jtoomim]

I pushed a couple of new commits to rawtx. These should fix the issues that @mansilladev reported, and also fix an issue with mining with this code on BCH caused by BCH bitcoind not giving the 'weight' attribute for transactions in getblocktemplate results. I mined a block on BCH with this new code this morning.

My initial hopes about reducing memory usage with this commit appear to be ill-founded. It now looks like this code increases memory usage. Grumble.

[Azskygod]

@jtoomim We successfully merge mined Litecoin and Dogecoin blocks today with the new commits to rawtx and no errors!

• https://chainz.cryptoid.info/ltc/tx.dws?118873187.htm
• https://dogechain.info/tx/010678af84097f19ea1dfaaf5ee0584a5f7fa05129ef78624b701e376967915b

Please reply with your Dogecoin address so I can disburse the 9,200 DOGE bug bounty. Thank you and @mansilladev for supporting P2Pool!

[jtoomim]

DDnEa13uArBr95ZFP8We4TYqLwezkV4tcu @Azskygod

[Azskygod]

@jtoomim Confirmed. Bug bounty of 9,199.99363364 DOGE sent. Thank you!

• https://dogechain.info/tx/398d9784c286647f96a204c85925f4391de92c0b487b2b32d97b2b1344da0005

[ikolubr]

Hi guys, I don't know if it is same error or related, but I've been seeing this error in the last couple days, a week or two:

Error getting work from bitcoind:
Traceback (most recent call last):
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/root/p2poolSoloMWEB/p2pool/util/deferral.py", line 41, in f
    result = yield func(*args, **kwargs)
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/root/p2poolSoloMWEB/p2pool/bitcoin/helper.py", line 102, in getwork
    unpacked = bitcoin_data.tx_type.unpack(packed)
  File "/root/p2poolSoloMWEB/p2pool/util/pack.py", line 55, in unpack
    obj = self._unpack(data, ignore_trailing)
  File "/root/p2poolSoloMWEB/p2pool/util/pack.py", line 42, in _unpack
    obj = self.read(data)
  File "/root/p2poolSoloMWEB/p2pool/bitcoin/data.py", line 178, in read
    locktime = self._int_type.read(file)
  File "/root/p2poolSoloMWEB/p2pool/util/pack.py", line 175, in read
    return struct.unpack(self.desc, data)[0]
struct.error: unpack str size too short for format

And:

Unhandled Error
Traceback (most recent call last):
  File "/root/p2poolSoloMWEB/p2pool/main.py", line 723, in run
    reactor.run()
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet
/base.py", line 1194, in run
    self.mainLoop()
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet
/base.py", line 1203, in mainLoop
    self.runUntilCurrent()
  File "/usr/local/lib/pypy2.7/dist-packages/Twisted-15.4.0-py2.7-linux-x86_64.egg/twisted/internet
/base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
--- <exception caught here> ---
  File "/root/p2poolSoloMWEB/p2pool/bitcoin/stratum.py", line 75, in _send_work
    x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else
 self.username))
  File "/root/p2poolSoloMWEB/p2pool/work.py", line 238, in preprocess_request
    raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind

After a few minutes it goes back to normal. I'm using master since the last merges jtoomim did.

Thank you,

[jtoomim]

@ikolubr The rawtx branch fixes that.

[ikolubr]

@ikolubr The rawtx branch fixes that.

Thank you. I thought you had merged all into master. ;)

[jtoomim]

@ikolubr Not yet. The rawtx branch appears to be a regression for memory usage, so until that has been solved, I'm reluctant to merge it and worsen p2pool for non-LTC users.