Monero Research Lab Meeting - Wed 10 July 2024, 17:00 UTC

Log:

< rucknium:monero.social > Meeting time! https://github.com/monero-project/meta/issues/1037

< rucknium:monero.social > 1) Greetings

< rbrunner > Hello

< 0xfffc:monero.social > Hi everyone

< jberman > waves

< vtnerd:monero.social > Hi

< rucknium:monero.social > 2) Updates. What is everyone working on?

< 0xfffc:monero.social > (1) Fixed a bug, (2) working on start up speed up: https://github.com/0xFFFC0000/monero/pull/26

< rucknium:monero.social > me: Helping with stressnet. Reading papers about gossip networks and analyzing the Monero p2p transaction gossip messages.

< jeffro256:monero.social > howdy

< jberman > me: completed first pass at growing and trimming the tree (db and tests implemented, tests passing)

< vtnerd:monero.social > Not much to report, between work contracts atm

< jeffro256:monero.social > me: Jamtis-RCT. Specifically, the conversion functions between X25519 and ed25519 points for legacy address support. If anyone knows a way to find the ed25519 (x, y) values from X25519's x value without doing explicitly finding the y value for X25519 by performing a square root op, please let me know

_< sgp:monero.social >__ Hello

< rucknium:monero.social > 3) Stress testing monerod https://github.com/monero-project/monero/issues/9348

< jberman > ("the tree" = fcmp curve trees merkle tree

< rucknium:monero.social > I think we found another problem that could prevent block sync. Sometimes a valid block can be placed on the m_invalid_blocks list. It looks like a node can get on a short alt chain. Then it marks one of the main chain's blocks as invalid. Because the block is "invalid", it stops syncing to the main chain and bans nodes that send the "invalid" block.

< rucknium:monero.social > I guess that nodes are applying the alt chain's rules for validity to the main chain, and think it's invalid. Probably it has something to do with the 100 block median block size, the penalty area, and how much miners can pay themselves. Maybe miners pay themselves too much on the main chain's block when the alt chain's rules are applied.

< rucknium:monero.social > The nodes can start syncing again if they shut down and restart or if the node operator inputs flush_cache bad-blocks into the monerod console.

< rucknium:monero.social > In the incident that I analyzed closely, the alt chain was 2 blocks long

< rucknium:monero.social > I guess the problem is the block size computation because AFAIK this type of thing doesn't happen on the mainnet, but mainnet usually doesn't increase the 100-block median block size.

< rucknium:monero.social > AFAIK, there is no plan yet to try to debug this

< rucknium:monero.social > This week I started spamming 144in/2out txs to see if the out-of-memory issue that some people saw on mainnet would happen. I've seen a modest increase in RAM usage, but nothing that would cause a major problem. Most nodes on stressnet have 30 connections or fewer, so the OOM issue may be due to number of connections more than how many inputs txs have

< rucknium:monero.social > But the CPU usage of nodes has increased because verifying inputs requires more CPU per byte of data than verifying outputs.

< rucknium:monero.social > A low-end node (4GB RAM, 2 CPU threads) took 30 seconds per block recently to catch back up with chain tip with 4MB blocks mostly filled with 144in/2out txs.

< rucknium:monero.social > Any more comments on stressnet?

< jberman > sounds like a nice list of daemon issues has formed for would-be takers to pursue further

< rucknium:monero.social > Here's the relevant part of the log file when the nodes fail to sync, by the way:

< rucknium:monero.social > D [: INC] first block hash , last

< rucknium:monero.social > D block found in main chain

< rucknium:monero.social > D block <33a5c86ea4a291ddf810df8620784ec8c9fea4014280a0d14f4bb6f6e0c17348> found in alternative chains

< rucknium:monero.social > D block <4a76385a6ddb03134ce72d88c8e08c26737f43a5fb88ca2f278ef067e39fbaf2> found in m_invalid_blocks

< rucknium:monero.social > E [: INC] Block is invalid or known without known type, dropping connection

< rucknium:monero.social > 4) Potential measures against a black marble attack. https://github.com/monero-project/research-lab/issues/119

< rucknium:monero.social > I don't have much to report on this issue today except I found a paper that proposes an alternative to Dandelion++:

< rucknium:monero.social > Franzoni, F., & Daza, V. (2022). Clover: An anonymous transaction relay protocol for the bitcoin p2p network. https://moneroresearch.info/index.php?action=resource_RESOURCEVIEW_CORE&id=222

< rucknium:monero.social > I just skimmed the paper. It claims that Clover is as good as D++, simpler, and works well for nodes that have no incoming connections (i.e. its ports are closed).

< jeffro256:monero.social > Hmmmm that's a really interesting issue. I wonder what the value of even having a running "invalid blocks" list is, realistically. Obviously it would speed up failing known repeated bad blocks for honest peers, but the PoW check should be the very very first check performed on block data. This will mathematically limit the number of non-trivial bad blocks that can exist. An unhone<clipped messag

< jeffro256:monero.social > st peer, however, can send any number of garbage blocks that don't pass PoW verification.

< rucknium:monero.social > The last part is interesting. A few months ago we discussed a potential issue with D++ when nodes have no incoming connections. D++ treats incoming and outgoing connections differently for its graph theory to work.

< rucknium:monero.social > jeffro256: In the log files I found an error message about the "invalid" block pointing to this line of code: https://github.com/monero-project/monero/blob/master/src/cryptonote_core/blockchain.cpp#L1217

< rucknium:monero.social > Right above that is a comment: "FIXME: Why do we keep invalid blocks around? Possibly in case we hear about them again so we can immediately dismiss them, but needs some looking into."

< rbrunner > Waiting for a fix for 4 years :)

< rucknium:monero.social > I've only skimmed the Clover paper. The D++ paper looks at a lot of different threat models. It is a dense paper. On a skim, the Clover paper looks like it analyzes fewer threat models. Anyway, I will keep the paper and come back to it later.

< rbrunner > Ah, now, that remark is 9 years old.

< rucknium:monero.social > My naive view is to try to fix the miscalculation of the block validity first.

< rucknium:monero.social > 5) Research Pre-Seraphis Full-Chain Membership Proofs. https://www.getmonero.org/2024/04/27/fcmps.html

< rucknium:monero.social > kayabanerve, kayabanerve

< rucknium:monero.social > By the way, I've listed "Uniformity of Monero's hash-to-point function" as a separate agenda item. (It's next on the list.)

< kayabanerve:monero.social > I have no updates on either.

< kayabanerve:monero.social > But thank you :)

< rucknium:monero.social > I skimmed a few related papers on the hash-to-point function. One of them proves that a different hash-to-point function is approximately uniformly distributed. I wonder: would it be useful to do some preliminary statistical tests of the uniformity of Monero's function? If it fails one or more tests, that's a good reason for a researcher to look at its potential biasedness more formally.

< rucknium:monero.social > Diehard-like tests.

< rucknium:monero.social > kayabanerve: What do you think? ^

< kayabanerve:monero.social > I'm honestly unsure the methodology for testing. If you have learned it and want to, I'd be happy to comment as I can, but my expertise on the subject only went as far as to know to hire someone to ensure we're not messing up :p

< rucknium:monero.social > AFAIK, what I would need would be a large number of the points to test if they are uniformly distributed or (even better) an easy-to-use function that I could use to generate points at will.

< rucknium:monero.social > I would have to look into this more, but that's what I think would be required.

< rucknium:monero.social > And then I would have to understand elliptic curves a little if the test is actually uniformity on the curve instead of just uniformity on the set of integers

< jeffro256:monero.social > Rucknium: you probably know this already, but it's important to note that a random ed25519/X25519 public key (and therefore key images) as we serialize them will NOT look like a uniform random string since not all (X, Y) combos are valid. And of those (X, Y) combos, we only really use 1/8 of them (the one's without cofactor)

< rucknium:monero.social > If it doesn't fail any tests, that doesn't necessarily mean that it's unbiased. But if it does fail tests, then that could make a researcher more interested in it.

< kayabanerve:matrix.org > There's a C++ and a Rust impl

< kayabanerve:matrix.org > They explicitly shouldn't be uniform over the byte array.

< kayabanerve:matrix.org > Presumably, the field element should be

< boog900:monero.social > the original CryptoNote whitepaper claims the hash to point function, as used for original ring signatures, does not need to be perfect: `None of the proofs demands Hp to be an ideal cryptographic hash function. It’s main purpose is to get a pseudo-random base

< boog900:monero.social > for image key I = xHp(xG) in some determined way.`

< boog900:monero.social > do our other usages?

< kayabanerve:matrix.org > The original proof's implementation does allow forgeries of you can predict the HtP IIRC.

< kayabanerve:matrix.org > ... if you can control its output to a solved for point?

< kayabanerve:matrix.org > I'm unsure how to phrase it. I don't think that's the risk, due to it using keccak to start, and I wouldn't be surprised if it's fine if it offers a notably reduced bit count than desirable for a hash fn of its length.

< kayabanerve:matrix.org > But I wouldn't take that informal comment at face value.

< rucknium:monero.social > Any other comments on this or agenda items?

< jeffro256:monero.social > I believe I know of a way that that a valid block could be added to the invalid blocks list. In Blockchain::switch_to_alternative_blockchain, if handle_block_to_main_chain returns false for any reason, we add the alt block and its children to the invalid blocks list. However, handle_block_to_main_chain can fail for temporary non-deterministic reasons, one of them being tha<clipped messag

< jeffro256:monero.social > t the transactions of that block are not found inside of the mempool. This might have happened during the stressnet because the mempool got full. So if the node attempts reorging from a worse alt chain to the main chain, but a transaction is missing, the main chain block permanently gets added to the invalid blocks list

< jeffro256:monero.social > I haven't confirmed it yet, so take it with a grain of salt, but that's maybe a possibility

< rucknium:monero.social > "one of them being that the transactions of that block are not found inside of the mempool" If that's true, that is a strange design decision.

< rucknium:monero.social > Transaction propagation on stressnet is not perfect.

< jeffro256:monero.social > Well the node inherently can't determine the validity of a block if it doesn't have each tx on hand

< rucknium:monero.social > Doesn't the fluffy block protocol ask nodes for "missing" txs from the node that sent the block?

< rucknium:monero.social > There are log messages about requesting the missing txs

< jeffro256:monero.social > Yes, usually

< rucknium:monero.social > I mean, not necessarily in this specific bug, but one of the log categories will give you that log message sometimes

< rucknium:monero.social > Maybe the sending node doesn't have the tx in its txpool either?

< rucknium:monero.social > But it would have it in its blockchain

< jeffro256:monero.social > But if a block gets added as an alt block, and the txs were ALREADY in the pool, then the txs aren't forced to stay in the pool like they are with a normal fluffy block directly to the main chain.

< rucknium:monero.social > We can end the meeting here. And continue the debugging discussion after the meeting

< jeffro256:monero.social > So this sequence of events could happen: 1) TX T gets propagated, 2) alt block B1 gets added as alt block, 3) full mempool causes TX T to get pushed out 4) alt block B2 has B1 as previous block and gets added as an alt block 5) we attempt to reorg to block B2 6) TX T is not present 7) blocks B1 and B2 get added as invalid blocks permanently

< jeffro256:monero.social > (B1 contains TX T btw)

< rucknium:monero.social > I think this was happening even when the tx pool wasn't full by the way.

< jeffro256:monero.social > Would you be able to send me the full log please? I would like to see which reason is logged as the initial failure for handle_block_to_main_chain

< rucknium:monero.social > Yes. When the bug occurred I just had log level 0. But the bug seems to happen about once every 3 days per node. Is there a log level I should set on nodes to get more info?

< jeffro256:monero.social > 3 days is actually the interval after which old txs are dropped from the mempool. That might have something to do with it

< jeffro256:monero.social > For logging consensus verification errors, you can use log category "verify" and on ERROR level

monero-project / meta

Monero Research Lab Meeting - Wed 10 July 2024, 17:00 UTC #1037

1034