search

ferencberes / ethp2psim

Ethereum P2P network privacy simulator
https://arxiv.org/abs/2306.15024
GNU General Public License v3.0
6 stars 2 forks source link

Evaluation #2

Closed seresistvanandras closed 1 year ago

seresistvanandras commented 2 years ago

We need to implement a class that evaluates the results of a simulation or of an adversary/protocol. Input: A matrix whose rows are the simulated messages and the columns are the network participants. The adversary assigns a probability distribution to each message denoting its guess of the message originators. Input2: the ground truth data of the actual message/tx/block originators. Long-term goal: it should be a sparse matrix to optimize memory usage. Output: precision, recall, entropy, accuracy, NDCG, etc.

Food for thought: Define a meaningful relevance metric for this context. Maybe somehow include the distance from the actual message originator? or the attacker's assigned probability etc.

ferencberes commented 2 years ago

Later, let's get back to the question of evaluation with respect to messages sent by adversaries. Should we eliminate this scenario? Adversaries are out there only to eavesdrop and not to interact, right?

ferencberes commented 1 year ago

Currently, hit ratio, inverse rank and NDCG seem to be good metrics for evaluation. Entropy is also supported but it cannot take into account the ground truth (message source information).