logging neighbors, devp2p hello, eth status info into mysql

[SimonSK]

All commits so far. I was going to submit multiple PRs, but branches kept diverging, so here it is. I will try my best to summarize them below.

Minor changes and other changes related to the last PRs (#13, #14):

1. Log messages related to sql statements are now moved to Debug (if err) and Trace (if ok)
2. mysql/mysqldb.py: some changes to sql table schemes
3. p2p/discover: sql related functions in udp.go are now moved to sql_stmt.go. addNodeMetaInfo is removed as we don't actually care about the node addresses from Neighbors packets.

DEVp2p Hello (p2p package):

1. p2p/node_info.go
   • The structs used to manage node information and relevant functions are added here.
   • Info struct contains a node's DEVp2p Hello and Ethereum Status info and also keeps track of the most recent db entry's row ID.
   • knownNodeInfos is a struct that wraps a map of Infos. It contains a mutex lock.
   • storeNodeInfo does the following: If a new node (not in the in-memory database): 1. add a new entry to the sql db, 2. get the entry's row ID, 3. store the information in-memory, 4. add the node as a new static node. Else, check if any information (except timestamp and remote_port) changed. If changed, treat it as a new node; otherwise, update the existing db entry.
   • KnownNodes is called by the Javascript Console to print out all node information. I implemented it for debugging purpose.
2. p2p/rlpx.go
   • doProtoHandshake and readProtocolHandshake in p2p/rlpx.go are modified to return the message received time.
   • doProtoHandshake behavior changed to attempt to read the peer's Hello message before we send out ours. This change is to avoid situations where the peers reject us based on our information before their information reach us.
3. p2p/rlpx_test.go
   • minor changes to remain consistent with changes made in p2p/rlpx.go
4. p2p/server.go
   • Line 160: unlike other prepared statements, GetRowIDStmt is made public because the Ethereum protocol manager also uses it.
   • Line 385: p2p/sql_stmt.go:initSql sets up db connection and prepares statements when the p2p server starts.
   • Line 370: p2p/sql_stmt.go:CloseSql closes the statements and db connection when the server stops. It is made public because node package also needs to call it when the p2p server fails to start properly.
   • Line 799: p2p/node_info.go:storeNodeInfo is called inside SetupConn. This is where storing to mysql db and updating in-memory database occur.
   • Line 785: p2p/sql_stmt.go:addNodeMetaInfo is manually called if the handshake fails because the peer said "TooManyPeers".
5. p2p/sql_stmt.go: sql related functions
   • Line 38: during the init, loadKnownNodeInfos is called to fill the in-memory database with the known information from our sql db. While loading, each node address is added to StaticNodes so that they are re-dialed automatically (currently at the default hardcoded frequency 30s).

Ethereum Status (eth package):

1. eth/backend.go
   • The protocol manager shares the p2p server's GetRowIDStmt and KnownNodeInfos (in-memory database). It directly accesses the map of node infos (not the knownNodeInfos wrapper) because it won't need the mutex lock (it does not change/add anything new to the map).
2. eth/handler.go
   • Line 93: Forgot to add this back while reverting the change in the PR #13
   • Line 164: eth/sql_stmt.go:prepareSqlStmts prepares statements when the eth protocol manager starts.
   • Line 188: eth/sql_stmt.go:closeSqlStmts closes the statements when the manager stops.
   • Line 209: statusDataWrapper is used to pass the peer's status info (with received timestamp and handshake failure error) to the protocol manager.
   • Line 221: eth/node_info.go:storeEthNodeInfo is called. This is where storing to mysql db and updating in-memory database occur.
   • Line 289: eth/node_info.go:storeEthNodeInfo is called when unexpected Status messages arrive.
3. eth/node_info.go
   • storeEthNodeInfo only considers nodes that are present in the in-memory database. If the current information contains no Eth Status information, add them to the existing node entry. If eth information changed: 1. add a new node entry to the sql db, 2. get the rowid; else, just update the eth information of the existing entry.
4. eth/peer.go
   • Line 238: validStatus returns true if the peer's status resulted in error because of the specified mismatches.
   • Handshake and readStatus modified to handle statusDataWrapper
5. eth/sql_stmt.go: sql related functions

internal/web3ext/web3ext.go and node/api.go: to allow calling KnownNodes through the Javascript console

TODOs (added to #15):

• ORM
• ~make re-dial frequency as an configurable option~
• ~update daofork check~

[SimonSK]

After performing DAO fork check through BlockHeaders/GetBlockHeaders exchange, dao_fork field on the database and DAOForkSupport field in the Info struct are updated.

1. The field is changed from bool to a signed int. 1 = true (mainnet), 0 (or NULL on db) = unknown, -1 = false (classic).
2. eth/node_info.go
   • Line 70~ (storeDAOForkSupportInfo): if current DAOForkSupport is 0, then simply update the existing entry as it is the first time we learned about the node's DAOForkSupport status. Otherwise, if the DAOForkSupport status changed for some reason, add a new entry, copying all other information from the previous entry.

[SimonSK]

If any part of the initial sql preparation (p2p/server.go:385) returns error, the client goes through proper shutdown procedures. If the db handle exists but a prepared statement variable is null, the client immediately exits using log.Crit. If the client fails to execute a prepared statement, it logs the error as well as the values that were supposed to be added to the db.

After the changes,

• Trace: successful event that doesn't involve changing data, like open/closing database handle or preparing statements
• Debug: successful event that involves changing data, like executing prepared statements
• Error: failure of those mentioned above 2
• Crit: failing to initialize/prepare sql related variables or prepared statements missing when expected

[SimonSK]

Another TODO added to #15 is removing case-study style log messages from node-finder. That was the first thing I brought from case-study thinking it would be useful, but now I find it too much. I plan to remove them in the next PR, which will also include the following two:

• make re-dial frequency as an configurable option
• call all sql execution functions as go routine