search

moneroexamples / openmonero

Fully open sourced implementation of MyMonero backend
Other
163 stars 135 forks source link

MySQL server has gone away #121

Open moneroexamples opened 5 years ago

moneroexamples commented 5 years ago 
# ERR: SQLException in /home/mwo2/openmonero/src/MySqlAccounts.cpp(select) on line 258
# ERR: MySQL server has gone away
# ERR: SQLException in /home/mwo2/openmonero/src/MySqlAccounts.cpp(insert) on line 285
# ERR: MySQL server has gone away

Sometimes it happens. Don't know yet why.

Previously had simillar problem:

https://github.com/moneroexamples/openmonero/issues/10

italocoin-project commented 5 years ago

This could be fixed with force connect check this maybe can help, i haven't encountered this issue maybe because of this

https://github.com/italocoin-project/bittube-light-wallet/commit/d34cf46abbf59feec7415aa17bcaa94797973b92

moneroexamples commented 5 years ago

Thx. Will check that.

moneroexamples commented 5 years ago

Thanks. I checked the code and this may work, and will reconnect you. But the problem I see is that the query data will be lost. The code that is calling these queries will not try to re-call it again. So you may end up with missing outputs or inputs in your database after reconnecting in catch part.

I think for reconnection, the better place would be before the query is executed, i.e., where conn->check_if_connected();. So maybe this could be modified, so that it reconnects.

There can also be other reasons why the connection is failing, e.g., default settings in mysql/mariadb being inadequate.

I will try to investigate what's happening. Maybe mysql/mariadb logs will some info about that.

italocoin-project commented 5 years ago

No problem, i see, this could bring up some problems if it would miss outputs or inputs from mysql DB, but then, if the input gets missed, on a rescan it should be reeaded, because the rescan will find the imputs/outputs from blockchain's db, isn't it? let me know if you find a better way to deal with timeouts, thanks :)

moneroexamples commented 5 years ago

no problem and yes, would have to rescan to find missing txs/outputs/inputs.

I just setup regular mariadb on my linux box, instead of using docker, and will try to get some logs out of it.

moneroexamples commented 5 years ago

I added ReconnectOption to mysql, so now it will reconnect when connection drops.

https://github.com/moneroexamples/openmonero/blob/9e3b72fb75167a535986c79894d0b272250cfed7/src/db/MySqlAccounts.cpp#L228-L229

There still can be a problem, but at least connection should not be dropped permanently. If I manually kill the connection in mariadb I get this, but then it reconnects:

# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(select) on line 267                                                                                                                                                                                   
# ERR: Lost connection to MySQL server during query

In the current devel branch I also added Ctrl+C support to greacfully stop openmonero, and the logging got extended.

italocoin-project commented 5 years ago

You think this option is better then force connect that i've mentioned above? nice work with Ctrl+C support, great job man.

I have another issue: MDB_READERS_FULL limit basicaly when i get MANY connections i get the MDB_READERS_FULL limit under heavy load the db leaks readers or something. Did you encountered this problem?

moneroexamples commented 5 years ago

You think this option is better then force connect that i've mentioned above?

I think so. Its directly implemented in mysql++ and propagated to mysql. It's also recommended in mysql++ docs regarding the timeouts: https://tangentsoft.com/mysqlpp/doc/html/userman/tutorial.html#conn-timeout

Will see how it will work. Can always remove it if its not helpful.

I have another issue: MDB_READERS_FULL limit

I had it maybe only once. It was also reported before: https://github.com/moneroexamples/openmonero/issues/97

This is one reason why Ctr+C was added. With it, openmoenro will close the lmdb's blockchain database that it opens for reading. You can see this when you run openmonero with -m 3 (level 3 log for monero). This will show you that database is closed properly now when Ctrl+C is detected:

2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:547  Blockchain::deinit
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:549  Stopping blockchain read/write activity
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1448 BlockchainLMDB::close
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1464 BlockchainLMDB::sync
2019-01-23 00:16:01.733     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:564  Local blockchain read/write activity stopped successfully
2019-01-23 00:16:01.733     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1178 BlockchainLMDB::~BlockchainLMDB

When you force close it, openmonero has no chance to close the lmdb which it opens and it seems to keep accumulating the open connections. Before I just deleting lmdb.lck (e.g. in home/mwo/.bitmonero/lmdb/lock.mdb) and it solved the problem in my case so that new one is created. Not sure about this though.

Similar issue was also reported before for monero https://github.com/monero-project/monero/issues/2741 with suggested solution.

when i get MANY connections

How many is MANY?

italocoin-project commented 5 years ago

How many is MANY?

Send more than 126 different login request to openmonero and you will get the issue, let me know

yes, now it actually closes properly, what if it gets killed?

moneroexamples commented 5 years ago

126 different login request to the mywallet

I see. Probably will need to rethink how lmdb is accessed in openmonero. At the moment, there will be 126 search tx threads accessing the lmdb at the same time. So maybe will have to implement some pooling when only few threads access lmdb at the same time, or move to using rpc to deamon instead of accessing lmdb directly.

what if it gets killed?

Thank it will be like before, all connections, threads will be terminated immedietly. I think it's better to let the all threads, mysql and lmdb close properly when openmonero stops.

italocoin-project commented 5 years ago

or move to using rpc to deamon instead of accessing lmdb directly

rpc to daemon could solve the problem for good.

maybe will have to implement some pooling when only few threads access lmdb at the same time

It means that someone will have to wait until the threads gets freed again, i think we should test them both

moneroexamples commented 5 years ago

rpc to daemon should also be much easier to implement, as the second possible solution would require major changes to architecture of openmonero. But rpc will also wont be simultaneous, as is needs also synchronization.

italocoin-project commented 5 years ago

rpc to daemon should also be much easier to implement, as the second possible solution would require major changes to architecture of openmonero.

I agree

But rpc will also not be simultaneous, as is needs also synchronization.

Thats also true, but i think the best solution is the rpc to daemon, how is mymonero handling with many connections? I assume that they have over 126 consecutive login requests

moneroexamples commented 5 years ago

mymonero handling with many connections?

Don't know. Their backend is closed sourced. But from what I know they use go and mysql, so I think rpc also would be easiest.

But for mass scale, applications such as openmonero/mymonero are "easily" parallelized, as you can spin up, e.g., 10 instances of the backend and daemons, on 10 servers. One possible way of doing this with some details has already been proposed for openmonero: https://github.com/moneroexamples/openmonero/issues/95#issuecomment-414677408

italocoin-project commented 5 years ago

so I think rpc also would be easiest.

Yes, that would be the best option, do you plan to implement RPC to DAEMON soon? And about more instances, that would be a good idea too, a system to sync all mysql db on all instances would solve the panic in users if they would see that the wallet syncs from 0 again

moneroexamples commented 5 years ago

Yes, that would be the best option, do you plan to implement RPC to DAEMON soon?

Yes, though "soon" in monero world does not mean anything:-) But yes, this and mysql-thing will be the issues I will be focusing the most at present. When complete? No timeframe on this. Hopefully "soon":-)

Just spin up new branch for that: https://github.com/moneroexamples/openmonero/tree/new_rpc So all new changes will be there for experimentation.

italocoin-project commented 5 years ago

Nice, i will merge it tomorrow morning and test out the branch, i will let you know if i face issues, thanks :)

moneroexamples commented 5 years ago

Not sure I will have there anything new for tomorrow, but this is the branch when the rpc changes will be taking place.

No problem. Good you can provide extra info and much needed testing for openmonero. Thanks.

italocoin-project commented 5 years ago

Thank you for your hard work, i really like your projects and work, you can count on my help for testing out, i use openmonero for my project to, thanks for that :)

moneroexamples commented 5 years ago

I made a seprate issue for the LMDB readers limit: https://github.com/moneroexamples/openmonero/issues/127