selsta
commented
3 years ago Does this issue happen with v0.17.1.8? https://www.getmonero.org/downloads/
Closed arizvisa closed 3 years ago
selsta
commented
3 years ago Does this issue happen with v0.17.1.8? https://www.getmonero.org/downloads/
arizvisa
commented
3 years ago Just resumed synchronization with commit 36dfd41e01ebc852e094f268433344404c042591, and it seems that I'm past that RandomX allocation issue.
2021-01-04 16:58:25.710 I [127.0.0.1:18080 OUT] Sync data returned a new top block candidate: XXXXXXX -> XXXXXXX [Your node is XXXXX blocks (x.x months) behind]
2021-01-04 16:58:25.710 I SYNCHRONIZATION started
2021-01-04 16:58:35.194 I Synced XXXXXXX/XXXXXXX (XX%, XXXXX left)
2021-01-04 16:58:37.133 I Synced XXXXXXX/XXXXXXX (XX%, XXXXX left)I'm stuck on a hotspot for a bit, but later this week (when I get to a place the b/w is free) I'll resume from the previous height that was causing those previous reports and let you guys know prior to closing this issue.
Voker57
commented
3 years ago Still happens for me with 0.18.1.8, monero quickly swells to 6gb RAM on certain block.
dEBRUYNE-1
commented
3 years ago @Voker57 - Please try compiling the release-v0.17 branch.
https://github.com/monero-project/monero#compiling-monero-from-source
moneromooo-monero
commented
3 years ago Should now be fixed. I'll close once you confirm. Nothing to do with blocks fwiw.
arizvisa
commented
3 years ago kk.
ah. weird then, as it kept reproducing at around the same synchronization point.
arizvisa
commented
3 years ago Okay. With commit 36dfd41e01ebc852e094f268433344404c042591 that got me past the RandomX issue. I encounter the following, but after killing the process I'm able to resume synchronization.
2021-01-10 02:01:46.664 I Synced XXXXXXX/XXXXXXX (XX%, XXXXX left)
2021-01-10 02:01:46.940 I Synced XXXXXXX/XXXXXXX (XX%, XXXXX left)
2021-01-10 02:01:48.955 I [batch] DB resize needed
2021-01-10 02:01:48.955 I Synced XXXXXXX/XXXXXXX (XX%, XXXXX left)
2021-01-10 02:04:19.062 W Failed to set new mapsize: Cannot allocate memoryI'm rebuilding and re-copying the database I have from XX%, so I'll let y'all know in a bit about the symptoms with commit 8fef32e45c80aec41f25be9d1d8fb75adc883c64 which should be the latest (as of now is tagged v0.17.1.9) in the release-v0.17 branch.
moneromooo-monero
commented
3 years ago Do you have enough disk space ?
arizvisa
commented
3 years ago $ df -h | grep monero
/dev/sdb1 239G 161G 78G 68% /run/media/user/moneroAfter I restore the former DB, I'll resume synchronization w/ a binary built from the v0.17.1.9 tag and full logging enabled. I'm just rsync'ing from my backup usb over my monero one, so the bitmonero.log is being directly overwritten rather than rolled over.
arizvisa
commented
3 years ago Okay. Not sure if it's happening at the same place, but the logging has blocked at the following, and the daemon isn't replying to my "help" command.
2021-01-10 03:22:19.086 T BlockchainLMDB::get_block_timestamp
2021-01-10 03:22:19.086 T Blockchain::check_block_timestamp
2021-01-10 03:22:19.086 T Blockchain::get_difficulty_for_next_block
2021-01-10 03:22:19.086 T BlockchainLMDB::height
2021-01-10 03:22:19.086 T Blockchain::get_tail_id
2021-01-10 03:22:19.086 T BlockchainLMDB::top_block_hash
2021-01-10 03:22:19.086 T BlockchainLMDB::height
2021-01-10 03:22:19.086 T BlockchainLMDB::get_block_hash_from_height
2021-01-10 03:22:19.086 T Blockchain::get_tail_id
2021-01-10 03:22:19.086 T BlockchainLMDB::top_block_hash
2021-01-10 03:22:19.086 T BlockchainLMDB::heightMemory usage seems to be slowly increasing...
$ free -h
total used free shared buff/cache available
Mem: 15Gi 4.0Gi 1.4Gi 222Mi 10Gi 10Gi
Swap: 13Gi 169Mi 13Githen a minute later:
~$ free -h
total used free shared buff/cache available
Mem: 15Gi 4.0Gi 571Mi 217Mi 10Gi 10Gi
Swap: 13Gi 169Mi 13Gi
$ free -h
total used free shared buff/cache available
Mem: 15Gi 4.0Gi 558Mi 217Mi 10Gi 10Gi
Swap: 13Gi 169Mi 13GiAhh.."sync_info" just responded (see attached file: blah.txt) and I realized I forgot to blacklist tor endpoints prior to running it this time...
"status" returned too:
status
Height: XXXXXXX/XXXXXXX (XX.X%) on mainnet, not mining, net hash 1.43 GH/s, v12, 12(out)+0(in) connections, uptime 0d 0h 24m 8sI'm backing up the log-level=4 logs and will attach them in a sec...
arizvisa
commented
3 years ago It seems like just "help" stopped emitting things for some reason...I snapshot the logs and moved them to another hd, but am waiting till it gets to past XX% which is closer to where the issue was that I first reported since I'm not sure if this is first weird symptom is actually related to my original report or not.
arizvisa
commented
3 years ago Okay. It look a while running it all day, but now I get.
2021-01-11 13:23:52.614 D Run server thread name: P2P
2021-01-11 13:23:52.614 D Run server thread name: P2P
2021-01-11 13:23:52.614 D Run server thread name: P2P
2021-01-11 13:23:52.614 D JOINING all threads
2021-01-11 13:23:53.616 D STARTED PEERLIST IDLE HANDSHAKE
2021-01-11 13:23:53.616 D FINISHED PEERLIST IDLE HANDSHAKE
2021-01-11 13:23:53.617 T Blockchain::get_current_blockchain_height
2021-01-11 13:23:53.617 T mdb_txn_safe: destructor
2021-01-11 13:23:53.617 T BlockchainLMDB::height
2021-01-11 13:23:53.618 T batch transaction: aborted
2021-01-11 13:23:53.618 T BlockchainLMDB::block_rtxn_start
2021-01-11 13:23:53.620 D Checking for a new monero version for source
2021-01-11 13:23:53.620 D Checking updates for source monero
2021-01-11 13:23:53.621 T mdb_txn_safe: destructor
2021-01-11 13:23:53.621 D get_next_needed_pruning_stripe: want height XXXXXXX (XXXXXXX from blockchain, XXXXXXX from block queue), stripe 7 (0/12 on it and 0 on 8, 0 others) -> 7 (+0), current peers
2021-01-11 13:23:53.621 D Making expected connection, type 0, 0/2 connections
2021-01-11 13:23:53.622 D Considering connecting (out) to anchor peer: xxxxxxxxxxxxxxxx 127.0.0.1:18080
2021-01-11 13:23:53.624 D test, connection constructor set m_connection_type=2
2021-01-11 13:23:53.624 D connections_ size now 1
2021-01-11 13:23:53.625 D Trying to connect to 127.0.0.1:18080, bind_ip = 0.0.0.0
terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>'
what(): boost: mutex lock failed in pthread_mutex_lock: Invalid argumentblock position is at:
monero-logs7/monero.20210110221238.log-2021-01-11-04-40-12:2021-01-11 04:40:03.482 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1644 Synced XXXXXXXX/XXXXXXXX (XX%, XXXXX left) (1.699609 sec, 11.767412 blocks/sec), 99.380096 MB queued in 80 spans, stripe 7 -> 7: [moooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo.oo.o..]
monero-logs7/monero.20210110221238.log-2021-01-11-04-40-12:2021-01-11 04:40:06.995 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1644 Synced XXXXXXXX/XXXXXXXX (XX%, XXXXX left) (3.512267 sec, 5.694328 blocks/sec), 97.158722 MB queued in 80 spans, stripe 7 -> 7: [moooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo.o...]
monero-logs7/monero.20210110221238.log-2021-01-11-04-40-12:2021-01-11 04:40:12.372 [P2P1] INFO global src/cryptonote_protocol/cryptonote_protocol_handler.inl:1644 Synced XXXXXXX/XXXXXXX (XX%, XXXXX left) (5.375898 sec, 3.720309 blocks/sec), 96.088898 MB queued in 85 spans, stripe 7 -> 7: [moooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo..]So this is definitely at the time that I previously encountered the memory issue.
One moment while I tar up all the different logs and sort them out, get symbols for core files, etc.
[768264.445649] traps: monerod[1100109] general protection fault ip:7fa1f8629967 sp:7f89028fa640 error:0 in libc-2.32.so[7fa1f8629000+14f000]
moneromooo-monero
commented
3 years ago Do you have a stack trace for this ? gdb monerod run usual-arguments-here bt full
moneromooo-monero
commented
3 years ago up 1 info locals print this print *this
arizvisa
commented
3 years ago Okay. Looks like the memory exhaustion problem is solved w/ commit 8fef32e45c80aec41f25be9d1d8fb75adc883c64. Closing this issue.
moneromooo-monero
commented
3 years ago Would be nice to fix the crash though.
This has been occurring at the _exact_ same time during multiple synchronizations. Some issues are titled with memory leak, but this 100% seems like a malicious block or flaw in the implementation of cryptonote. The readline interface does not interpret commands after encountering the memory issue, so normally you're forced to kill the process. Prior to commit b2221881a172f445ca2bb9e962033796d5cf70ac, however, this would seem to corrupt the lmdb as anytime I'd load
monerodagain, non-recoverable mutex errors would occur.Condition 1:
Backtrace:
I have a backup of the db at around 94%, so I can probably reproduce this with full logs if necessary. Upon restarting monerod (commit b2221881a172f445ca2bb9e962033796d5cf70ac), later there's an issue with RandomX.
Condition 2:
Backtrace:
lmk if you'd prefer I demangle these symbols...