selsta
commented
9 months ago What type of storage is F:\? HDD, SSD, ...?
Open Erquint opened 9 months ago
selsta
commented
9 months ago What type of storage is F:\? HDD, SSD, ...?
Erquint
commented
9 months ago My fastest old HDD. Here's a Cross Platform Disk Test result:
This doesn't matter. If it took three days and gave me an actually deflated pruned DB — I wouldn't complain much.
Was it supposed to copy all of the txs_prunable? It all looked good until that started.
selsta
commented
9 months ago We have a pull request to speed up monero-blockchain-prune: https://github.com/monero-project/monero/pull/8503
Together with an SSD it should take about 1-2h, we just haven't included the code yet in the latest release version, only master branch.
HDDs have slow random read / write speeds so they will always be significantly slower than SSDs in monero.
Regarding your reported issue, the easiest way to solve it would be to re-sync from scratch with setting --prune-blockchain from the beginning. This issue should stay open until someone reproduces it.
selsta
commented
9 months ago So I had a node that was, as I recall, initially ran with pruning, then without, then pruned again using the Monero daemon.
Can you explain this in more detail? You can't unprune a daemon so it was always pruned, unless you deleted the blockchain in-between?
Erquint
commented
9 months ago No, I didn't delete it. Just ran the daemon without the pruning option after initial DB creation in GUI wallet, where I picked the pruned option, until it gradually reached 172 GB, which seems like the size of the full ledger.
It almost sounds like another issue to be opened for monerod separately, but I don't have detailed statistics to illustrate it.
Unless you wanna explain to me how a supposedly pruned DB of either/both monerod and monero-blockchain-prune is 172 GB.
monero-blockchain-prune reported the DB created by monerod as pruned.
And there was some option to check if it's pruned in monerod itself, IIRC, which also reported it as pruned.
Erquint
commented
9 months ago As for the HDD and speed… Look, the point I'm making is… The pruned entries are already marked for rewriting/discarding. All that I would expect of a program intended to deflate the DB, to discard those stained entries, is to simply sequentially copy the entries that aren't stained, skipping the stained ones. How complicated could that really be? I'd be able to copy this entire disk twice over in a matter of minutes, hours at most with the slow built-in OS copier. Not days. It's insult to injury when it copies the entries it promised to skip and then also takes an eternity to do what I never wanted it to do.
hyc
commented
9 months ago No, I didn't delete it. Just ran the daemon without the pruning option
That doesn't matter. Once you use the option, that's recorded in the DB so it will always be pruning after that point. Only deleting the blockchain DB would reset that option.
Erquint
commented
9 months ago You'd think that, but 172 GB… Refresh the page. I edited the message you responded to with more detail. Expected GitHub's dynamic UI to reflect my changes in real time. Detailed it in the head post too now.
nahuhh
commented
9 months ago "A" appears to be OPs case:
A) Internally Pruned
--prune-blockchain
= The database is now pruned internally, but is still ~170gb on disk (expected) monero-blockchain-prune --copy-pruned-database
= copies the database from 2 (~170gb, expected?)if OP did 3b (below):
3b. Run monero-blockchain-prune w/o the copy flag
= does this shrink the db? Or should this error out and say db is already pruned?
edit: op says this does indeed throw an error
Perhaps op expected this to shrink the db instead of throwing an error
This use case should work: B) Unpruned BC
monero-blockchain-prune
= prunes the database from 1 (expected)nahuhh
commented
9 months ago Unless you wanna explain to me how a supposedly pruned DB of either/both
monerodandmonero-blockchain-pruneis 172 GB.
pruning doesn't free already allocated space on disk. It frees space within the db and wont allocate more until the "internal" db reaches the size on disk.
monero-blockchain-prunereported the DB created bymonerodas pruned. And there was some option to check if it's pruned inmoneroditself, IIRC, which also reported it as pruned.
if you ever ran --prune-blockchain, it was pruned (pruned != shrunken) and monerod is reporting correctly.
This appears to be caused by usage of --copy-pruned-database.
I'm interested in if it shrinks a pruned but overallocated BC if the copy flag if dropped.
selsta
commented
9 months ago I think this is an issue with a bugged / corrupted database, @Erquint says they initially started monerod with the --pruned-blockchain flag, which means it should have never gotten that big in the first place.
Erquint
commented
9 months ago Correction to a thing many of you seem to misinterpret: I didn't start the DB with a pruning flag. The official Monero GUI Wallet offered the two choices and I picked pruned in the GUI when initializing. I added the flag at some later point. But as we established, if the GUI did its job — the flag should do nothing.
Admittedly, I initialized the DB a very long time ago.
@nahuhh
Perhaps op expected this to shrink the db instead of throwing an error
It would be nice if it did what's on the tin. Can't argue there.
Run
monero-blockchain-prune --copy-pruned-database= copies the database from 2 (~170gb, expected?)
If you suppose this behavior expected, care to explain what the purpose is?
To basically replicate the function of monerod --prune-blockchain?
If that's the case — why didn't the monero-blockchain-prune complain that the DB is already pruned with the --copy-pruned-database flag as it did without it? This inconsistency created much confusion, without it even being clear if either both or none modes should raise that complaint.
pruning doesn't free already allocated space
Well aware. Everything I wrote in this here entire issue is under that very assumption.
I think you missed the monero-blockchain-prune part of my question. That's what was supposed to shrink it. Surely, running all of the options on the table must've produced a shrunk DB file — otherwise shrinking wouldn't even be a thing we're able to discuss as an applicable concept.
Alright, so if we go with the assumption that monero-blockchain-prune --copy-pruned-database worked as expected, then the issue boils down to monero-blockchain-prune (no such flag) not working as expected, printing Blockchain is already pruned, use --copy-pruned-database to copy it anyway and halting.
Initially setting up the node in the GUI wallet a long time ago I picked the pruned option. The CLI option for pruning was not added but I assumed a DB seeded as pruned wouldn't be able to grow to the full ledger size. I was wrong.
Ran the daemon ever since without the pruning CLI option until it gradually reached 172 GB, which seems like the size of the full ledger. Added the CLI option and ran like that recently.
There was some option to check if it's pruned in
moneroditself, IIRC, which reported it as pruned.However, pruning with
moneroddoes not reduce the file size, as it only marks entries available for overwriting. I wanted to deflate the database by discarding the marked entries from disk. And that's whatmonero-blockchain-pruneis supposedly for.Running
monero-blockchain-pruneprintsBlockchain is already pruned, use --copy-pruned-database to copy it anyway. So I added that option.P.S. At the point where it rapidly shot past the expected size, it started using all of the available memory on Windows 10 on the file handle of the initial DB — probably a read cache/buffer.