some path issue in my win10/cygwin env causing NO input .dat files to be processed

[jrp27514]

got the parser working exactly once, then upgraded win10 to wsl and cant get rusty working after another 100 attempts. Seems some problem w/ path to infiles. I am pointing rusty to valid input blk00209.dat (and later) files. I added a couple comments in the parser to confirm it is not getting past that but no clue why and I do not know rust syntax.

I do know it is finding the rustyin files because if I give it an empty folder I see: "thread 'main' has overflow its stack" As I suspect some cygwin/win10/path issue I have tried various usages of / \ and \ path variants but not getting anywhere

I have tried w/ both debun and release builds. Any help or clues **GREATLY APPRECIATED

./target/debug/rusty-blockparser -d k:\temp\rustyin -s 209 balances k:\temp\rustyout [19:01:29] INFO - main: Starting rusty-blockparser v0.8.2 ... [19:01:29] INFO - index: Reading index from k:\temp\rustyin\index ... [19:01:29] INFO - rusty_blockparser::blockchain::parser::index: jrp before advance iter [19:01:29] INFO - rusty_blockparser::blockchain::parser::index: jrp after advance iter [19:01:29] INFO - index: Got longest chain with 0 blocks ... [19:01:29] INFO - blkfile: Reading files from k:\temp\rustyin ... [19:01:29] INFO - parser: Parsing Bitcoin blockchain (range=209..) ... [19:01:29] INFO - callback: Using balances with dump folder: k:\temp\rustyout ... [19:01:29] INFO - parser: Done. Processed 0 blocks in 0.00 minutes. (avg: 0 blocks/sec) [19:01:29] INFO - callback: Done. Dumped 0 addresses.

[gcarq]

How much memory do you have and can you rule out running out of memory?

Also consider issuing cargo clean and build it again and always run the release builds as the debug builds are very slow and the stack might overflow sooner as no optimizations are applied.

You also might want to use this branch if you are using it for bitcoin to rule out any bugs: https://github.com/gcarq/rusty-blockparser/tree/use-bitcoin-lib.

If nothing of the above helps, you might want to increase the stack size manually to see if that fixes it, .e.g.:

$ export RUSTFLAGS="-C link-args=-Wl,-zstack-size=4194304"
$ cargo build --release 

EDIT: I misread your post, please ignore the things above.

Do you have pruning enabled on the node?

[jrp27514]

Michael,

Thanks so much for the reply. To answer your questions:

1) I have 8G mem 2)not sure how to rule out if that is sufficient or not. But as I said I ran an even larger sequence of .dat files prior and that worked. Currently I am attempting to process a folder w/ blk00209.dat.. 213 (5 ttl files) 3)I DL'd the 0.8.0 version as you suggested and am getting same results as w/ the 0.8.2 4)I tried the export options and cargo clean and re-built the release ver as you suggested also

just to re-iterate this is in win10 cygwin which worked the first time for me, then I upgraded to WSL w/ native ubuntu and all hell broke loose, so reverting to cygwin now, but can't say there is causality there or not.

Also to clarify the prob as I see it is that rusty is not seeing any blocks of data worthy of processing, as it terminates almost immediately with this output:

[12:27:53] INFO - main: Starting rusty-blockparser v0.8.0 ... [12:27:53] INFO - index: Reading index from k:/Temp/rustyin\index ... [12:27:53] INFO - index: Got longest chain with 0 blocks ... [12:27:53] INFO - blkfile: Reading files from k:/Temp/rustyin ... [12:27:53] INFO - parser: Parsing Bitcoin blockchain (range=209..) ... [12:27:53] INFO - callback: Using balances with dump folder: k:/Temp/rustyout ... [12:27:53] INFO - parser: Done. Processed 0 blocks in 0.00 minutes. (avg: 0 blocks/sec) [12:27:53] INFO - callback: Done. Dumped 0 addresses.

And further I know rusty is finding the files because if I empty out the rustyin folder or point it to any invalid source file path I get these results: $ ./target/release/rusty-blockparser -d k:/Temp/rustyin -s 209 balances k:/Temp/rustyout [12:36:47] INFO - main: Starting rusty-blockparser v0.8.0 ... [12:36:47] INFO - index: Reading index from k:/Temp/rustyin\index ... [12:36:47] INFO - index: Got longest chain with 0 blocks ... [12:36:47] INFO - blkfile: Reading files from k:/Temp/rustyin ...

thread 'main' has overflowed its stack

Thanks much for any clues as rusty is quite a nice tool, Jim

On Tue, Dec 27, 2022 at 4:57 AM Michael Egger @.***> wrote:

How much memory do you have and can you rule out running out of memory?

Also consider issuing cargo clean and build it again and always run the release builds as the debug builds are very slow and the stack might overflow sooner as no optimizations are applied.

You also might want to use this branch if you are using it for bitcoin to rule out any bugs: https://github.com/gcarq/rusty-blockparser/tree/use-bitcoin-lib.

If nothing of the above helps, you might want to increase the stack size manually to see if that fixes it, .e.g.:

$ export RUSTFLAGS="-C link-args=-Wl,-zstack-size=4194304" $ cargo build --release

— Reply to this email directly, view it on GitHub https://github.com/gcarq/rusty-blockparser/issues/86#issuecomment-1365770458, or unsubscribe https://github.com/notifications/unsubscribe-auth/A45FV53JTVPBDIK7NVNXRHDWPK4Q7ANCNFSM6AAAAAATIWCLHA . You are receiving this because you authored the thread.Message ID: @.***>

[jrp27514]

Hi, 2 more comments: the fact that I see "thread 'main' has overflowed its stack" on bad pathnames seems like a surprising result. Is that really the expected output ?

This morning I also installed centos 8 on a laptop and followed crate.io to get cargo and rust (and had to add addition build tools due to linker not found err) but built and ran rusty in this pure centos env and still getting 0 Addresses found. So I am either missing some crucial setup step or something else is wrong

Thanks

On Tue, Dec 27, 2022 at 4:57 AM Michael Egger @.***> wrote:

How much memory do you have and can you rule out running out of memory?

Also consider issuing cargo clean and build it again and always run the release builds as the debug builds are very slow and the stack might overflow sooner as no optimizations are applied.

You also might want to use this branch if you are using it for bitcoin to rule out any bugs: https://github.com/gcarq/rusty-blockparser/tree/use-bitcoin-lib.

If nothing of the above helps, you might want to increase the stack size manually to see if that fixes it, .e.g.:

$ export RUSTFLAGS="-C link-args=-Wl,-zstack-size=4194304" $ cargo build --release

— Reply to this email directly, view it on GitHub https://github.com/gcarq/rusty-blockparser/issues/86#issuecomment-1365770458, or unsubscribe https://github.com/notifications/unsubscribe-auth/A45FV53JTVPBDIK7NVNXRHDWPK4Q7ANCNFSM6AAAAAATIWCLHA . You are receiving this because you authored the thread.Message ID: @.***>

[gcarq]

Hi, 2 more comments: the fact that I see "thread 'main' has overflowed its stack" on bad pathnames seems like a surprising result. Is that really the expected output ?

I never tried to run it on bad pathnames, so I can't say for sure.

So I am either missing some crucial setup step or something else is wrong

How are you downloading the bitcoin blockchain? Are you sure block pruning is disabled on your bitcoin node? Can you rule out any permissions issues? e.g.: Bitcoin running as root in a docker container and thus a different user cant see the blockchain resulting in no files found? Please also post the debugging output (passing -v multiple times to rusty-blockparser)

[jrp27514]

Thanks,

turns out user error is the prob. I failed to understand the importance of the index files. Because rusty lets you specify a specific start and end blk.dat file I assumed the index folder was not that relevant. If I point rusty to a working folder w/ valid index files it works. I was trying to separate my working stuff from the live bitcoind files

Thanks for asking the hard questions

On Wed, Dec 28, 2022 at 8:44 AM Michael Egger @.***> wrote:

Hi, 2 more comments: the fact that I see "thread 'main' has overflowed its stack" on bad pathnames seems like a surprising result. Is that really the expected output ?

I never tried to run it on bad pathnames, so I can't say for sure.

So I am either missing some crucial setup step or something else is wrong

How are you downloading the bitcoin blockchain? Are you sure block pruning is disabled on your bitcoin node? Can you rule out any permissions issues? e.g.: Bitcoin running as root in a docker container and thus a different user cant see the blockchain resulting in no files found? Please also post the debugging output (passing -v multiple times to rusty-blockparser)

— Reply to this email directly, view it on GitHub https://github.com/gcarq/rusty-blockparser/issues/86#issuecomment-1366663388, or unsubscribe https://github.com/notifications/unsubscribe-auth/A45FV52DRAIWBVVCOUFUCRTWPQ73PANCNFSM6AAAAAATIWCLHA . You are receiving this because you authored the thread.Message ID: @.***>

[jrp27514]

Michael, a quick usage question please (am using 0.8.2)

I'm a bit puzzled at results of: A) ..balances -s 0 -e 100000 B) ..balances -s 0 -e 100

I would expect 1st 100 entries of both csv outputs to match but they don't. What am I not understanding?

Thanks, Jim

On Tue, Dec 27, 2022 at 4:57 AM Michael Egger @.***> wrote:

How much memory do you have and can you rule out running out of memory?

Also consider issuing cargo clean and build it again and always run the release builds as the debug builds are very slow and the stack might overflow sooner as no optimizations are applied.

You also might want to use this branch if you are using it for bitcoin to rule out any bugs: https://github.com/gcarq/rusty-blockparser/tree/use-bitcoin-lib.

If nothing of the above helps, you might want to increase the stack size manually to see if that fixes it, .e.g.:

$ export RUSTFLAGS="-C link-args=-Wl,-zstack-size=4194304" $ cargo build --release

— Reply to this email directly, view it on GitHub https://github.com/gcarq/rusty-blockparser/issues/86#issuecomment-1365770458, or unsubscribe https://github.com/notifications/unsubscribe-auth/A45FV53JTVPBDIK7NVNXRHDWPK4Q7ANCNFSM6AAAAAATIWCLHA . You are receiving this because you authored the thread.Message ID: @.***>

[gcarq]

Example A) creates a balance snapshot up to block 100000 and example B) up to block 100. If someone made a transaction in block 101 that e.g. transfers all balance to a new account it would be reflected in example A) but not B).

[gcarq]

I'm closing this as the initial path issue has been solved. Feel free to open a new issue

[jrp27514]

Michael,

I happened to see a video where rusty used to accept a thread parameter -t but doesn't seem to advertise that option now. Any idea how to use more cores/threads ??

Thanks

On Fri, Dec 30, 2022 at 8:49 AM J.R. Pauley @.***> wrote:

Michael, a quick usage question please (am using 0.8.2)

I'm a bit puzzled at results of: A) ..balances -s 0 -e 100000 B) ..balances -s 0 -e 100

I would expect 1st 100 entries of both csv outputs to match but they don't. What am I not understanding?

Thanks, Jim

On Tue, Dec 27, 2022 at 4:57 AM Michael Egger @.***> wrote:

How much memory do you have and can you rule out running out of memory?

Also consider issuing cargo clean and build it again and always run the release builds as the debug builds are very slow and the stack might overflow sooner as no optimizations are applied.

You also might want to use this branch if you are using it for bitcoin to rule out any bugs: https://github.com/gcarq/rusty-blockparser/tree/use-bitcoin-lib.

If nothing of the above helps, you might want to increase the stack size manually to see if that fixes it, .e.g.:

$ export RUSTFLAGS="-C link-args=-Wl,-zstack-size=4194304" $ cargo build --release

— Reply to this email directly, view it on GitHub https://github.com/gcarq/rusty-blockparser/issues/86#issuecomment-1365770458, or unsubscribe https://github.com/notifications/unsubscribe-auth/A45FV53JTVPBDIK7NVNXRHDWPK4Q7ANCNFSM6AAAAAATIWCLHA . You are receiving this because you authored the thread.Message ID: @.***>

[gcarq]

This is no longer possible, as this had a major flaw and has been removed as it was no longer relevant in 0.7.0.

Please create a new issue if you have a question, that helps me and other people who have the same problem and allows collaboration. This is not a support thread. Thanks