Cannot load any results

pouyiouk commented 4 years ago

I have been facing an issue for the past two days, trying to actually use the tool. I did follow the instructions exactly as stated in the readme. Several issues: The backend loads fine on port 3000 of localhost but it is empty. If I try to:

bundle exec rails utils:load_test_data I get as a response: rails aborted! Missing environment variables
If I do try the single transaction or a selection of blocks commands i.e: bundle exec rails utils:scan_blocks BTC_USERNAME=user BTC_PASSWORD=password FILES_PATH=~/bitsee/public/files/ FILES_URL=/files BLOCK_BEGIN=230037 BLOCK_END=230100, I do get a different response: rails aborted! Errors in response
And to actually confirm that I do have some sort of connection, if I do supply wrong credentials in the BTC_USERNAME and BTC_PASSWORD fields, I do get a response from the RPC client of Bitcoin: ThreadRPCServer incorrect password attempt from 127.0.0.1:33154

So what might be the issue that I am facing here? Is this something with the working environment and incompatibility with updated gems or something I am missing out?

pouyiouk commented 4 years ago

The problem was with Ubuntu 20.20, after installing a Xenial flavour of Ubuntu everything works much smoother without the deprecated warnings. So the initial problem was environment wise. I am still facing some issues though. Now I can proceed and scan a block but I do still get the same error as before. "Errors in Response".

dewyatt commented 4 years ago

I am still facing some issues though. Now I can proceed and scan a block but I do still get the same error as before. "Errors in Response".

I haven't touched this project in quite a while but it still works for me using the jeshan/bitcore-node daemon with the following bitcoin.conf:

server=1
whitelist=127.0.0.1
txindex=1
addressindex=1
timestampindex=1
spentindex=1
zmqpubrawtx=tcp://127.0.0.1:28332
zmqpubhashblock=tcp://127.0.0.1:28332
rpcallowip=127.0.0.1
rpcuser=bitcoin
rpcpassword=local321
uacomment=bitcore

Note the extra indexes that have been enabled.

"Errors in Response" comes from here, you may try adding some Rails.logger.info calls to debug that but I may not be able to provide much support right now.

pouyiouk commented 4 years ago

I have tried the bitcore-node daemon with the bitcoin.conf parameters you specified above, but still getting the "Errors in Response". I will try the logger.info calls to try and debug it. I will update accordingly. Thank you for the response.

dewyatt commented 4 years ago

@pouyiouk I had time and pushed a small change to log those errors (typically in log/development.log). Some other things to check:

Is the bitcore node fully synced? That could take a few days at least.
Is the bitcore node using livenet? During startup, it should say info: Using network: livenet.

pouyiouk commented 4 years ago

I can confirm for both that the full node has synced and running using the -rpc flags on livenet. Should I specifically specify the flag on the command? It is running on livenet though. bitcoind -datadir=/bitcoin_0.20/Bitcoin_Chain/Bitcoind_data_dir/ -server -rpcuser=user -rpcpassword=password -rpcserialversion=0

I will update to latest version and give it a go again and see what those errors are. Thank you for the help!

pouyiouk commented 4 years ago

After looking at the log file, the error outputs the following error: Error: No such mempool transaction. Blockchain transactions are still in the process of being indexed. Use gettransaction for wallet transactions. Error: The genesis block coinbase is not considered an ordinary transaction and cannot be retrieved

I can confirm though that everything has synced from the terminal window. I might be able to fully understand now what might be the issue.

Update 1: After changing the block count from 0 to 1 is currently scanning the chain until the current head.

Update 2: I do get the following though during the scanning of blocks: 7dbd3f2a711350eea19f95fa6e4ea78b01adfd669e2627158c4e6be1093f43c0 Invalid encoding, skipping No files have been generated or shown yet on the backend server.

Update 3: It scanned the complete blockchain but no results have shown on localhost - rail server is running. The page is loaded but it is empty. Some blocks returned the message aaf00549d60fd8405157b37daca2b971e07f48c5aaa606a2445eb5e89d6fe752 ******* Potentially interesting ******* application/x-cpio; charset=binary

but apart from that no file extraction nor any updates have been shown on localhost:3000 page.

Update 4:

I was able to load results by scanning known transactions that contain data in them. Tool works as intented. Is there a way to scan all transactions and identify which ones hold the data and for them to be extracted directly instead of going manually one by one?

dewyatt commented 4 years ago

@pouyiouk Sorry, I didn't get any notifications on your updates here. I should have some time to look soon.

dewyatt commented 4 years ago

@pouyiouk You should be able to scan the chain with something like:

$ export BTC_USERNAME=bitcoin BTC_PASSWORD=local321 FILES_PATH=~/bitsee/public/files/ FILES_URL=/files
$ bundle exec rails utils:scan_blocks BLOCK_BEGIN=416735 BLOCK_END=416835

That should discover 111 secrets, including a jpeg and a png:

$ bundle exec rails console
irb(main):001:0> Secret.count
   (0.1ms)  SELECT COUNT(*) FROM "secrets"
=> 111

Does that not work for you?

EDIT: To scan the entire chain, something like bundle exec rails utils:scan_blocks BLOCK_BEGIN=1 should work.

dewyatt commented 4 years ago

For reference, the following sequence of commands worked on an Ubuntu 16.04 AMI in EC2:

sudo apt update
sudo apt -y install gcc g++ make automake autoconf libtool libbz2-dev libsqlite3-dev zlib1g-dev libmagic-dev docker.io ruby ruby-bundler ruby-dev nodejs
sudo mkdir /root/bitcore
sudo docker run --env BITCOIN_NETWORK=livenet --restart=unless-stopped -d --network=host -v /root/bitcore:/root/bitcoin-node/data --name livenet jeshan/bitcore-node
# ...wait until bitcore daemon has synced

git clone https://github.com/dewyatt/bitsee
cd bitsee
bundle install
bundle exec rails db:migrate
export BTC_USERNAME=bitcoin BTC_PASSWORD=local321 FILES_PATH=~/bitsee/public/files/ FILES_URL=/files
bundle exec rails utils:scan_blocks BLOCK_BEGIN=1

pouyiouk commented 4 years ago

The backend does not show any files or text when scanning blocks. The above command does work and it iterates +1 after each block is done. But no files shown in the page.

If I do however scan transactions individually I do get the files on the backend. So to do that I run bundle exec rails utils:scan_tx TX=4b0cd7e191ef0a14a9b6ab1c5900be534118c20a332ff26407648168d2722a2e

pouyiouk commented 4 years ago

The command does work when scanning blocks, but I do not see the results on the backend. To see results I need to scan individually each transaction that is known to contain data.

dewyatt commented 4 years ago

@pouyiouk I'm not sure I can help without further information, maybe a screencast recording or something. The sequence of commands I posted above scans the entire blockchain and the results are correctly generated on the backend and displayed on the frontend.

It works on a fresh Ubuntu 16.04 installation every time, so I don't know if it is something with your specific installation or if there is a misunderstanding somewhere.

I'm assuming you have no local modifications to the code too.

dewyatt / bitsee

Cannot load any results #9