johannesmols
commented
4 years ago Hi @benitoSan, thanks for your message. I'm happy to hear you are interested in this project. It does exactly what you are describing. Essentially, this project can extract all OP_RETURN transactions from a local Bitcoin copy and do some basic analysis on them (such as average payload length, protocols used, ...). On top of the analysis part, all payload of the transactions are stored as well. You can read the paper I published about this project here.
To start off, you will want to download the BitcoinRpcMiner (all relevant code is in in the organization). You will also want to install Bitcoin Core and start downloading the entire blockchain. Make sure to run it with the parameter -txindex so that all non-wallet transactions are indexed as well. In your config file of Bitcoin Core, open the RPC and set a username, password, and port. You will have to enter that information in the config.py file afterwards.
You will also have to install a database, I used MS SQL Server. Create a database using the script in the "sql scripts" folder, and put the connection information in the config.py as well.
When you run the app.py file (install the required packages using "pip install -r requirements.txt" first), it will connect to the Bitcoin Core RPC and database, and start parsing all blocks from the beginning. This will take a long time (~2 weeks on a fast computer). You can always interrupt the script and at the next restart it will figure out by itself where it left off. Once it parsed the entire blockchain it will continue running and check for new transactions continously. The database is ~40 GB in total.
This project here is an API that allows external services to query the database for specific information. The third project in the organization is a front-end application that visualizes the database to a user and allows them to search for specific transactions. But you can totally just work with the dataminer if you are only interested in extracting OP_RETURN transactions to a database.
Good luck and let me know if you have any issues :)
Excellent job you did with this project! I'm doing a research for an university project and I would like to know more about this project you did. I need to extract transactions that used OpReturn by blocks and from what I saw this project does exactly that, correct? If so could you share some tips on how to run the program?