search

cosme12 / SimpleCoin

Just a really simple, insecure and incomplete implementation of a blockchain for a cryptocurrency made in Python as educational material. In other words, a simple Bitcoin clone.
http://copitosystem.com
MIT License
1.78k stars 397 forks source link

Move blockchain out of memory #7

Open cosme12 opened 6 years ago

cosme12 commented 6 years ago

As blockchain gets bigger, it will be necessary to be moved out of memory. Since this is a simplified version of a blockchain, saving it in a text file may be enough.

kennethgoodman commented 6 years ago

feather and pandas should do it if we only allow one input to one output transactions.

For more complex transactions, json with a fast json library: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-json-libraries/

rdenadai commented 6 years ago

Perhaps TinyDB could be a option? If you don't need concurrent access to the file...

http://tinydb.readthedocs.io/en/latest/

Just store the blockchain in it as a json format...

Aareon commented 6 years ago

I would suggest a combination of SQLite/PostgreSQL and SQLAlchemy. Preferably Postgres in production.

I’ve never personally used Feather or Pandas, but I have seen the combo that I’ve suggested used in production by other block chains, and it works more than well.

In my opinion, I don’t believe TinyDB or any flat file option would be efficient enough.

rdenadai commented 6 years ago

@Aareon as i said, if concurrent access is not necessary...

I use TinyDB in a project, but since then i need concurrent access to my database so i change it to rethinkdb.

It was a simple migration between both of the, by simple rewriting my DAO class...

Postgresql + SQLAlchemy are always welcome but it brings a new layer of complexy to the project which main goal claims to be simple...

Aareon commented 6 years ago

Neither one of those solutions are ACID compliant. I don’t think we want to use something that has a chance of losing data. It may be simpler, but does not by any means mean better. My solution may be more complicated, but it will be much more reliable. If you’re okay with losing data in your ledger, be my guest.

HourGlss commented 5 years ago

Right now I can mine between 33-36 blocks, depending on how I choose to handle the Pipes. The limiting factor is the Pipe though, not the size of the blockchain in memory. do a test and don't send information around in pipes. I can handle thousands of them, with pipes, less than 40.

HourGlss commented 5 years ago

I fixed it using Queue and restructuring the way that multiprocessing works

HourGlss commented 5 years ago

Implemented a blockchain class which utilizes a sqldatabase https://github.com/hourglss/hourcrypto WORK IS ONGOING, PROJECT IS CURRENTLY nonfunctional.