Improve document sharing across Rein network

[weex]

The goal of this job is to improve document sharing across the Rein network, specifically by improving upload logic in the client.

Rein has server and client components which exchange signed documents. Currently a client will only upload its own documents but to enable the network to be more global, I want clients to share downloaded documents to other servers.

This task is done when: 1) the client uploads all valid documents it knows about to each of the servers to which it's connected and 2) when a client requests documents from a server, the server does not send duplicates even though the same doc may have been uploaded by more than one client.

Note, this job is posted in Rein with job id: hemtujcmkfthj8w56xjn but ZenifiedFromI2P is working on it voluntarily. If you're interested to help develop for Rein, please contact @weex in #Rein on freenode irc.

[ZenifiedFromI2P]

I think SQLAlchemy is useless complexity for this, for signed documents, a key-value storage would be better.

[weex]

Sqlalchemy is a bit tricky but I believe it's better than raw queries. As for the relational vs nosql question, I'm open to seeing what nosql would look like but have much more experience with mysql / sqlite. Thinking in tables seems much more natural to me than unstructured documents.

[ZenifiedFromI2P]

Use JSON for serialization?

On 12/21/2016 10:03 PM, David Sterry wrote:

Sqlalchemy is a bit tricky but I believe it's better than raw queries. As for the relational vs nosql question, I'm open to seeing what nosql would look like but have much more experience with mysql / sqlite. Thinking in tables seems much more natural to me than unstructured documents.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ReinProject/python-rein/issues/75#issuecomment-268653648, or mute the thread https://github.com/notifications/unsubscribe-auth/AVMcH8p9l_hZU_tzDzofVxyA9COU_5p1ks5rKaJLgaJpZM4LRJvC.

[weex]

Yes, we definitely need to switch to json instead of this "Keyword:" and newline format. This requires a standardized serialization method that allows contents to be signed and not malleable. It would help lay the groundwork for encryption and better formatting.

[ZenifiedFromI2P]

Well, I think a better solution will be to use MsgPack (more efficient and compact)..

A better alternative to SQLAlchemy will be to put a centralized index key and a set of key-value stores in a key-value storage like LMDB..

But it may require breaking changes and redesign of the DB layer from scratch.

[alexanderdushkin]

I personally like MongoDB, but introducing it would require all users to have a local MongoDB installation. However, Mongo is all about documents, so it would be really easy to use it.