search

stellar / stellar-protocol

Developer discussion about possible changes to the protocol.
525 stars 303 forks source link

IPFS support #32

Closed nullstyle closed 3 years ago

nullstyle commented 8 years ago

Motivation

Adding support for natively addressing objects stored within ipfs (https://ipfs.io/) would allow, in my opinion, many novel systems that use the combined power of ipfs's "permanant web" and the Stellar network's "financial web".

Ipfs uses multihash (https://github.com/jbenet/multihash) to address objects and peers, and adding support to the stellar memo union would allow it to natively address objects stored within ipfs.

Detailed Design

Current Draft Proposals

jedmccaleb commented 8 years ago

If the intent is that this memo is referencing something in IPFS, I think the memo should be explicit about that and we call it MEMO_IPFS otherwise you still have the question of where to get the preimage of the MULTIHASH.

Unless the idea is to add MEMO_MULTIHASH and then when people are storing their memo's in IPFS they change their homedomain to "IPFS" or something to tell people where to look up the memo.

sunny-g commented 8 years ago

Is there a reason that this should be baked into the stellar protocol as opposed to being implemented as a 3rd-party protocol on top of memos?

For example, all IPFS hashes start with Qm..., though I know that not all hashes that start with Qm... have a preimage in IPFS.

nullstyle commented 8 years ago

@jedmccaleb, yeah MEMOIPFS would work for me as well. In the future we could add other MEMO* discriminants to support other projects that might use multihash.

@sunny-g I don't think we should be building implicit procotols on top of stellar. I'm happy to discuss ways that we might support good integration with ipfs outside of the core protocol, but using something as implicit as a string prefix to determine processing behavior would be a very poor recommendation, IMO. Furthermore, base58 encoded hashes would not fit into our current length constraints for MEMO_TEXT memos (meaning we would have to extend the allowed length to support ipfs).

My recommendation for adding multihash support to our XDR schema would be to specifically provide a strong, explicit integration point for other projects/protocols that use multihash.

sunny-g commented 8 years ago

@nullstyle logical enough, it seemed to me to be out of the scope of the stellar protocol, but that's not for me to decide! Is there an implementation timetable/roadmap for proposed protocol changes?

nullstyle commented 8 years ago

@sunny-g It's certainly possible it could be out of the scope of our core protocol! That's one of the reasons I opened the issue here :) I need to think through other ways that we might integrate some more, but given the current network's capabilities I have yet to find a clean integration.

IMO, we could potentially side-step a ton of complication for integrators by having a good integration story for IPFS. The "message server" protocol (and the complications with how it interacts with the memo system) in particular seems like something that could be avoided by supporting ipfs.

nullstyle commented 8 years ago

As far as timeline goes, I haven't even begun to think about that... I'd first like to see if this is something that is interesting to integrators and other developers at large

jedmccaleb commented 8 years ago

@nullstyle "The "message server" protocol (and the complications with how it interacts with the memo system) in particular seems like something that could be avoided by supporting ipfs." really? isn't it kinda the same? You are either fetching the preimage from a webserver or from IPFS.

nullstyle commented 8 years ago

@jedmccaleb The point is about re-using knowledge. For example, you could replace all of the specifications for retrieving the pre-image of a memo with:

I don't think it would replace the message server protocol, but it could provide a simple alternative.

jedmccaleb commented 8 years ago

Right I guess you can skip homedomain and the stellar.toml part so it is simpler.

bartekn commented 8 years ago

I don't know IFPS but I watched the Intro video on their website and it seems that sometimes there is a lag when downloading data (like a few seconds). So we definitely should check latency and reliability before implementing this because some services need to be fast and reliable.

nullstyle commented 8 years ago

@bartekn IMO that shouldn't factor much into the discussion, as it doesn't really factor much into the discussion when it comes to using the other memo types. The logical equivalent of "ipfs is unreliable" is "the message server containing the memo preimage is down" and for "ipfs has latency" is "the message server has latency".

What concerns do you have?

bartekn commented 8 years ago

Actually, I like the idea. I simply don't know if IPFS is production ready. For example, what if there are no nodes that host a specific file at the moment or what if there's only one node that is overloaded and you have to wait 30 seconds for a small JSON? On the other hand you can quite easily build a simple service (like message server) with a guaranteed uptime using AWS or other cloud providers.

nullstyle commented 8 years ago

Cool, good to know.

One of the cool things about ipfs is that any member of the network can easily rehost data to get better availability. Someone could write a simple daemon that watches for MEMO_IPFS transactions and then simply runs: ipfs pin <MEMO_HASH> and their server would then download and keep that object available for others to retrieve.

bartekn commented 8 years ago

Pinning sounds great. I will check IPFS more this weekend.

MikeFair commented 7 years ago

+1 on more ipfs integration.

I'd like to see taking it further and run ipfs at every Stellar node that makes sense by installing it in the Docker image; certainly running it for every horizon server. Then create an ipfs cluster out of all those nodes and "pin" any ipfs addresses Stellar considers "required" to the Stellar cluster. Substitute ipfs for any other preferred distributed P2P file/content sharing system.

Then do some registration work to allow fetching Stellar.toml files from ipfs instead of only DNS/HTTP. Probably by using the Stellar.org IPNS namespace services.

The fact Stellar is using IPFS for this part behind the scenes would remain hidden from the protocol. (The protocol users don't really see what happens when they do "some_id *federation_id".)

... If MEMO_IPFS is a good idea then so is MEMO_HTTP; which then brings up MEMO_FTP/SCP/TORRENT. Same thing goes for ManagedData on Accounts.

What about in place of MEMO_IPFS, add a new record type ATTACH (same 64 byte value size). ATTACH also provides "name" and "protocol" fields. The convention is ATTACH contains a referral address to an outside system; MEMO contains raw data directly on the record.

maximgx commented 6 years ago

So much hype around stellar + ipfs, but you can't actually attach a multi-hash to the transaction after years. So much use cases need this NOW

MikeFair commented 6 years ago

While it's not native, there are two ways to do it I know of. Neither is perfectly what you'd want, but they work for special cases.

1) Use a MEMO_HASH and store the "Hash" instead of the "IPFS MultiHash" which means stripping off the encoding and the algorithm bits to get the raw hash. While losing the encoding data isn't so bad (because the ledger is defining a binary encoding value for the data), losing the algorithm information is a big loss because IPFS expects to be hash algorithm agnostic. Somehow restoring the algorithm byte to the data would be a big win, but at that point the hash information is variable and not gauranteed to be a sha256 or 32 byte value anymore.

2) You don't attach IPFS records to the transactions, but instead attach them to an account in its custom data, and you put the account id in the memo field. You then use the Transaction Hash as the "data key" and the IPFS CID as the data. This makes an account into a kind of directory hosting many attachments.

3) Use multiple MEMO_TEXT entries. Use a prefix identifier to sort the MEMO entries into an order when "reading the CID" back out.

There are obviously other schemes, and IPFS isn't the only p2p decentralized content distribution network out there. It'd be great to see a generic "MEMO_MULTIHASH" or "MEMO_ATTACH" that could take data maybe up to 256 bytes?

jedmccaleb commented 6 years ago

Someone should turn this into a more general SEP about interfacing generically with IPFS

brianebert commented 5 years ago

All,

Thanks for this discussion. @nullstyle, thanks for getting it started.

I’d like semantic support for the sha256 hashes used by IPFS right now. Stellar doesn’t have to change memo storage size, and nobody has to trust a third party for preimage location and hash format. Integrators’ lives also become slightly easier for bringing the hash transformation into Stellar.

Per @jedmccaleb I’ve submitted a pr for an SEP on this. Jed, I apologize in advance for not writing an SEP general to the IPFS multihash, as you may have suggested in your last comment. I’m happy to do so if people seem motivated to support the a multihash spec.

I’ve changed the Memo_IPFS type designator above to MemoIPFS_Qm in hopes that some day there will be a lot of other MemoIPFSs, and Memo_IPFS can be used to refer to their union. Hopefully that day will support a true multihash.

@bartekn, your reservations regarding IPFS latency are a matter of operations. You are correct, when an IPFS hash is requested, if the requested node doesn’t have a copy, it can take a long time to find the content. The good news is that once the node fetches its copy it keeps it, so the next request is served with similar latency to a database query. Adding a Stellar memo type to key on makes it easier to watch for transactions referring to IPFS and cache the documentation near the blockchain access. An IPFS memo type may encourage IPFS developers toward Stellar.

So…how about it?

brianebert commented 5 years ago

Hey all,

I just noticed something. I was working with IPFS and Stellar, then I went on a vacation, had my retina detach, and it’s been a long process starting to function again. After visiting this thread and writing the SEP I returned to IPFS docs, to find a new addressing version.

This obviously complicates the question of IPFS support. It looks like they’re going to continue supporting the current hashes, v0, so for now I’m going to keep using them, but I don’t feel as bold about pressing Stellar or anyone else to make this better. I would really have liked consensus for a trustable preimage and if people are interested I’d love to help, but…

If anyone else has a favorite distributed, content addressable, torrent based data store I’d love to hear.

Thank you.

brianebert commented 5 years ago

Sorry for another post on this, but these content identifier (CIDs) simplify handing IPFS hashes.

There is code available that parses multihashes from multi base strings, as well as ingesting binary. Hashes could be accepted by Stellar or not, depending upon whether they fit into available Memo storage. Any 256 bit hash could be handled without changing Memo storage allocation. Metadata would be written by selecting automatically from a (hidden) cannon of IPFS Memo.type supported..

Is there any appetite for this for Stellar? I can revise that SEP fo CIDs. Gladly.

johansten commented 5 years ago

As much as I love IPFS I don't think it's a good idea to add native support for CIDs/multihashes, because I don't see why IPFS should get preferential treatment over any other content-addresseable storage scheme.

A hash is a hash. IPFS mostly use SHA256, and the raw hashes fit right into a Memo.hash.

MikeFair commented 5 years ago

Because multihash is a hash agnostic storage scheme. It's got all the details required to support any hash.

What that hash goes to is app determined.

And data hash algorithms, over long lived periods of time (which a blockchain has), change.

In the future we might have hash algorithms implemented directly on the MMU chips where we just load up a few pointers and lengths and a clock cycle later, get a pointer to the value back; but it won't be a SHA256 hash it will be some FPH1024 (Future Perfect Hash length 1024 bits); or some other different hash algorithm (SHA3 is looking promising these days).

Multihash gives you the ability to understand what Hash Algorithm you are looking at. Its type, and how to compute it, is not assumed. Which is why IPFS invented the Multihash scheme; to deal with the fact that different blocks in their system are going to use different hash algorithms and have to work across the system at the same time. The fact IPFS is all SHA2-256 right now is more just a function of being at an early development stage.

Supporting MultiHash isn't saying support IPFS per se; it's saying "Things that use transactions on the blockchain might want to store a hash about their own application data that went into this transaction because they can't/shouldn't put the data itself onto the chain."

IPFS is just one such application. Reciepts or Work Orders in a Point of Sale or Purchase Order system might be another. And those applications might also change Hash algorithms in their own data over time too.

MikeFair commented 5 years ago

@nullstyle, @jedmccaleb I just reread the entire thread and echoing the last exchange @johansten and I just had; I'm strongly inclined to agree with @nullstyle's initial Title "MEMO_MULTIHASH" support.

@nullstyle What do you think of changing the MEMO struct to implement the pieces/parts of IPFS' MultiHash v1 scheme? Namely:

I don't think this overall proposal is really as much about IPFS as it is "BlockChains and their Apps make use of hashes"; lots of them, and IPFS is simply one such app many of us are pointing to for utility.

@johansten's comment, along with allusions by others before it, is correct; Stellar doesn't need to promote IPFS or integration with IPFS directly. IPFS, is like many apps, it uses hashes, there are many different apps using many different hash algorithms.

Supporting a MEMO that can identify what style/type/size/encoding of hash that is being MEMOd onto the transaction is a really nice thing for Stellar to do and makes apps lives easier by not having to hard code exactly what kind of data is in the hash and can do some sanity tests to make sure it is looking at what it expects to be looking at. Without a type differentiation, all 32-bit hash values are valid for all 32-bit algorithms and there's nothing in the data to distinguish to an app things have changed in the future.

As part of my work with GraphQL I actually wrote a query that could retrieve a transaction by its MEMO_HASH because that was something my app had access to from a third party system more easily than finding the txn hash. The query was "Please locate, if it exists, the Stellar Transaction we did that correlated to this internal system transaction; if not found, execute the required Stellar Transaction."

Being able to fetch txns by MEMO_ from Horizon would be awesome (Though I know this query can happen with or without this MEMO_MULTIHASH support).

Early on @nullstyle asked "should we support hashes longer than 64 bytes?" I say yes, this would be nice to have; my instinct is 256 bytes (I guess four 64 byte values?).

I don't think it should be arbitrary length or people will store files in them. However, 64 bytes is 512 bits, and a 1024-bit public key isn't that unusual, so supporting up to 2048 bits (256 bytes) seems like it should hold people's Hashing storage needs for a very long time. That includes using it for other things like suboptimal binary encodings (like Hex), extra space to provide more context ("ipfs:"), or combining a short array of hashes into a single field [64-bit, 64-bit].

Mike

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 30 days with no activity. It will be closed in 30 days unless the stale label is removed, or a comment is posted.