search

ipfs / js-ipfs

IPFS implementation in JavaScript
https://js.ipfs.tech
Other
7.44k stars 1.25k forks source link

Basic spec compliance needed to get a node-ipfs running #13

Closed daviddias closed 8 years ago

daviddias commented 9 years ago

minimum spec

The go is to go through these modules, understand what stays, what leaves and what is missing compared to the go-ipfs. (probably will draw some ASCII art after I get this into my head, sometimes helps to think and communicate better)

candidate attack plan:

daviddias commented 9 years ago

@jbenet "candidate attack plan" from the meeting notes is ready :)

jbenet commented 9 years ago

@diasdavid candidate attack plan looks great to me!

daviddias commented 9 years ago

From @jbenet on IRC

we should whittle down assignments to one person (a single Directly Responsible Individual)

Agree, I've put myself doing "Get go-ipfs and node-ipfs talking", it will help pave the path for the remaining tasks and how to partition them.

jbenet commented 9 years ago

Great! :+1:

jbenet commented 9 years ago

we can probably assign the rest once we have a clearer scope. also, i think @krl began working on the node merkledag.

jbenet commented 9 years ago

For basic wire protocol interop, all we need is (lots of steps but each one is relatively small):

(could actually talk without spdy first)

jbenet commented 9 years ago

@whyrusleeping have time this week to take on any of the go things above? (part the spdy part?)

jbenet commented 9 years ago

@diasdavid @mappum @krl @travisperson -- what things from the wire interop list might you be able to take this week?

if we had a thing by the end of the week that spoke the wire protocol, it would be a great start.

jbenet commented 9 years ago

(and anyone else watching, feel free to help! \o/)

daviddias commented 9 years ago

After thesis delivery on Friday morning, my intent is to to focus entirely on:

I'll be online, so anyone that wants to work at the same time in this stuff, just let me know

mafintosh commented 9 years ago

@jbenet @diasdavid in multistream, is the newline included in the length? (if so i more or less already have that module done)

whyrusleeping commented 9 years ago

@jbenet i was going to work on the records ipns stuff, but i can do this instead.

jbenet commented 9 years ago

@mafintosh yes, the newline is included in the length. what's your module?

whyrusleeping commented 9 years ago

ill take on "write go-multistream and go-multistream-select (may be just one pkg)"

jbenet commented 9 years ago

@whyrusleeping edit my message and sign yourself up? + link to relevant issues there as they're made?

daviddias commented 9 years ago

@jbenet do we want multistream to be aware if the stream is binary/plain text? If we have, is the binary version already have a defined expected length? (To know how much bytes should be reserved for the length in the beginning)

jbenet commented 9 years ago

Multistream itself shouldn't care-- it just defines the header. As for the expected length, it's a varint, but for now the go impl uses 'encoding/binary', which bounds it to a uint64 maximum (larger varints are treated by the implementation as errors, so the functionality is there to grow, but we can do the easy safe thing for now).

Also I need to adjust the docs for multistream-select -- @whyrusleeping and I adjusted some things. (Made "ls\n" and "na\n" and ls output messages varint prefixed too)

@whyrusleeping imemented:

- https://github.com/whyrusleeping/go-multistream/

- https://github.com/whyrusleeping/mss-nc

So take a look at those meanwhile?

— Sent from Mailbox

On Fri, Jun 5, 2015 at 10:38 AM, David Dias notifications@github.com wrote:

@jbenet do we want multistream to be aware if the stream is binary/plain text? If we have, is the binary version already have a defined expected length? (To know how much bytes should be reserved for the length in the beginning)

Reply to this email directly or view it on GitHub: https://github.com/ipfs/node-ipfs/issues/13#issuecomment-109373286

jbenet commented 9 years ago

(Also we should make sure whether the encoding/binary varint is what we want. I'd like a flavor of varint that means "<127" is the same byte value (I.e. Highest order bit is continuation), so impls can check easily)

— Sent from Mailbox

On Fri, Jun 5, 2015 at 10:38 AM, David Dias notifications@github.com wrote:

@jbenet do we want multistream to be aware if the stream is binary/plain text? If we have, is the binary version already have a defined expected length? (To know how much bytes should be reserved for the length in the beginning)

Reply to this email directly or view it on GitHub: https://github.com/ipfs/node-ipfs/issues/13#issuecomment-109373286

daviddias commented 9 years ago

update:

I believe that in parallel to figuring out what is missing to full interop, I can start looking on having spdy on top of this, so then it is all of the goodness from routing layer and above :)

jbenet commented 9 years ago

@diasdavid

update:

great! :+1:

So this is for multistream-select specifically, right?

There's already an ack, but the ack "confirms the protocol name", and is also the same thing as starting a conversation, too:

nc localhost 9090
< /ipfs/QmdRK.../multistream-select/0.3.0  # i speak multistream-select/0.3.0
> /ipfs/QmdRK.../multistream-select/0.3.0  # ok, let's speak multistream-select/0.3.0
> /ipfs/QmVXZ.../ipfs-dht/0.2.3            # i want to speak ipfs-dht/0.2.3
< /ipfs/QmVXZ.../ipfs-dht/0.2.3            # ok let's speak ipfs-dht/0.2.3 -- in a sense acts as an ACK

If we added an ok\n message, we'd get:

nc localhost 9090
< /ipfs/QmdRK.../multistream-select/0.3.0  # i speak multistream-select/0.3.0
> /ipfs/QmdRK.../multistream-select/0.3.0  # ok, let's speak multistream-select/0.3.0
> /ipfs/QmVXZ.../ipfs-dht/0.2.3            # i want to speak ipfs-dht/0.2.3
< ok                                       # ACK.
< /ipfs/QmVXZ.../ipfs-dht/0.2.3            # ok let's speak ipfs-dht/0.2.3 -- supefluous ack? but wait, what if

Without this extra ACK, you could have a totally one-sided conversation -- meaning a push-stream that is not duplex, without needing to send anything back:

# example of me sending stuff to a silent listener
nc localhost 9090
> /ipfs/QmdRK.../multistream-select/0.3.0  # let's speak multistream-select/0.3.0
> /ipfs/QmVXZ.../ipfs-dht/0.2.3            # i'm going to speak ipfs-dht/0.2.3
> <dht-message>
> <dht-message>
> <dht-message>

# example of me dialing an endpoint that just outputs a message and exits
nc localhost 9090
< /ipfs/QmdRK.../multistream-select/0.3.0  # let's speak multistream-select/0.3.0
< /ipfs/QmBBQ.../time/0.1.2.            # i'm going to speak time/0.1.2
< 2015-06-07 14:32:11

Not sure if this makes it clear, hard to explain. Anyway, this makes it useful for protocols that are simple information sources/sinks, or aren't "online" (e.g. "offline protocols" are those where you dont need to maintain a synchronous connection). So a "consumer" of the protocol can just encounter a partial stream like:

/ipfs/QmPGP.../proof-of-retrievability/x.x.x
<proof-on-A>
<proof-on-B>
<proof-on-C>
<proof-on-D>

And the "consumer" can consume this without needing to talk synchronously with whoever the "producer" was.

oh dang, name clash. d-d-d-double name clash (https://www.npmjs.org/package/multistream).

We can also call this pkg https://www.npmjs.com/package/multistream-select or some variation on mss or whatever.

yay! \o/

but when writing back to pick the right protocol on the go side, it goes silent. Need to make the code more verbose to understand what I'm missing.

Hmm, interesting.

  • Also, from what I understood that was on the spec, it is (1)up to the sender to send the multistream/version it is looking for, when it connects to the client and not (2)up to the receiver to tell the client which multistream/version it expects after the client connects, right? go-multistream uses the second option.

Sorry, my mistake for not making this clearer. I didn't mean to imply a requirement for an ordering of "who states the protocol first".

I think that in order to support "one-sided conversations" it's up to both to find agreement. In some situations, you want the dialer to poke the server by opening a connection, and the server says "hey, I speak multistream-select/0.1.2, buddy", and have the client (who's dialing anyway) adjust to the server's demands. In other cases, the clients could be very dumb + constrained, so they'll just speak out one protocol, and the server can be lenient. We observe cases like this all over, so I sort of see this as a "semi-synchronous" protocol, where the "ls" and "na" interactive part is definitely for synchronous use, but you can use the whole thing without ever being online at the same time, or even hearing back:

# i'm just gonna write you a letter and drop it off somewhere.
/ipfs/QmdRK.../multistream-select/0.3.0
/ipfs/QmSML.../snail-mail/1.2.3
Hello dear,

I am Svetlana. I am of late widow of Prince Abu Mabu of Nigeria. 
I am in need of a business partner whom could recieve my dear
Prince's moneys in a bank in <COUNTRY> so that I may invest it for the
future of the Prince's charities. Of course, I will pay you dearly for
this service. I am also in need of a citizenship at <COUNTRY>. I am now
am now single.

Love,
Princess Svetlana Mabu

I believe that in parallel to figuring out what is missing to full interop, I can start looking on having spdy on top of this, so then it is all of the goodness from routing layer and above :)

Awesome! \o/

daviddias commented 9 years ago

Thank you @jbenet, that was indeed a good explanation! I've updated the module (and published it to make sure we don't loose that name), categorising the two modes of working as silent-broadcast and interactive-select (feel free to recommend better names :)). All the updates, tests and examples here: https://github.com/diasdavid/node-multistream

daviddias commented 9 years ago

ok, got what was the problem with the interop problems I was having between go and node multistream impls. It seems that sorribas impl of length-prefixed-message doesn't include a '\n', and since we changed the protocol to include it, well, it barfed :). I'll just add the '\n' myself to the buffer that is passed onto lpm when a write occurs and trim it for the reads :). On it :)

whyrusleeping commented 9 years ago

TODO:

daviddias commented 8 years ago

Oh, how time flies :) June 1, feels just like yesterday :)