Allowing users to announce new items in their share

[GoogleCodeExporter]

Code:

http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fclient%2Fdc.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fcommon%2Fcore.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fcommon%2Fstate.py
http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=23&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Flocal_config.py

Summary:

2 new packets:
WR - sync packet sent with every YR packet, which contains the whole "new
items" list
WN - broadcast packet sent every time someone announces a new item

3 new commands:
ihave [stuff] - announce to the network, ie. broadcast a WN packet
newstuff [args] - show the list of new items dtella currently knows about
notify <on | off> - whether dtella should tell the user when it receives
WR/WN packets

Original issue reported on code.google.com by infinity0x@gmail.com on 20 Dec 2008 at 10:19

[GoogleCodeExporter]

Original comment by jacob.feisley on 20 Dec 2008 at 10:30

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

er, core.py had a few bugs in. see diff r10-r24 instead:

http://code.google.com/p/cam-dc-dtella-patch/source/diff?old=10&r=24&format=unid
iff&path=%2Ftrunk%2Fdtella-cambridge-1.2.3%2Fdtella%2Fcommon%2Fcore.py

also, newitems_numlim in local_config should probably be reduced to 32, since 
large
UDP packets less likely to be sent successfully.

Original comment by infinity0x@gmail.com on 21 Dec 2008 at 7:01

[GoogleCodeExporter]

Some comments:

I've always said this sort of functionality belonged in a bot, because 
implementing
it efficiently in a distributed fashion is an order of magnitude more complex 
and
difficult.

Your effort is valiant, but overall it kinda makes me go "ewwww" because it 
violates
a lot of Dtella's inherent design assumptions.

- Broadcasts include an absolute 32-bit timestamp.  It is not correct to assume 
that
all nodes have synchronized clocks.
- On sync, you receive a full copy of the items list from every node, and 
ignore all
but the first.  This is very wasteful, and it's possible for one confused node 
to
mess everyone up.
- On sync, the node list is sent immediately after the regular sync reply, but 
IP
doesn't guarantee that the packets won't be reordered or dropped.

In terms of style:
- Most lines exceed 80 columns.
- The use of map/filter in Python is generally discouraged.

An alternative, but much more predictable, way to implement this sort of thing 
is:
- Make every node in charge of its own (smallish) list of new items, and 
include this
personal list within every Sync Reply (YR) and Node Status (NS).  
(Unfortunately,
those packets aren't extensible without breaking compatibility... my fault, 
sorry.)
- When a node displays the list of new stuff, it should merge items together 
from all
the active nodes.
- When a node goes offline, its list of items disappear with it.

But, ultimately, I believe in keeping the mainline Dtella as simple as 
possible. 
This sort of "keep track of global non-critical state" thing would be best done 
using
a simple bot attached to the network.

Don't take this personally, and feel free to continue arguing your case.

Original comment by sparkm...@gmail.com on 5 Jan 2009 at 1:17

[GoogleCodeExporter]

> It is not correct to assume that all nodes have synchronized clocks.

Yeah I know, but this feature isn't time critical, which is why i ignored it. 
¬.¬

> On sync, you receive a full copy of the items list from every node, and 
ignore all
but the first. 

A node syncs with *every* node? I thought that only *one* sync reply is ever 
sent to
any node.

> An alternative, but much more predictable, way to implement this sort of 
thing is:
> - Make every node in charge of its own (smallish) list of new items, and 
include
this personal list within every Sync Reply (YR) and Node Status (NS).

I agree, this is much neater. But I didn't know that each node was in contact 
with
*every* other node; I was told that each node was only in contact with 3 other 
nodes.

> (Unfortunately, those packets aren't extensible without breaking 
compatibility...
my fault, sorry.)

Yeah, this was why I coded a separate sync packet.

> But, ultimately, I believe in keeping the mainline Dtella as simple as 
possible. 

That's fair enough. I guess we could keep applying patches to your main branch.

One idea would be to make packets extensible, and to have this feature as an 
optional
extension, (which would require it to be re-coded in the "in charge of its own 
list
of items" way).

Thanks for the comments and information - it was very useful. :)

Original comment by infinity0x@gmail.com on 5 Jan 2009 at 1:56

[GoogleCodeExporter]

> A node syncs with *every* node? I thought that only *one* sync reply is ever 
sent to
> any node.

A sync reply contains the nick+status for one node.  When a new node is joining 
the
network, it receives a YR directly from every single node, in order to assemble 
the
global state.

After the sync is over, a node will only exchange broadcast traffic with a 
handful of
neighbors.  Private messages are always direct node-to-node though.

Original comment by sparkm...@gmail.com on 5 Jan 2009 at 4:00

[GoogleCodeExporter]

ah, that makes more sense, and is more elegant than how i thought it worked. if 
i'd
known this when i started coding, i probably would have done it in the way that 
you
just suggested (each node keeps their own list).

Original comment by infinity0x@gmail.com on 5 Jan 2009 at 4:46

[GoogleCodeExporter]

Original comment by jacob.feisley on 2 Mar 2009 at 7:33

• Added labels: Component-Core

[GoogleCodeExporter]

This issue will most likely not be included in the mainline code, therefore, I 
am
marking it as WontFix to close out this ticket.

Original comment by jacob.feisley on 2 Sep 2009 at 4:20

• Changed state: WontFix