Bootstrap function specifications

[LivInTheLookingGlass]

Earlier today I posted an update to the javascript code which implemented a rough version of the bootstrap function. This function has been present for some time, but always threw a "not implemented" error. This issue is being opened to try defining this function, and taking criticisms of it.

This function is meant to find a network without knowing the address of those using it.

I will be defining this in Javascript, for ease of it Promise support.

function bootstrap(archetype, addr, port, protocol, seeding)  {
    const seed_list = [['euclid.nmu.edu', 44565], ['example.com', 44565]];
    let conn = new m.chord.chord_socket('0.0.0.0', 44565, new m.base.protocol('bootstrapper', 'SSL'));
    if (seeding !== false)  {
        conn.join();
    }
    for (let seed of seed_list) {
        try {
            conn.connect(seed[0], seed[1]);
        } catch (e) {}
    }
    let ret = new archetype(addr, port, protocol);
    conn.get(protocol.id).then((val)=>{
        let peers = JSON.parse(val);
        let new_peers = new Set();
        new_peers.add([addr, port]);
        for (let addr of peers) {
            try {
                ret.connect(addr[0], addr[1]);
                new_peers.add(addr);
            } catch (e) {}
        }
        conn.set(protocol.id, JSON.stringify(new Array(new_peers)));
        if (seeding === false) {
            conn.close();
        }
    });
    return ret;
}

Some notes:

1. This probably needs some API changes to be implemented correctly. They are as follows:
   1. close is not present in Javascript
   2. on_first_connection would help significantly, since handshakes are asynchronous
   3. sync.sync_socket would not be supported under this definition, since it's leasing argument comes before its protocol.
   4. Adding a connect function which obeys the nodes max_connections property would be good
2. It's unclear to me whether this will make seeding be persistent. Probably the connection is cleaned up on function exit. This means a module-level variable needs to be defined to support this.
3. This definitely creates a race condition if multiple people try to connect at the same time.
4. The dependency on SSL support means that we won't be able to support this on browsers. If WSS support is achieved (on this and Python), we should substitute it for that.

[LivInTheLookingGlass]

Perhaps problem 3 could be resolved by placing locks. Of course, then the problem is a race condition in the lock itself...

[LivInTheLookingGlass]

This feels like an instance to borrow from Paxos. They have a pretty good system for resolving conflicts between nodes. I just need to adapt it for here.

[LivInTheLookingGlass]

See branch features/apply_delta for a slightly-modified Python implementation.

def bootstrap(socket_type, proto, addr, port, *args, **kargs):
    from os import path
    from time import sleep
    from random import (shuffle, randint)
    from warnings import warn
    from umsgpack import (pack, packb, unpack, unpackb)

    ret = socket_type(addr, port, *args, prot=proto, **kargs)
    datafile = path.join(path.split(__file__)[0], 'seeders.msgpack')
    dict_ = {}
    seed_protocol = Protocol('bootstrap', proto.encryption)
    if proto == seed_protocol and socket_type == DHTSocket:
        seed = cast(DHTSocket, ret)
    else:
        seed = DHTSocket(addr, randint(32768, 65535), prot=seed_protocol)

    with open(datafile, 'rb') as database:
        database.seek(0)
        dict_ = unpack(database)

    for seeder in dict_[proto.encryption].values():
        try:
            seed.connect(*seeder)
        except Exception:
            continue

    @seed.once('connect')
    def on_connect(_):
        request = seed.get(proto.id)

        @request.then
        def on_receipt(dct):
            conns = list(dct.values()) if isinstance(dct, dict) else []
            shuffle(conns)
            for info in conns:
                if len(ret.routing_table) > 4:
                    break
                else:
                    try:
                        ret.connect(*info)
                    except Exception:
                        continue
            seed.apply_delta(cast(bytes, proto.id), {ret.id: ret.out_addr}).catch(warn)

        on_receipt.catch(warn)

        for id_, node in seed.routing_table.items():
            if id_ not in dict_[proto.encryption]:
                dict_[proto.encryption][id_] = list(node.addr)

        with open(datafile, 'wb') as database:
            pack(dict_, database)

    return ret

[LivInTheLookingGlass]

This definitely needs to be refactored. The database management should be spun off to two different methods.

[LivInTheLookingGlass]

A Javascript translation might be something like:

function bootstrap(socket_type, proto, addr, port, args)  {
    let ret = new socket_type(addr, port, proto, ...args);
    let dict = get_database();
    let seed_protocol = new Protocol('bootstrap', proto.encryption);
    if (proto.id === seed_protocol.id && socket_type === DHTSocket) {
        let seed = ret;
    }
    else    {
        let seed = new DHTSocket(addr, Math.random() * 32768 + 32767, seed_protocol);
    }

    for (let seeder of dict_[proto.encryption]) {
        try {
            seed.connect(...seeder);
        }
        catch(e)    {}
    }

    seed.once('connect', function on_connect(_) {
        request = seed.get(proto.id);
        request.then(function on_receipt(dct)   {
            conns = tuple(dct.values()) if isinstance(dct, dict) else ()
            for (let info of new Set(conns) {
                if (ret.routing_table.size > 4) {
                    break;
                }
                else    {
                    try {
                        ret.connect(...info);
                    }
                    catch(e)    {}
                }
            }
            seed.apply_delta(proto.id, {ret.id: ret.out_addr}).catch(console.warn)
        });

        on_receipt.catch(console.warn)

        for (let id_ of seed.routing_table.keys())  {
            if (dict[proto.encryption][id_] === undefined)   {
                let node = seed.routing_table.get(id_);
                dict[proto.encryption][id_] = node.addr;
            }
        }

        update_database(dict)
    });

    return ret
}

[LivInTheLookingGlass]

Javascript now has a mostly-working implementation. I know it works for finding the bootstrap network. I think it will not work for other networks, however. There seems to be an error that I'm not checking for.