Local Network Traversal - Multicast Discovery

[joshuakarp]

Created by @CMCDragonkai

Specification

There are two types of Data Flow in the MDNS System, Polling (Pull), and Announcements/Responses (Push). When a Node joins the MDNS group, the records are pushed to all other nodes. However, for the joined node to discover other nodes, it needs to conduct polling queries that other nodes respond to.

Sending Queries

The MDNS spec states that query records can have additional records, but we won't care to do this as it isn't necessary. Queries won't have any other records in the query record, much like a standard DNS packet (albeit an mdns query packet can contain multiple questions).

In the case that a responder is binded to 2 interfaces that are connected to the same network (such as a laptop with WiFi + ethernet connected), the queries asking for the ip for a hostname of the responder will receive multiple responses with different ip addresses.

This behavior is documented in: RFC 6762 14.

Control Flow

Unlike other mDNS libraries, we're going to use an AsyncIterator in order to have the consumer to have more control over the querying. An example of this would be:

async function* query({...}: Service, minimumDelay: number = 1, maximumDelay: number = 3600) {
   let delay = minimumDelay;
   while (true) {
    await this.sendPacket(...);
    delay *= 2;
    yield delay;
  }
}

The query system has been decided to have it's runtime contained within MDNS rather than being consumer-driven. This means that scheduled background queries will have to be managed by a TaskManager (similar to polykey)

Data Flow

Receiving Announcements/Responses (Pull)

Data Flow

Because queries are basically fire and forget, the main part comes in the form of receiving query responses from the multicast group. Hence, our querier needs to be able to collect records with a fan-in approach using a muxer that is reactive:

This can also be interpreted as a series of state transitions to completely build a service.

There also needs to be consideration that if the threshold for a muxer to complete is not reached, that additional queries are sent off in order to reach the finished state.

The decision tree for such would be as follows:

Control Flow

Instances of MDNS will extend EventTarget in order to emit events for service discovery/removal/etc.

class MDNS extends EventTarget {
}

The cache will be managed using a timer that is set to the soonest record TTL, rather than a timer for each record. The cache will also need to be an LRU in order to make sure that malicious responders cannot overwhelm it.

Sending Announcements

Control Flow

This will need to be experimented with a little. Currently the decisions are:

• registerService cannot be called before start is called.
• create should take in services in place of this.
• stop should deregister all services
• destroy should remove everything from the instance

  class MDNS extends EventTarget {
  create()
  start()
  stop()
  register()
  deregister()
  }

Types

Messages can be Queries or Announcements or Responses. This can be expressed as:

type MessageType = "query" | "announcement" | "response";
type Message = [MessageType, ResourceRecord] & ["query", QuestionRecord];
const message = ["query", {...}];

Parser / Generator

The Parsing and Generation together are not isomorphic, as different parsed UInt8array packets can result in the same packet structure.

Every worker parser function will return the value wrapped in an object of this type:

type Parsed<T> = {
  data: T;
  remainder: UInt8Array;
}

The point of this is so that whatever hasn't been parsed get returned in .remainder so we don't keep track of the offset manually. This means that each worker function also needs to take in a second uint8array representing the original data structure.

1. DNS Packet Parser Generator Utilities
   • Parser - parsePacket(Uint8array): Packet
   • Headers - parseHeader(Uint8array): {id: ..., flags: PacketFlags, counts: {...}}
   • Id - parseId(Uint8array): number
   • Flags - parseFlags(Uint8Array): PacketFlags
   • Counts - parseCount(Uint8Array): number
   • Question Records - parseQuestionRecords(Uint8Array): {...}
     • parseQuestionRecord(Uint8Array): {...}
   • Resource Records - parseResourceRecords(Uint8Array): {...}
     • parseResourceRecord(Uint8Array): {...}
     • parseResourceRecordName(Uint8Array): string
     • parseResourceRecordType(Uint8Array): A/CNAME
     • parseResourceRecordClass(Uint8Array): IN
     • parseResourceRecordLength(Uint8array): number
     • parseResourceRecordData(Uint8array): {...}
     • parseARecordData(Uint8array): {...}
     • parseAAAARecordData(Uint8array): {...}
     • parseCNAMERecordData(Uint8array): {...}
     • parseSRVRecordData(Uint8array): {...}
     • parseTXTRecordData(Uint8array): Map<string, string>
     • parseOPTRecordData(Uint8array): {...}
     • parseNSECRecordData(Uint8array): {...}
   • String Pointer Cycle Detection
     • Everytime a string is parsed, we take reference of the beginning and end of the string so that pointers cannot point to a start of a string that would infinite loop. A separate index table for the path of the dereferences to make sure deadlock doesn't happen.
   • Errors at each parsing function instead of letting the data view failing
     • ErrorDNSParse - Generic error with message that contains information for different exceptions. Ie. id parse failed at ...
   • Record Keys - parseResourceRecordKey and parseQuestionRecordKey and parseRecordKey - parseLabels.
   • Generator - generatePacket(Packet): UInt8Array
   • Header generateHeader(id, flags, counts...)
     • Id
     • Flags - generateFlags({ ... }): Uint8Array
     • Counts - generateCount(number): Uint8Array
   • Question Records - generateQuestionRecords(): Uint8Array - flatMap(generateQuestion)
     • generateQuestionRecord(): Uint8Array
   • Resource Records (KV) - generateResourceRecords()
     • generateRecord(): Uint8array -
     • generateRecordName - "abc.com" - ...RecordKey
     • generateRecordType - A/CNAME
     • generateRecordClass - IN
     • generateRecordLength
     • generateRecordData
     • generateARecordData(string): Uint8array
     • generateAAAARecordData(string): Uint8array
     • generateCNAMERecordData(string): Uint8array
     • generateSRVRecordData(SRVRecordValue): Uint8array
     • generateTXTRecordData(Map<string, string>): Uint8array
     • generateOPTRecordData(Uint8array): Uint8array
     • generateNSECRecordData(): Uint8array
   • Integrated into MDNS
2. MDNS
   • Querying
   • MDNS.query()
     • query services of a type
   • MDNS.registerService()
   • MDNS.unregisterService()
   • Responding
   • Listening to queries
   • Responding to all queries with all records
   • Respond to unicast
   • Truncated bit

Testing

We can use two MDNS instances to interact with each other to test both query and respond on separate ports.

Additional Context

The following discussion from 'Refactoring Network Module' MR should be addressed:

• @CMCDragonkai: (+3 comments)

  Next things:

  1. Multicast discovery (bring back what we had in the old code)
  2. Update our peer table
  3. Implement kademlia on peer table
  4. Refactor GRPC commands for this
  5. Produce test harness for multiple nodes on a local network

  Multicast Discovery

  DNS-SD and mDNS are related protocols for discovery on a local network. This local network is designated usually by a "home router" or a switch. But basically some arbitrary subnet with gateways that are part of a "single network".

  The way this works is that there's a special address (for IPv4 and IPv6) that a mDNS software both listens and sends UDP packets on. This is known as a "multicast" address.

  The UDP packet contains information structured as a "DNS message", basically the same kind of message you get when you actually query a DNS server to resolve a hostname.

  What we have to do is register a hostname for our application. Let's say "polykey", this registration is done on IANA (just like how we registered the OID for our company). This list is here: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml and our application for an assignment is here: https://www.iana.org/form/ports-services

  Now once we build a multicast mDNS client/server, we can then advertise on the multicast address by sending a DNS UDP packet that claims "polykey.local" is registered to IP at PORT. Simultaneously we can listen on the multicast address for DNS packets, and filter for all claims to "polykey.local".

  Since multiple PK agents can occur on the same subnet, this means we may either have to differentiate on a subdomain of our hostname, or append a renew record to the hostname. It is possible for a given hostname to contain multiple DNS records (and these may indicate IP and PORT assignments). I'm not sure if this is the right way to do this. Need to do a bit of research.

  At the end of the day, PK node table gets populated with this initial multicast discovery. We have to have a mDNS client (to send multicast) and mDNS server (to listen for multicast). The mDNS server would be listening for these packets.

  Note that there is software like avahi, that acts like an mDNS server/client, this is used interactively by other software that don't have native mDNS capabilities. This is not important to us... we will be advertising directly.

  According to this: https://askubuntu.com/a/1105895 it may be possible that the "service type" is different from the hostname. Therefore it's possible to randomly allocate a hostname, and only listen for a given "service type". In which case _polykey_._tcp would be the service type, and you could get hostnames like polykey-PUBLICKEYFINGERPRINT.local, and thus that should be considered unique. Therefore from a service discovery point of view, the only thing static is the IP used for multicast, and the "service type" which is globally setup at IANA. The hostnames, IPs and even ports are all dynamically setup.

  The node library https://github.com/agnat/node_mdns binds into system-level services and provides the ability to listen and query mdns packets: https://github.com/agnat/node_mdns

  A lower level library in pure-js is here: https://github.com/mafintosh/multicast-dns that is used with https://github.com/mafintosh/multicast-dns-service-types according to this issue: https://github.com/mafintosh/multicast-dns/issues/6#issuecomment-91034709

• https://news.ycombinator.com/item?id=8229792

• https://blog.apnic.net/2022/05/03/how-nat-traversal-works-concerning-cgnats/

• https://github.com/MatrixAI/Polykey/issues/487#issuecomment-1294558470

• https://github.com/MatrixAI/Polykey/issues/487#issuecomment-1294742114

• https://support.citrix.com/article/CTX205483/how-to-accommodate-hairpinning-behaviour-in-netscaler

• mDNS RFC https://datatracker.ietf.org/doc/html/rfc6762

• DNS-SD RFC https://datatracker.ietf.org/doc/html/rfc6763

• Domain Names RFC https://datatracker.ietf.org/doc/html/rfc1035

• Extension Mechanisms for DNS RFC https://datatracker.ietf.org/doc/html/rfc6891

• NSEC RFC https://datatracker.ietf.org/doc/html/rfc3845

Tasks

• [x] Parser - 5.5 days
  • [x] Packet Header - 0.5 days
  • [x] Packet Flags - 0.5 days
  • [x] Questions - 0.5 days
  • [x] Resource Records - 4 days
• [x] Generator 5.5 days
  • [x] Packet Header - 0.5 days
  • [x] Packet Flags - 0.5 days
  • [x] Questions - 0.5 days
  • [x] Resource Records - 4 days
• [x] MDNSCache - 2.5 days
  • [x] Multi-Keyed Maps for ResourceRecord Querying - 0.5 days
  • [x] TTL Expiry Record Invalidation - 0.5 days
  • [x] Reverse Host to Record Mapping - 0.5 days
  • [x] LRU to Prevent DoS - 0.5 days
  • [x] Support use as local resource record in-memory database - 0.5 days
• [x] TaskManager - ? days
  • [x] Migrate to in-memory - ? days
• [x] MDNS - 11.5 days
  • [x] UDP Networking Logic - 2 days
  • [x] Socket Binding to Multiple Interfaces - 1 days
  • [x] Error Handling - 1 days
  • [x] Querier - 4 days
  • [x] Service Querying - 2.5 days
    • [x] Record Aggregation for Muxxing Together Services - 0.5 days
    • [x] Querying For a Service's Missing Records - 0.5 days
    • [x] Emitting Expired Services - 0.5 days
  • [x] Unicast 1.5 days
    • [x] Checking for Unicast Availability - 1 days
    • [x] Sending queries with unicast enabled - 0.5 days
  • [x] Responder 5.5 days
  • [x] Service Registration - 0.5 days
  • [x] Filter Messages Received from Multicast Interface Loopback - 0.5 days
  • [x] Unicast
    • [x] Responding to Unicast Queries - 0.5 days

[CMCDragonkai]

The old networking code is located here: https://github.com/MatrixAI/js-polykey/tree/3340fc7508e46a6021d1bd6d9005c99ea598e205/src-old/network

There may be some artifacts worth fetching out. Especially implementation of the local network discovery.

[CMCDragonkai]

Just a note, AWS's subnets in the VPC by default doesn't support multicast. And thus no mdns.

However this can be enabled by creating a transit gateway: https://docs.aws.amazon.com/vpc/latest/tgw/working-with-multicast.html.

Haven't tried it though.

Multicast is a way of doing "automatic service discovery" (by bootstrapping off a known location).

Alternative ways include using AWS's own service discovery, but that's not portable, and limited to specifically ECS or whatever aws provides there. And the only usage for that is to be able to auto-discover a seed node cluster in AWS like we are doing for testnet and mainnet.

Without automatic service discovery, deployment of agents into testnet and mainnet has to occur one by one, where each subsequent agent is deployed like a congaline, and has given knowledge of the other agents.

I wonder though... perhaps if I did use the auto-sd, maybe I could pass the SD domain/hostnaem directly as the seed node specification, and rely on our DHT to make use of it. Then we would be using AWS's SD but in a portable manner.

---

For the seed nodes, this can be worked around by using the DNS hostname testnet.polykey.io instead of having the different IPs... (different IPs is complex due to local IPs vs external IPs).

[CMCDragonkai]

Manual testing in https://github.com/MatrixAI/Polykey/issues/487#issuecomment-1294742114 has revealed an urgent need for some kind of "local network traversal".

Basically if 2 nodes are on the same subnet/LAN and thus have the same public IP address, hole punching using the relay signalling message will not work.

This is because the router may not support hairpinning. And therefore the packets just get dropped.

This can happen if 2 nodes are running on the same computer, and are using different ports. And it can also happen if 2 nodes are running on the same subnet, and are using different private IPs and ports.

It could also happen in more general way where in larger corporate networks.

Or even in larger CGNAT scenarios, where the home router themselves are not given a public IP address. Like imagine buying a bunch of nokia 5G routers for home usage, and now every home in a local area may be part of the same CGNAT IP.

This can seriously hamper connectivity!

[CMCDragonkai]

Local multicase is necessary for local discovery. This means that a given NodeId may have multiple "valid" IP addresses/ports to access depending on who's the one asking the question.

To deal with this, we have to refactor the NG to be capable of dealing with the inherent ambiguity of node addresses.

Changing our NG key path to BucketIndex/NodeId/Host/Port, because different mechanisms may end up discovering different valid host and port combinations.

Tailscale seems to support some sort of local discovery, combined with detections of whether hairpinning works, and whether the immediate router supports PMP or PCP.

»» ~
 ♖ tailscale netcheck                                                                                    pts/2 16:22:29

Report:
    * UDP: true
    * IPv4: yes, 120.18.72.95:2703
    * IPv6: no
    * MappingVariesByDestIP: false
    * HairPinning: false
    * PortMapping: 
    * Nearest DERP: Sydney
    * DERP latency:
        - syd: 56.3ms  (Sydney)
        - sin: 126.8ms (Singapore)
        - blr: 173.2ms (Bangalore)
        - tok: 173.3ms (Tokyo)
        - hkg: 197.2ms (Hong Kong)
        - sfo: 197.3ms (San Francisco)
        - lax: 201.7ms (Los Angeles)
        - sea: 207.5ms (Seattle)
        - den: 207.6ms (Denver)
        - ord: 221.5ms (Chicago)
        - dfw: 250.8ms (Dallas)
        - tor: 250.9ms (Toronto)
        - nyc: 257.7ms (New York City)
        - hnl: 257.7ms (Honolulu)
        - mia: 257.8ms (Miami)
        - lhr: 314.2ms (London)
        - par: 325.1ms (Paris)
        - ams: 330.6ms (Amsterdam)
        - mad: 330.8ms (Madrid)
        - fra: 330.8ms (Frankfurt)
        - sao: 370.5ms (São Paulo)
        - waw: 370.7ms (Warsaw)
        - dbi: 471.1ms (Dubai)
        - jnb: 505.1ms (Johannesburg)

Meaning that a multitude of methods can be tried before falling back on some centralised relay.

[CMCDragonkai]

For the most immediate use case, I think we can solve the problem of:

1. 2 or more nodes on the same machine
2. 2 or more nodes on the same home LAN

With the introduction of multicast discovery and expanding our NG to take that ambiguity and resolving it.

[CMCDragonkai]

The fact that signalling does work though means that the signalling node is a common source of coordination. It can help do relaying, but it can also help the 2 nodes try to discover each other on the local networks too.

If we aren't afraid of "leaking private data", it's possible to provide private network information to the seed node.

If we want to hide that information from the seed node, it's possible to encrypt this data for the other node, and rely on the seed node to relay encrypted information to each other. This kind of leads to zero knowledge protocols too. https://www.theguardian.com/technology/2022/oct/29/privacy-problem-tech-enhancing-data-political-legal (I think these are called privacy enhancing protocols).

[CMCDragonkai]

Wanted to mention that look to tailscale for inspiration.

They also keep track of all local IP:port bindings, and send that off to the tailscale server which is then distributed to other clients, and so other clients can actually just "attempt" DCs to the local IP:port. This actually sometimes works very well, and avoids any need to use complicated MDNS, multicast, PMP... etc protocols.

This is still a signaling system.

@amydevs first attempt to prototype the multicast system locally on a single computer, then we between computers on a home router (office router). And then look at the network and node graph modules to see how it can be integrated.

You need to have a read of the MDNS and multicast RFCs. There are existing libraries for this, but only use those as a guide. You do not need a library for implementation here.

[amydevs]

What I think I've found so far: For DNS-SD (Service Discovery) to work through mdns, two things need to happen.

1. A host announcement needs to be made to register an 'A' record between the local IP of the device and it's hostname. This is typically done automatically by the OS on Windows and MacOS, and Avahi on Linux. It would be adequate to use the existing hostnames in these cases. Otherwise, we can check if the 'A' record already exists, and if not, create our own in case the user has network discovery disabled. The hostname of the device can be derived from os.hostname(), whether or not the network discovery is enabled or not.
2. A service announcement needs to be made with a 'SRV' record with the host set to the hostname of the device in the first step. A port and service type can be set as well. We can set this to the QUIC RPC-Server port, with the service being something like _polykey._udp. (More is needed than just a SRV record, a TXT record is also needed, will add on this later)

These records are created by sending a DNS Response packet to the well-known mdns address.

A quick way to check if the service records have been correctly created is with avahi-browse --all on Linux or dns-sd -B _services._dns-sd._udp on Mac.

[amydevs]

The RFC mentions as a recommendation that all clients of a machine should use a single shared MDNS implementation (bonjour, avahi, etc.). I think we ignore this recommendation as:

1. We want this to work even if the user hasn't installed avahi / enabled network discovery (as is the default on windows machines).
2. Linking into native libraries for mdns is a bit overkill when all we're doing is sending data through sockets.

[CMCDragonkai]

Does the naming matter? Or we just choose Polykey.

[CMCDragonkai]

Our standard port is 1314. Does this need to be fixed or can we do this for any port PK binds to?

[CMCDragonkai]

Just wondering do we need to do a host announcement? It seems like this is an OS thing. Can we just expect that it is already done?

[CMCDragonkai]

And if it isn't done what happens? Multicast just doesn't work? Then so be it.

[amydevs]

@CMCDragonkai

Does the naming matter? Or we just choose Polykey.

It doesn't matter

Our standard port is 1314. Does this need to be fixed or can we do this for any port PK binds to?

We can create the announcement after the port is bound. It does not need to be fixed and can be updated after the first announcement if needed.

Just wondering do we need to do a host announcement? It seems like this is an OS thing. Can we just expect that it is already done?

It's typically already done by the OS. Although it seems to be off for my device even though I have avahi installed, as trying to resolve my host or local ip address does not turn up with anything. (Edit: I figured it out, in configuration.nix, services.avahi.publish.enable and services.avahi.publish.addresses need to be both set to true)

And if it isn't done what happens? Multicast just doesn't work? Then so be it.

If it isn't done, the hostname of the device doesn't correspond to any mdns records, so SRV records that point to that hostname can't be used by a resolver to get the corresponding ip address

[tegefaulkes]

I got curious and looked at how to do multicast using the node UDP socket. I made a quick example.

test('multicast' , async () => {
  const socket1 = dgram.createSocket({
    type: 'udp4',
    reuseAddr: false,
  });
  const socket1Bind = utils.promisify(socket1.bind).bind(socket1);
  const socket1Send = utils.promisify(socket1.send).bind(socket1);

  await socket1Bind(55555);
  socket1.setMulticastLoopback(true);
  socket1.addMembership('224.0.0.114');
  socket1.addListener('message', (m) => console.log('socket1 message: ', m.toString()));

  const socket2 = dgram.createSocket({
    type: 'udp4',
    reuseAddr: false,
  });
  const socket2Bind = utils.promisify(socket2.bind).bind(socket2);
  const socket2Send = utils.promisify(socket2.send).bind(socket2);
  await socket2Bind(55556);
  socket2.setMulticastLoopback(true);
  socket2.addMembership('224.0.0.114');
  socket2.addListener('message', (m) => console.log('socket2 message: ', m.toString()));

  await socket1Send('The quick brown fox jumped over the lazy dog', 55556);
  // message received on socket2, port seems to matter here.
  // > `socket2 message:  The quick brown fox jumped over the lazy dog`

  await sleep(5000);
})

[CMCDragonkai]

1. Avahi recommends not to have multiple mdns stacks running at the same time. They argue that you should use the DBUS api https://www.avahi.org/doxygen/html/
2. https://github.com/libp2p/js-libp2p-mdns - this library appears to fit the mdns spec but doesn't appear to directly integrate to avahi.
3. It seems we can choose to do multicast directly such as https://teukka.tech/peer-discovery.html and basically register/pick a specific unused multicast address https://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml

So... I'd try both. Try multicasting directly and review the libp2p library and see if avahi picks it up.

Then also consider somehow sending instructions to avahi over DBUS if available.

Note that bonjour has its own binding too on Mac and Windows.

[amydevs]

The js-mdns library has been created as a TypeScript implementation of a mDNS/DNS-SD responder/advertiser/stack.

The current plan is to first release the pure TypeScript implementation. This implementation is expected to ignore any other mDNS stack running on the system, and be aware of the issues regarding this.

After that has been achieved, we can add functionality to integrate with common mDNS stacks such as

• Avahi using the DBUS API
• Apple's mDNSResponder
• Win32 with C++ API Calls
• iOS/Android

[CMCDragonkai]

Yep start speccing out the expected interface and classes and methods.

[CMCDragonkai]

So basically it turns out that we will need 3 things:

1. File-based discovery using /var/run/polykey - for cases where PK is running on localhost, and multicast is not enabled by default for the loopback interface
2. MDNS which should work for PK agents bound to the public interfaces, and across home routers or whether routers/switches have multicast enabled
3. Providing all bound local interface IPs to the mainnet/testnet, similar to tailscale and thus end up providing a list of potential IPs to try out

Important lesson, just because you discover it, doesn't necessarily mean you can connect to it. Until decentralised relaying is possible, discovery for now gives you potential nodes that exist, but it doesn't mean you can actually connect to it. This can happen if agents are connected to loopback only... and other agents are connected on public interface, or even both. Other agents on other networks won't be able to reach it.

Network reachability is quite complex due to how complex networks themselves can be. Decentralised will enable increased connectivity into deeper agents, but that may also not be what users want, but this is something we will have to research later.

[amydevs]

Points from the RFC to implement

Required:

• [x] Responding Unicast Response Bit / Responding to Unicast Queries on Port 5353 - https://datatracker.ietf.org/doc/html/rfc6762#section-5.4 / https://datatracker.ietf.org/doc/html/rfc6762#section-5.5
  • The current library that i'm using as a placeholder for dns packet de-serialization does not support the unicast response bit. This will have to be implemented once I have implemented my own. Take notes from the native node dns package: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/1775bb238c921b39161c02f6907973e4d07c6f82/types/node/dns.d.ts#L169
• [x] Correctly Implement DNS Rules https://datatracker.ietf.org/doc/html/rfc6762#section-6
  • The current library that i'm using as a placeholder for dns packet de-serialization does not parse "ANY" qtype's. Once again, have to implement my own.
• [x] Recommended RR TTLs https://datatracker.ietf.org/doc/html/rfc6762#section-10 the responder can omit from their message.
• [x] Multicast Interface Loopback needs to filter out own packets by an Arbitrary Query ID (ID's only matter to legacy resolvers, such as standard DNS querier)
  • This isn't apart of the spec, we just need to do this as multicastLoopback is enabled.
• [x] DNS Name Compression
  • When parsing for utf-8 names, check if the first character is 0xC0, if it is, take the next 14 bits as the pointer. Then traverse the packet to the byte defined by the pointer number, then start reading the name from there.
  • I've created some tests for this with fastcheck + jest. However, it is not optimal. Preferably, I would like to generate a fastcheck list of questions, and then choose a question at random, and slot my pointer question in at random.

[amydevs]

Here's a JSON representation of the minimum amount of resource records required for a service to be discoverable:

[
    {
        "name": "123.local", // hostname can be anything + '.local'
        "type": "A",
        "class": "IN",
        "data": "192.168.1.104",
        "ttl": 120, // 120 seconds is the well-known TTL of all hostname and hostname related records in the mDNS RFC
        "flush": true
    },
    {
        "name": "123.local",
        "type": "AAAA",
        "class": "IN",
        "data": "fe80::f137:6ff0:9dbf:c5bd",
        "ttl": 120,
        "flush": true
    },
    {
        "name": "_services._dns-sd._udp.local", // this pointer for the discovery of service types on the network
        "type": "PTR",
        "class": "IN",
        "ttl": 120,
        "data": "_polykey._udp.local",
        "flush": true
    },
    {
        "name": "_polykey._udp.local", // this is for the discovery of the particular service instance we're running when someone queries for the service type
        "type": "PTR",
        "class": "IN",
        "ttl": 120,
        "data": "123._polykey._udp.local",
        "flush": true
    },
    {
        "name": "123._polykey._udp.local", // This is the "name" of the service instance. We can realistically have it as "<anything>._polykey._udp.local" as long as the corresponding SRV record has the correct hostname in it's RRDATA section. But it's more convenient to have the hostname as the instance, as it's generally convention, and we can assume that if the hostname on the network is unique, then the service name should be too.
        "type": "TXT", // A TXT record is required for each service instance, the data can be what ever key-value entries we want. It's usually to express the configuration of a service on a particular machine.
        "class": "IN",
        "ttl": 120,
        "data": [],
        "flush": true
    },
    {
        "name": "123._polykey._udp.local", // This defines the port service instance.
        "type": "SRV",
        "class": "IN",
        "ttl": 120,
        "data": {
            "port": 1234,
            "target": "123.local",
            "priority": 0,
            "weight": 0
        },
        "flush": true
    }
]

Note that the flush bit is only needed to be true on the announcement of a new record.

[amydevs]

Parser Structure:

function parserTop(input_packet: UInt8Array, original_packet: UInt8Array) {
    let remaining = original_packet;

    const dv = new DataView(input);
    const id = dv.get(...);
    const type = dv.get(...);

    input = input.subarray(...);

    // Whatever left to parse is set as the input variable for the next parser down the tree. So that we can shift the dataviews over the uint8array without copies!
    [parsedPart1, remaining] = parsePart1(remaining, original_packet);

    [parsedPart2, remaining] = parsePart2(remaining, original_packet);

   return [parsedPart, remaining];
}

Also, revise the to/from to parse/generate as we'd like to throw errors on issues with parsing https://github.com/MatrixAI/MatrixAI-Graph/issues/44

[CMCDragonkai]

I updated the issue above, but it's still lacking alot of the spec. The spec should fleshed out with all the pieces of the parser/generator.

As well as how the MDNS will behave in each section.

Then a task list can be created.

The spec should be tree-oriented. The task list can be a hierarchical task list.

This would be priority Monday morning.

[CMCDragonkai]

Have a look at our recent work in js-async-monitor and js-async-cancellable and maybe we can use that in MDNS in case there are concurrent state mutations.

Are there any concurrent state mutations in the MDNS code actually?

[amydevs]

Have a look at our recent work in js-async-monitor and js-async-cancellable and maybe we can use that in MDNS in case there are concurrent state mutations.

Are there any concurrent state mutations in the MDNS code actually?

the building of a service could be counted as a concurrent state mutation maybe?

[amydevs]

currently i've got the querying implemented by just parsing responses, adding records to a cache, and checking what records are missing according to the cache, and querying for them. If i get a response with the records i require, i collate all the records in the cache regarding a service, and emit it on the MDNS instance, if not, no-op. (roughly following decision tree above).

However, to treat a service as a state that transitions into itself, i think it might be more suitable to create a class that represents a service, and use something like a builder pattern.

[CMCDragonkai]

Have a look at our recent work in js-async-monitor and js-async-cancellable and maybe we can use that in MDNS in case there are concurrent state mutations. Are there any concurrent state mutations in the MDNS code actually?

the building of a service could be counted as a concurrent state mutation maybe?

No concurrent state mutation is when 2 or more concurrent calls to class methods can occur at the same time, and these methods end up mutating a shared state. If they mutate shared state in partial-ways (non-atomic), that can result in a race condition, data clobbering and other weird scenarios.

Please identify if there is, and list them out methods that may be called concurrently. Any method of the class that is async can be called concurrently and identify whether they may clobber each other's state.

[CMCDragonkai]

currently i've got the querying implemented by just parsing responses, adding records to a cache, and checking what records are missing according to the cache, and querying for them. If i get a response with the records i require, i collate all the records in the cache regarding a service, and emit it on the MDNS instance, if not, no-op. (roughly following decision tree above).

However, to treat a service as a state that transitions into itself, i think it might be more suitable to create a class that represents a service, and use something like a builder pattern.

I believe your "service" should be a POJO, not an class/object. You want to keep it as simple as possible here. And it's perfectly possible to build out the POJO dataflow.

Make sure to separate your dataflow model from your controlflow model.

[CMCDragonkai]

As I explained on Friday, it's possible to consider that the service is always just 1 state, and all transitions transit from the same state to the same state.

It's when you break apart variations of that 1 state, is when you can start considering a much more detailed state machine.

However when dealing with a database, you have potentially infinite state. This means you cannot use a finite state machine to represent it since you don't have a "finite" set of states.

The way to get around this is to encapsulate the infinite-ness somewhere so it's no longer observable, or considered within the model. (All models are maps, and they all drop detail). By doing this, you can have a finite model that represents the important control flows but not it won't show the "unimportant details".

Use models as a way of doing abstract reasoning.

[CMCDragonkai]

@amydevs any update on the API since, you are now going to use something like startQuery and stopQuery.

And how are you going to incorporate the Task type from TaskManager?

Plus what about the insertion sort?

[CMCDragonkai]

The multicast system is like a broadcast within the multicast. While this library can default to the standard MDNS group address. In PK we should select a MDNS group address that is unlikely to be used by other MDNS stacks.

Go to here https://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml and select an address that is not already being used. Also what is the difference between "Routable" and "Non Routable" (mdns is non-routable).

I suggest 224.0.0.250 it's 1 bit less than 224.0.0.251 as standard MDNS. Do the same for IPv6.

[amydevs]

Done:

• The dns packet parser has been finished.
• Service registration is possible and can be resolved from other mDNS responders like Bonjour and Avahi.
• Services on the local network can also be constructed from incoming responses and announcements.

To-Do:

• Make all private methods protected
• Default value for object paramaters should be an empty object ({})
• Use StartStop for mDNS instead of CreateDestroyStartStop
• Rewrite mDNS to utilize multiple sockets rather than just 1.
• Edit TaskManager to work in-memory.
• Make the querier schedule queries with the TaskManager.
• Incorportate LRU into MDNSCache
• MDNSCache should have a Map<string, ResourceRecord> that has a key that combines rrtype, rrclass, name, and rdata

[CMCDragonkai]

@amydevs you still have to update all the expected dates above to 0.5/1 day increments.

And you should write down what the next steps are.

[CMCDragonkai]

I've updated the entire project to align with the project structure in other projects now.

[CMCDragonkai]

We're still pushing to staging atm. So eventually it will need to be squash rebased.

[CMCDragonkai]

The index.ts was missing all major exports.

@amydevs make sure you're reviewing all my commits from this the last one so you're aware of the situation.

[CMCDragonkai]

Review

Lifecycle

In MDNS.start:

    hostname = `${os.hostname()}.local` as Hostname,

Is used as a parameter.

However if you expect the user to put in the hostname fully, you would not expect a .local to exist.

So instead you should take hostname = os.hostname() as Hostname.

Then add the .local afterwards as per the MDNS spec.

This is assuming .local is required by the spec?

Also because MDNS.createMDNS is supposed to call MDNS.start, you need to propagate all parameters to the top level of createMDNS.

Default parameters are on createMDNS, not on start.

All of this is you should notice on all of our existing code, there's a reason we write it like this, and if you are deviating from it, you must ask a question about it or document why you are doing it.

Furthermore the createMDNS or creator functions must always be asynchronous. Did you see the js-async-init tests to see how to use it. Always compare to existing repos like js-db, js-encryptedfs and in Polykey under the acl domain etc.

The defaults must then be on the creator, not on the constructor.

I changed to StartStop. Remember that static functions and properties must always be on top of instance functions and properties.

[CMCDragonkai]

Some relevant discussion about sockets and interfaces and IPs:

And https://chat.openai.com/share/0b86282d-7073-41c5-ba97-40de9902ae01

[CMCDragonkai]

So the reason we need multiple sockets (1 for each interface) when dealing with ::, because when we receive a packet on the multicast group. We only want to respond via unicast or multicast on the interface that we received the packet from.

If we used a single socket for ::, then we would end up sending a response to all interfaces, even if those interfaces are not on the same network.

This is why even when binding to wildcard, we need to use multiple interfaces.

[CMCDragonkai]

Found out that IPv6 interfaces might have a link local address.

In that case when using :: or ::0 we need to detect if the address has fe80:: as the starting prefix, then attach the %interfacename to the end of the Host string.

Otherwise it's not possible to bind to them as well with an additional socket.

Also made me realise that for a given deployment of the MDNS stack we end up creating quite a few sockets.

[tegefaulkes]

If I recall, there's also the prefix fd00 and fc00 for local and private network IPv6 addresses.

[CMCDragonkai]

I'm not sure if they suffer from the same problems with binding however.

@amydevs I've been working on a generic table data structure: https://github.com/MatrixAI/js-table

If it works well, perhaps we switch to using it since it seems like a common occurrence. Let me know if the API suits the MDNS cache.

One thing I'm thinking about is dealing with compound keys. Without compound keys, any need to lookup multiple keys requires a set intersection applied to all the row index arrays. However to make the lookup a bit more efficient, you can pre-setup the compound keys at the beginning of the table requiring ['k1', 'k2'] as a compound key. In doing so, one should also supply a hash function. Otherwise the default concatenator will just be + operator. However not all types actually support it. So all keys of records are always strings, which can be concatenated with just +, but their values may not necessarily be so. For example symbols cannot be concatenated. But we cannot be sure of this until a row gets inserted, and we attempt to + them together.

Right now keys can be objects, and look up works the same way. There is the derivation functions that can be supplied to force them to look up as strings. This is optional.

[CMCDragonkai]

@amydevs js-table can be used at 1.0.0 pending release https://gitlab.com/MatrixAI/open-source/js-table/-/pipelines

[CMCDragonkai]

There was a commit that I forgot to push before.

I removed the type getter. It didn't make sense because MDNS could have multiple sockets, each with different types.

At the same time, if you are binding to ipv4, you get ipv4 type, ipv6 to ipv6 type, and if you use ipv4 mapped ipv6... I forgot what the type should be, but it should be the same as whatever js-quic does.

Furthermore each socket object should have type, host and port properties attached.

This means something like this:

[CMCDragonkai]

Releasing 1.1.0 of js-table with the iterator changed to give you [number, R] this gives you access to the row index too. So you can delete things while iterating. Can be useful for limiting the cache size. Cache size limitation can just be a "drop old row" policy rather than going for a full blown LRU.

Also the table indexes options supports hashing functions. The default hashing function has changed to now support null and undefined. But you can provide your own hashing function.. Prefer producing a string, unless you have a specific reason.

[CMCDragonkai]

Should also create your own special type XRow type. Where X is the thing you are putting into the table. This XRow type can be flattened as much as possible, that way you're able to index whatever deep keys you want.

[amydevs]

The uniqueness of a record in mDNS, (even for shared records) is defined by the record's name, class, type, and data. Hence, MDNSCache needs a way to compoundly index by those 4 fields. The main field that is the problem is data, calling JSON.stringify on the parsed structure is not canonical. Hence, instead we should use the canonicalize.

[amydevs]

i've moved and renamed MDNSCache to ResourceRecordCache to a separate folder to better reflect what it actually does. The reason for moving it to a separate folder is so that i can separate the utils, events, and errors for the cache

[amydevs]

What's left is:

• Splitting responses to separate interfaces
• Managing promises

[amydevs]

https://github.com/nodejs/node/issues/1690#issuecomment-224878448

After all, it seems that it is impossible to receive multicast messages from a socket directly binded to a specific interface. This makes things alot more complicated...

The solution ciao, bonjour-js, mdns-js, etc. have all chosen is the first method:

binding only to the PORT the multicast message is sent and joining the multicast group i.e.: as in receive-all-addresses.js. This way, you can receive every message sent to that port whether multicast or not.

However, as ciao states, this is leaky! and kind of defeats the whole purpose of the point of binding to each interface individually! https://github.com/homebridge/ciao/blob/a294fce273b19cac06fbc5dcdbb8db5e77caa68d/src/MDNSServer.ts#L518

The second option seems more sane:

binding to the PORT and the multicast address and joining the multicast group. This restricts the messages received to multicast only. As in receive-specific-address.js but binding to MULTICAST_IP instead of LOCAL_IP.

However, the obvious caveat is that we can't receive unicast messages.

My plan is just to focus on multicast for now. So we can bind onto the multicast group address, then use setMulticastInterface and addMembership targetting specific interface addresses in order to have a separate socket (and hence handler) for each interface.

However, only multicast messages will be able to be received on those sockets. Hence, unicast sockets also need to be binded later if we need them...