mDNS Support

[opichals]

It would be nice to develop at least a minimal mDNS support.

Features:

• respond to hostname.local mDNS queries
• respond to _services._dns-sd._udp.local queries
• should provide callback-based interface for providing specific service query responders

[opichals]

At the moment there is this hacked-up implementation gist.

[wilberforce]

@opichals

At this point I would like to get the ESP32 to do this -> respond to hostname.local mDNS queries rather than use the ESP-IDF libs would like to do in js. I tried your gist - wow - lots of output!

Any reason this is closed?

[opichals]

@wilberforce This issue is not closed, only the ESP8266 specific issue is as the intention is to have a .js module for it which is not board specific.

The gist was partially used to debug the dgram implementation so that's why it's got so much output. I have been working on a callback-based version of that recently. I am going to update the gist as soon as I get to it.

[opichals]

@wilberforce Gist updated with a version that doesn't hold the whole parsed DNS packet in memory and rather issues callback calls for every record.

As I mentioned I have been hacking on this recently and I still intend to extend it to provide the features from this issue description.

[wilberforce]

Thanks for working on this. I had to hard code the hostname wifi.gethostname is not implemented on the esp32 - I guess you on the esp8266?

I set the hostname to esp would you expect http://esp.local/ to work yet?

Have some debug output - looks like apple TV is giving unexpected output:

"message": "Field or method \"NaN\" does not already exist, and can't create it on Uint8Array",

MSG { "tid": 0, "flags": 0, "questions": 1, "answerRRs": 2,
  "authority": 0, "additionalRRs": 1,
  "records": [  ]
 }
REC {"name":"_sleep-proxy._udp.local","type":12,"clazz":1,"len":29}
REC {"name":"_sleep-proxy._udp.local","type":12,"clazz":1,"len":35,"ttl":4497,"data":[20,55,48,45,51,53,45,54,48,45,54,51,46,49,32,76,111,117,110,103,101,192,12]}
REC {"name":"_sleep-proxy._udp.local","type":12,"clazz":1,"len":40,"ttl":4499,"data":[25,55,48,45,51,53,45,54,48,45,54,51,46,49,32,70,97,109,105,108,121,32,82,111,111,109,192,12]}
REC {"name":"","type":41,"clazz":1440,"len":29,"ttl":4500,"data":[0,4,0,14,0,240,172,188,50,104,157,140,172,188,50,104,157,138]}
MEM {"free":2012,"usage":488,"total":2500,"history":592,"gc":0,"gctime":2.317}
memory {"free":1956,"usage":544,"total":2500,"history":592,"gc":0,"gctime":2.367}
s:message 192.168.15.9 5353
MSG { "tid": 0, "flags": 33792, "questions": 0, "answerRRs": 23,
  "authority": 0, "additionalRRs": 1,
  "records": [  ]
 }
REC {"name":"Family Room._companion-link._tcp.local","type":16,"clazz":32769,"len":95,"ttl":4500,"data":[22,114,112,66,65,61,65,67,58,66,67,58,51,50,58,54,56,58,57,68,58,56,66,9,114,112,70,108,61,48,120,56,50,11,114,112,86,114,61,49,52,48,46,51,49]}
REC {"name":"_services._dns-sd._udp.local","type":12,"clazz":1,"len":37,"ttl":4500,"data":[192,24]}
REC {"name":"_companion-link._tcp.local","type":12,"clazz":1,"len":14,"ttl":4500,"data":[192,12]}
REC {"name":"Family Room._device-info._tcp.local","type":16,"clazz":1,"len":50,"ttl":4500,"data":[12,109,111,100,101,108,61,74,52,50,100,65,80]}
REC {"name":"ACBC32689D8C@Family Room._raop._tcp.local","type":16,"clazz":32769,"len":232,"ttl":4500,"data":[10,99,110,61,48,44,49,44,50,44,51,7,100,97,61,116,114,117,101,8,101,116,61,48,44,51,44,53,21,102,116,61,48,120,53,65,55,70,70,70,70,55,44,48,120,53,53,70,68,69,8,115,102,61,48,120,50,52,52,8,109,100,61,48,44,49,44,50,13,97,109,61,65,112,112,108,101,84,86,53,44,51,67,112,107,61,55,55,52,53,57,51,97,55,52,49,97,56,55,98,98,54,97,57,57,101,50,98,53,51,98,48,49,54,54,54,52,50,49,56,48,98,49,100,102,98,101,102,99,48,99,48,99,57,56,54,55,56,102,101,98,102,98,102,99,50,99,54,50,98,6,116,112,61,85,68,80,8,118,110,61,54,53,53,51,55,9,118,115,61,51,54,53,46,53,51,7,111,118,61,49,49,46,51,4,118,118,61,50]}
REC {"name":"_services._dns-sd._udp.local","type":12,"clazz":1,"len":14,"ttl":4500,"data":[192,233]}
REC {"name":"_raop._tcp.local","type":12,"clazz":1,"len":14,"ttl":4500,"data":[192,208]}
REC {"name":"Family Room._airplay._tcp.local","type":16,"clazz":32769,"len":60,"ttl":4500,"data":[5,97,99,108,61,48,26,100,101,118,105,99,101,105,100,61,65,67,58,66,67,58,51,50,58,54,56]}
Error Error {
  "message": "Field or method \"NaN\" does not already exist, and can't create it on Uint8Array",
  "type": "Error",
  "stack": " at line 13 col 33\n            len = bytes[offset++];\n                                ^\nin function \"parseDnsName\" called from line 1 col 41\nvar dnsName = parseDnsName(bytes, offset);\n                                        ^\nin function \"parseRecord\" called from line 3 col 61\n...bytes, ctx.offset, isAnswer);\n                              ^\nin function \"handleRecords\" called from line 20 col 43\n    handleRecords(ctx, packet.answerRRs, 1);\n                                          ^\nin function \"processMessage\" called from line 9 col 18\n                });\n                 ^\n"
 }

[opichals]

For .js development I mostly run espruino binary on a mac. Once sufficiently tuned I try the boards. And yes, mostly ESP8266.

Hm, I guess I will need to print the byte contents before parsing so that we can debug the NaN you are seeing. And there is definitely more stuff to iron out as I am also seeing some sort of mem-leak or something (the jsvLock > 15 assert) under some circumstances.

I have updated the gist a bit more to parse PTR, SRV and TXT records.

[wilberforce]

I added the debug on ESP32:

function parseDnsName(bytes, offset) {
  console.log({fn:'parseDnsName',offset:offset,len:bytes.length,bytes:bytes});
    var start = offset;

So it looks like the offset is bigger than actual data...

{ "offset": 814, "len": 516, "bytes": new Uint8Array([11, 70, 97, 109, 105, 108, 121, 32, 82, 111, 111, 109, 15, 95, 99, 111, 109, 112, 97, 110, 105, 111, 110, 45, 108, 105, 110, 107, 4, 95, 116, 99, 112, 5, 108, 111, 99, 97, 108, 0, 0, 16, 128, 1, 0, 0, 17, 148, 0, 45, 22, 114, 112, 66, 65, 61, 65, 67, 58, 66, 67, 58, 51, 50, 58, 54, 56, 58, 57, 68, 58, 56, 66, 9, 114, 112, 70, 108, 61, 48, 120, 56, 50, 11, 114, 112, 86, 114, 61, 49, 52, 48, 46, 51, 49, 9, 95, 115, 101, 114, 118, 105, 99, 101, 115, 7, 95, 100, 110, 115, 45, 115, 100, 4, 95, 117, 100, 112, 192, 45, 0, 12, 0, 1, 0, 0, 17, 148, 0, 2, 192, 24, 192, 24, 0, 12, 0, 1, 0, 0, 17, 148, 0, 2, 192, 12, 11, 70, 97, 109, 105, 108, 121, 32, 82, 111, 111, 109, 12, 95, 100, 101, 118, 105, 99, 101, 45, 105, 110, 102, 111, 192, 40, 0, 16, 0, 1, 0, 0, 17, 148, 0, 13, 12, 109, 111, 100, 101, 108, 61, 74, 52, 50, 100, 65, 80, 24, 65, 67, 66, 67, 51, 50, 54, 56, 57, 68, 56, 67, 64, 70, 97, 109, 105, 108, 121, 32, 82, 111, 111, 109, 5, 95, 114, 97, 111, 112, 192, 40, 0, 16, 128, 1, 0, 0, 17, 148, 0, 189, 10, 99, 110, 61, 48, 44, 49, 44, 50, 44, 51, 7, 100, 97, 61, 116, 114, 117, 101, 8, 101, 116, 61, 48, 44, 51, 44, 53, 21, 102, 116, 61, 48, 120, 53, 65, 55, 70, 70, 70, 70, 55, 44, 48, 120, 53, 53, 70, 68, 69, 8, 115, 102, 61, 48, 120, 50, 52, 52, 8, 109, 100, 61, 48, 44, 49, 44, 50, 13, 97, 109, 61, 65, 112, 112, 108, 101, 84, 86, 53, 44, 51, 67, 112, 107, 61, 55, 55, 52, 53, 57, 51, 97, 55, 52, 49, 97, 56, 55, 98, 98, 54, 97, 57, 57, 101, 50, 98, 53, 51, 98, 48, 49, 54, 54, 54, 52, 50, 49, 56, 48, 98, 49, 100, 102, 98, 101, 102, 99, 48, 99, 48, 99, 57, 56, 54, 55, 56, 102, 101, 98, 102, 98, 102, 99, 50, 99, 54, 50, 98, 6, 116, 112, 61, 85, 68, 80, 8, 118, 110, 61, 54, 53, 53, 51, 55, 9, 118, 115, 61, 51, 54, 53, 46, 53, 51, 7, 111, 118, 61, 49, 49, 46, 51, 4, 118, 118, 61, 50, 192, 107, 0, 12, 0, 1, 0, 0, 17, 148, 0, 2, 192, 233, 192, 233, 0, 12, 0, 1, 0, 0, 17, 148, 0, 2, 192, 208, 11, 70, 97, 109, 105, 108, 121, 32, 82, 111, 111, 109, 8, 95, 97, 105, 114, 112, 108, 97, 121, 192, 40, 0, 16, 128, 1, 0, 0, 17, 148, 1, 69, 5, 97, 99, 108, 61, 48, 26, 100, 101, 118, 105, 99, 101, 105, 100, 61, 65, 67, 58, 66, 67, 58, 51, 50, 58, 54, 56]) } Error Error { "message": "Field or method \"NaN\" does not already exist, and can't create it on Uint8Array", "type": "Error", "stack": " at line 14 col 33\n len = bytes[offset++];\n ^\nin function \"parseDnsName\" called from line 1 col 41\nvar dnsName = parseDnsName(bytes, offset);\n ^\nin function \"parseRecord\" called from line 3 col 61\n...bytes, ctx.offset, isAnswer);\n ^\nin function \"handleRecords\" called from line 19 col 43\n handleRecords(ctx, packet.answerRRs, 1);\n ^\nin function \"processMessage\" called from line 9 col 18\n });\n ^\n" }

 at line 14 col 33
 len = bytes[offset++];
 ^in function "parseDnsName" called from line 1 col 41
 var dnsName = parseDnsName(bytes, offset);
 ^in function "parseRecord" called from line 3 col 61
 ..bytes, ctx.offset, isAnswer);
 ^in function "handleRecords" called from line 19 col 43
 handleRecords(ctx, packet.answerRRs, 1);
 ^in function "processMessage" called from line 9 col 18

Also run on linux. and got this:

REC {
  "name": "BRAVIA-4K-2015-930a430dbc269b6151b66d2470cca3a7._googlecast._tcp.local",
  "type": 33, "clazz": 32769, "len": 57, "ttl": 120,
  "data": { "port": 8009,
    "host": "930a430d-bc26-9b61-51b6-6d2470cca3a7.local"
   }
 }
ASSERT(jsvGetLocks(var) < 15) FAILED AT src/jsvar.c:646
  #1[r12,l2] Object {
    #2[r1,l2] Name String [1 blocks] "\xFF"      #3[r1,l2] Object {
        #6[r1,l2] ASSERT(jsvGetLocks(var) < 15) FAILED AT src/jsvar.c:632
EXITING.

[opichals]

@wilberforce Thanks for the debug info. I am able to reproduce as well. The NaN case seems to be caused by the fact that the dgram message is not a complete DNS packet.

Currently the UDP datagrams are simply cut to the max size of the net->chunkSize length. This would require implementing SO_RCVBUF setup so that boards that have enough RAM could receive the whole data.

As for the jsvGetLocks() assert I am suspecting the parseDnsName() just does too much recursion and would need to be rewritten not to.

[opichals]

I tested the createSocket() recvBufferSize option implementation for linux target only for now (sockopt(SO_RCVBUF)/ in https://github.com/espruino/Espruino/compare/master...opichals:createSocket-recvBufferSize-option but I am not completely sure we could afford iterating over all the connections inside every socketIdle() to find the max of all recvBufferSize options used.

Also I think the SO_RCVBUF support is compile-time option for the ESP-IDF and for the ESP32 and I am not sure about whether it is enabled by default. Not to mention other boards and SDKs.

@gfwilliams What do you think?

[wilberforce]

The NaN case seems to be caused by the fact that the dgram message is not a complete DNS packet.

Can we be brutal here and in the long case just ignore the data?

Also I think the SO_RCVBUF support is compile-time option for the ESP-IDF and for the ESP32 and I am not sure about whether it is enabled by default.

I can take a look and compile in if necessary

[opichals]

The NaN case seems to be caused by the fact that the dgram message is not a complete DNS packet.

Can we be brutal here and in the long case just ignore the data?

Yes, that's what I am planning on doing.

[opichals]

@wilberforce Updated the gist to avoid recursion in parseDnsName()(getting rid of the ASSERT(jsvGetLocks(var) < 15) issues) and also to check for the packet size inside.

So hopefully the errors seen earlier are gone.

[opichals]

The dgram.createSocket({ recvBufferSize: 1024 }) option landed in master in https://github.com/espruino/Espruino/pull/1472

[wilberforce]

Currently the UDP datagrams are simply cut to the max size of the net->chunkSize length. This would require implementing SO_RCVBUF setup so that boards that have enough RAM could receive the whole data.

I found the option so can compile into the libs for the ESP-idf in the build tools.

I have had some side issues with sockets - the default mtu is something like 1432 - and by default the buffers are 4x this for each connection - so with 10 connections you are looking at a lot of heap use. What seems to occur is that we run out of heap and sockets so op responding. Same code,appears ok on Linux.

Should I try a build with Mtu 536 like the chunk size and 2x the buffers? Also the number of connections seems to be 10 but the number of sockets 16 - I would have thought these should be the same?

[wilberforce]

https://github.com/espruino/Espruino/pull/1473

This adds the SO_RSVBUF support in the provisioning script for Esp32

[wilberforce]

@opichals fyi

I have added the native mDns to the esp32 port.

https://github.com/espruino/Espruino/pull/1585

[opichals]

@wilberforce Actually I am nearing the submission of the gist. I have been testing it for a while here for NTP server discovery (the resolvePTR function).

[mikabytes]

It's been three years. Is there any progress on getting mDNS to work for Espruino esp8266?

Only mention in the docs is the use of wifi.setHostname, which was not enough. I can't make outgoing requests to .local domains, nor can the esp8266 be discoverable by avahi-browse -a.