OTA network ports show in all 1.8.X versions but not in any 2.X versions of IDE

HomeSpan commented 2 years ago

Describe the problem

I am using the ArduinoOTA library that is included with Arduino-ESP32 (version 2.0.2) and under Arduino 1.8.19 every works fine. I can see all of my ESP2 devices under the Port menu in 1.8.19. I can also see the MDNS broadcasts are working. However, none of the devices show up under Arduino IDE 2.0.0 RC 6 (or any previous versions). Based on other posted issues it appears that there was some problem with showing device names, but nothing about the lack of any devices showing up at all. Are there other MDNS items that must be broadcast that are required by 2.0.0 that were not required by 1.8.19? Or perhaps a preference setting in IDE 2.0.0 that enables network ports to be shows?

To reproduce

Compile and load ArduinoOTA->BasicOTA Example from Arduino-ESP32. After running, board shows up as network port under IDE 1.8.19, but does show up an network port under 2.0.0 RC6.

Expected behavior

Expected board to show up under network port within 2.0.0 RC 6.

Arduino IDE version

2/0/0

Operating system

macOS

Operating system version

12.3.1

Additional context

No response

Issue checklist

[X] I searched for previous reports in the issue tracker
[X] I verified the problem still occurs when using the latest nightly build
[X] My report contains all necessary details

datort commented 2 years ago

I'm experiencing the same issue. Arduino 1.8.x works like a charm and lists my NodeMCU ESP boards for OTA update right after the start, but 2.0.0-rc6 doesn't show any board at all. Running the Arduino 2.0.0-rc6 IDE on an Intel Mac.

per1234 commented 2 years ago

Thanks @HomeSpan and @datort.

I think the best way to proceed with the investigation is to eliminate any unnecessary factors from the equation. So I'm going to ask you to to an experiment and report your results.

The network port discovery is done using a tool named mdns-discovery. If the problem still occurs when you use that tool directly, then we can eliminate the Arduino IDE and Arduino CLI from consideration as being related to the problem.

❗ NOTE: The following instructions will not solve the problem. They are only intended to gather additional information.

I'll provide the instructions here:

During all this, keep an eye out for anything that doesn't match the expected behavior as described at each step.

Open your Applications folder.
Open Utilities.
Double-click on "Terminal".

Run this command:

ls ~/Library/Arduino15/packages/builtin/tools/mdns-discovery

You will now see a list of the versions of mdns-discovery which are installed on your computer. Take note of the highest version on the list (recent versions of Arduino IDE 2.x always use the newest one) For example, if you see output like this:
```
0.9.2   1.0.2   1.0.5
```
then the newest version is "1.0.5".
Run the following command:
```
~/Library/Arduino15/packages/builtin/tools/mdns-discovery/<newest version>/mdns-discovery
```
(where <newest version> is the version number you found at step 3) mdns-discovery will now start. There won't be any obvious sign of this other than that there is no longer a command prompt at the cursor in the terminal.
Type the following command:
```
HELLO 1 "arduino-cli 0.21.0"
```
Press the Enter key. You should now see a response printed exactly like this:
```
{
 "eventType": "hello",
 "protocolVersion": 1,
 "message": "OK"
}
```
Unplug your Arduino board from your computer if you have it plugged in.
Type the following command:
```
START_SYNC
```
Press the Enter key. You should now see a response printed exactly like this:
```
{
 "eventType": "start_sync",
 "message": "OK"
}
```
You might also see some additional objects in the output depending on which network ports are available on your computer.

Plug your Arduino board into your computer You should eventually see a response printed that looks something like this:

{
 "eventType": "add",
 "port": {
   "address": "192.168.254.127",
   "label": "esp32-b4e62dbf693d at 192.168.254.127",
   "protocol": "network",
   "protocolLabel": "Network Port",
   "properties": {
     ".": "node32s",
     "auth_upload": "no",
     "board": "node32s",
     "hostname": "esp32-b4e62dbf693d.local.",
     "port": "3232",
     "ssh_upload": "no",
     "tcp_check": "no"
   }
 }
}

ⓘ This is only an example of what you might see. The output will be different depending on the board you connected.

Unplug your Arduino board from your computer You should eventually see a response printed that looks something like this:

{
 "eventType": "remove",
 "port": {
   "address": "192.168.254.127",
   "label": "esp32-b4e62dbf693d at 192.168.254.127",
   "protocol": "network",
   "protocolLabel": "Network Port",
   "properties": {
     ".": "node32s",
     "auth_upload": "no",
     "board": "node32s",
     "hostname": "esp32-b4e62dbf693d.local.",
     "port": "3232",
     "ssh_upload": "no",
     "tcp_check": "no"
   }
 }
}

ⓘ This is only an example of what you might see. The port data should be the same as the "add" event you saw when you plugged the board in.

You should continue to see the same results if you repeat steps (10) and (11).

Once you are done with your experiments with the "mdns-discovery" tool, follow these instructions to exit:

Type the following command:
```
STOP
```
Press the Enter key. You should now see a response printed exactly like this:
```
{
 "eventType": "stop",
 "message": "OK"
}
```
Type the following command:
```
QUIT
```
Press the Enter key. You should now see a response printed exactly like this:
```
{
 "eventType": "quit",
 "message": "OK"
}
```
You should now be back at the shell command line.

datort commented 2 years ago

Hey @per1234,

thanks for your support. I tried this but failed at 10. with this after the "START SYNC":

{
  "eventType": "start_sync",
  "message": "mdns lookup error: write udp6 [::]:52509-\u003e[ff02::fb]:5353: sendto: no route to host",
  "error": true
}
{
  "eventType": "start_sync",
  "message": "mdns lookup error: write udp6 [::]:58447-\u003e[ff02::fb]:5353: sendto: no route to host",
  "error": true
}
{
  "eventType": "start_sync",
  "message": "mdns lookup error: write udp6 [::]:54128-\u003e[ff02::fb]:5353: sendto: no route to host",
  "error": true
}
{
  "eventType": "start_sync",
  "message": "mdns lookup error: write udp6 [::]:60031-\u003e[ff02::fb]:5353: sendto: no route to host",
  "error": true
}
....

This loops all over again. Any suggestions? Shall I continue?

Kind Regards

PS: It's v1.05 for the discovery tool

per1234 commented 2 years ago

Thanks @datort! That is very useful. I can see that you are experiencing the issue tracked here: https://github.com/arduino/mdns-discovery/issues/24

https://github.com/arduino/mdns-discovery/issues/24 is almost entirely lacking in any useful detail, but I can see from a reference to it in another issue that it does indeed produce errors that follow the same format: https://github.com/arduino/arduino-cli/issues/1543#issuecomment-1006510542

So we know that this is the result of a deficiency in the mdns-discovery tool for you, and not anything specific to the Arduino IDE 2.x code base. I'll wait for the output from @HomeSpan to see whether their port discovery failure is also caused by the same.

Sadly, I am not aware of any workaround for the issue other than using Arduino IDE 1.x until there is a real fix.

HomeSpan commented 2 years ago

I just tried and have exactly the same issue - the v1.0.5 version loops with the same error message when I issue START_SYNC.

However, I also have v0.9.2 on my system and when I ran that version it did not loop in step 10 - it simply responded with start_sync being OK (in JSON format). To try this version with the Arduino IDE I deleted the 1.0.5 directory, thinking that the IDE would automatically use version 0.9.2, but it appears that the Arduino 2.0 IDE automatically installs version 1.0.5 during startup if that version is not found.

To get around this, I renamed the file mdns-discovery in the 1.0.5 folder to something else, and I copied the mdns-discovery executable file from the 0.9.2 folder into the 1.0.5 folder. I then restarted the Arduino 2.0 IDE and...it worked! All my devices that broadcast OTA now correctly show up in the Arduino 2.0 IDE as network ports.

As noted in some other threads, the only info provided is the IP address of each board, not the hostname (which is provided in Arduino 1.8.19). I suppose this is the issue that was fixed in v1.0.5, but something else must have broke that now causes it to completely fail, whereas v0.9.2 still works (though with the limitations of only showing the IP address).

Can a bug-report be created for v1.0.5?

per1234 commented 2 years ago

Thanks for the information and for sharing the workaround @HomeSpan!

Since we have determined that it is not directly caused by Arduino IDE 2.x, I have transferred the issue to the mdns-discovery repository's issue tracker.

sebromero commented 2 years ago

@per1234 Let me know if you need me to test something.

datort commented 2 years ago

Is there anything more I could provide to help get this sorted?

per1234 commented 2 years ago

Thanks all. I am mostly in the triage role here and don't know much about the subject. It will likely be very helpful for the developers to have the assistance of those who are able to reproduce the issue once the time comes for a more in depth investigation and fix.

If you are interested in going further with this, you can try to bisect the problem. @[]()HomeSpan found that the problem occurs with the 1.0.5 release, but not with the 0.9.2 release. That is useful, but it still leaves a large range of changes that might have introduced the problem:

https://github.com/arduino/mdns-discovery/compare/0.9.2...v1.0.5

The low hanging fruit would be to download the binaries for each of the releases from that range and bisect to the release. You would simply repeat the procedure I described in my previous reply, except using the mdns-discovery binary downloaded from the release instead of from the ~/Library/Arduino15/packages/builtin/tools/mdns-discovery folder.

The binaries are available for download from the "Assets" section of each of the release pages:

https://github.com/arduino/mdns-discovery/releases

After that, if you want to get a bit more fancy by bisecting to the specific commit. That will require checking the repository out to each commit in the range between the last "good" release and the first "bad" release, then building the project from the source code using Go.

The bisection might also be helpful for those who want to use the workaround described by @[]()HomeSpan because you could use the most recent available "good" version as the replacement for the "bad" 1.0.5.

HomeSpan commented 2 years ago

The issue begins with release 1.0.0, which seems to be a complete refactoring of 0.9.2. I noticed in the source code the v1.0.0 tries to consume an IP6 address. My boards are no broadcasting an IP6 address (only IP4), and the error message @datort noted above is related to IP6. Perhaps that is the source of the problem?

Maarkk commented 2 years ago

per1234 - What do you mean by "plug/unplug" Arduino board? My boards are just network devices, visible as network ports in Arduino IDE 1.x but not in most recent Arduino IDE 2 (i cannot physically plug them into main computer ;)) mdns-discovery worked ok till 10 - it showed "ok", but no other results...

per1234 commented 2 years ago

Hi @Maarkk I apologize for the confusion. The instructions were based on the ones I previously wrote for serial discovery troubleshooting. In this case, instead of:

Unplug your Arduino board from your computer if you have it plugged in.

I should have written this:

Disconnect your Arduino board from its power source if you have it powered.

Instead of:

Plug your Arduino board into your computer

I should have written:

Power your Arduino board

Instead of:

Unplug your Arduino board from your computer

I should have written:

Disconnect your Arduino board from its power source

Maarkk commented 2 years ago

Thanks - that clears the thing :) Anyway, symptoms are different in my case - no results from mdns-discovery (checked with Bonjour tool for Windows - it shows Arduino nodes/services).

per1234 commented 2 years ago

no results from mdns-discovery

I have the same problem on my Windows machine. It works perfectly on my Linux machine though.

datort commented 2 years ago

A few things i've learned the other day concerning the initial issue I'd like to share:

For me it turns out, that it's really just IPV6 related. The issue is the same on my Mac and Windows machine with mdns-discovery v1.0.5. After an update, my router supports (favors?) IPV6 addresses even in my local network. While Macs/PCs and the ESP board have their IPV4 addresses, Mac/PC also get's an IPV6.

If I switch off the IPV6 support in my router and let the Mac/PC update their DHCP-given addresses, Arduino IDE 2.x is able to find the boards. If have some more issues with Arudino IDE after that, but's probably another issue or some research on my side.

mcspr commented 2 years ago

Any updates? I have also noticed a weird issue running on Windows; mDNS packets only go to localhost interface, and nothing goes to the actual real network when running arduino-cli board list.

Maybe OT: looking at internals of arduino-cli and mdns-discovery... hashicorp/mdns example from README does work when I write a little program and select some specific interface via net.InterfaceByName() / net.InterfaceByIndex(). However, it does not work without specifying one, like it happens here.

whitevast commented 1 year ago

IDE version 1 always sees ESP-LINK boards, IDE version 2 can SOMETIMES see them. But not for long... If I change mdns-discovery.exe on version 0.9.2, the detection is always successful.

cmaglie commented 1 year ago

Closing as solved with the release 0.9.2 that has been published weeks ago now.

arduino / mdns-discovery