search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.32k stars 4.74k forks source link

Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono #82637

Open carlossanlop opened 1 year ago

carlossanlop commented 1 year ago

Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it. Edit. Removed the match string because the error message was matching un-related failures.

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893 Error message validated: Xamarin.Hosting: Failed to connect to port Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 6/28/2023 9:28:49 PM UTC

ghost commented 1 year ago

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

Issue Details
- Affected PR: https://github.com/dotnet/runtime/pull/82554 - Affected queue: `Build tvOS arm64 Release AllSubsets_Mono` - Job result: https://dev.azure.com/dnceng-public/public/public%20Team/_build/results?buildId=182544&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d&l=57 - Artifacts: https://dev.azure.com/dnceng-public/public/_build/results?buildId=182544&view=ms.vss-test-web.build-test-results-tab&runId=3552693&resultId=100001&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab - **console.log** output file repeatedly shows the USB connection error below: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-82554-merge-8bea006df9294c8cbf/System.Runtime.Tests/1/console.76e18c2e.log?helixlogtype=result ``` [13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61) [13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 54050 on the device and the port 54050 (8915) on the mac: 61 ``` It's a bit confusing, since there's a **net.dot.System.Runtime.Tests.log** output file that shows some succeeded System.Runtime test executions. Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it. ```json { "ErrorMessage": "Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)", "BuildRetry": false, "ErrorPattern": "", "ExcludeConsoleLog": false } ```
Author: carlossanlop
Assignees: -
Labels: `arch-arm64`, `blocking-clean-ci`, `area-VM-meta-mono`, `runtime-mono`, `os-tvos`, `Known Build Error`
Milestone: 7.0.x
lewing commented 1 year ago

https://github.com/dotnet/arcade/issues/11683

steveisok commented 1 year ago

Might be better to keep this open since it's tagged with a known build error label. I don't think we need to have blocking clean ci attached to it.

carlossanlop commented 1 year ago

I originally logged this issue for tvOS, but I'm also seeing it in iOS arm64. Here is a fresh example:

[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 52498 on the device and the port 52498 (4813) on the mac: 61
[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 52498 (4813) on device (error: 61)
kotlarmilos commented 1 year ago

https://helix.dot.net/api/jobs/4cbf9a6d-6389-43f9-9e2c-16540105525e/workitems?api-version=2019-06-17

Both tests pass based on the logs, but one of them is unable to establish a connection via TCP, while the other establishes a connection every time. This failure seems to be permanent and could be a good starting point for identifying the issue.

When I attempted to reproduce the error locally, I encountered the same issue. However, I have not opened any ports, which might be a possible cause. I assume that the CI machines have ports 49152-65535 (source https://support.apple.com/en-us/HT202944).

I wonder if certain ports are busy or if there might be a firewall misconfiguration.

@premun Any ideas why the error persists when running the functional test?

/cc @akoeplinger @steveisok

premun commented 1 year ago

@kotlarmilos if this was firewall misconfiguration, I think we should see the same set of machines failing. I don't think this is the case: https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAAx3MuwoCMRAF0N6vGLbabQI+ECy2EgsLBSvrMbmQoJmESaKr+PEu9odzTXo/VsSy+NLLQ0GHKdR9cqBxpN2KWBxdGhrOHEFBqO9SmcxybTi67cZwzg/Up0kZ0g3zUlqMrOEDsqlJ7Qe6venE1gf5HzNRiINSDrCetf4ANsm6/4QAAAA=

Seems like this is not tied to specific machines?

@premun Any ideas why the error persists when running the functional test?

Do I understand it correctly that you have 100% repro with a specific app?

kotlarmilos commented 1 year ago

Host: osx-arm64 Device: iPhone iOS 16.5.1 Xcode: 14.3.1 (14E300c)

  1. Install xharness locally (I don't believe it has anything with the repro, that is just my local setup):

    # Using bash on Linux/MacOS
curl -L https://aka.ms/get-xharness | bash -

  2. Download the bundle and place it in a test directory. tracing_eventpipe_tcp_issue.zip

  3. Run the following script from the local directory:

    ./xharness apple test --app "<LOCAL_DIRECTORY>/tracing_eventpipe_tcp_issue.app" --target "ios-device" --output-directory "<LOCAL_DIRECTORY>/output" --set-env=MONO_APPLE_APP_ENTRY_POINT_LIB_NAME=testdir-buffersize/buffersize.dll --set-env=MONO_APPLE_APP_ASSEMBLY_LOAD_PREFIX=testdir-buffersize --xcode /Applications/Xcode.app -v --launch-timeout 00:05:00

The test should start executing and may fail (due to other reasons), but the TCP connection is not established. The app fails locally with

fail: Application test run crashed
      TCP Tunnel Connection Failed to connect to TCP port
kotlarmilos commented 1 year ago

For the apple test command, XHarness expects the application to contain a TestRunner which is a library you can find in this repository. This library executes unit tests similarly how you would execute them on other platforms. However, the TestRunner from this repository contains more mechanisms that help to work around some issues (mostly in Apple platforms).

The runtime test did not utilize the TestRunner, and it was expected to fail. After further investigation and monitoring of the CI, it appears that the TCP failures might be related to the app. If such a failure occurs, ensure that the app functions correctly or consider using apple run commands.

ericstj commented 1 year ago

Removed [7.0] from the title since this appears to be happening across branches.

kotlarmilos commented 1 year ago

@ivanpovazan If you try to run the library tests using the apple run command, please share your results. I suspect the issue is either app-specific or related to the AppleTestRunner. We haven't observed this problem in runtime tests that utilize the apple run command.

ivanpovazan commented 4 months ago

@kotlarmilos and myself are looking into this.

matouskozak commented 3 months ago

Removing the Xamarin.Hosting: Failed to connect to port match string because it's too general and it's matching un-related failures. I'll keep the issue fyi: @ivanpovazan @kotlarmilos

jeffschwMSFT commented 2 months ago

removing 'blocking-clean-ci' as it has not failed in 30 days

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0