Support VS dev tunnels with Aspire

maryamariyan commented 1 year ago

Description:

This is a feature request to ensure we support dev tunnels with aspire, and that the workflow is tested end to end.

Potential key blocker with Aspire (needs investigation)

For the solution I describe below, I had to hardcode the BaseAddress endpoint within the TeamsApp, in order to make it communicate properly with another ApiService existing in the Aspire solution.

For this sample, the limitation might just be with how Microsoft Teams Apps are configured and setup in general and not necessarily with Aspire. Regardless, it would be good to use this use case to test how Aspire performs when using Dev Tunnels.

Todo:

[x] Share a link to a solution that uses Dev Tunnels (e.g. using Microsoft Teams App in an Aspire solution) UPDATE: The sample is now in the Azure-Samples/openai repo.

Experimentation done:

To test this end to end, I tested out how it would work to add a Microsoft Teams (chat bot) App as a service to an Aspire solution. Since the features are in preview, there were a few band-aids workarounds that I needed to to apply in order to make the entire end-to-end work.

In this issue, I describe steps via VS for making an solution using dev tunnels. You will notice the steps below are not convenient when using VS features. It is possible that the inconvenient steps are due to limitations with Teams Toolkit. I plan to log issues for the Teams Toolkit as well.

But, as far as Aspire is concerned I think we need to make sure endpoint discovery works well when we are in a Dev Tunnels setting.

Steps to build/run Aspire solution with Dev Tunnels:

Expand for build/run steps

The sample is now in the [Azure-Samples/openai](https://github.com/Azure-Samples/openai/tree/main/End_to_end_Solutions/GithubRepoAssistant) repo. The sample also has a README that goes through steps to run the solution. I ended up writing the following two hardcoded pieces: - [openai/End_to_end_Solutions/GithubRepoAssistant/TeamsApp/Program.cs at main · Azure-Samples/openai](https://github.com/Azure-Samples/openai/blob/main/End_to_end_Solutions/GithubRepoAssistant/TeamsApp/Program.cs#L29-L46) - [openai/End_to_end_Solutions/GithubRepoAssistant/GithubRepoAssistant.ApiService/Program.cs at main · Azure-Samples/openai](https://github.com/Azure-Samples/openai/blob/main/End_to_end_Solutions/GithubRepoAssistant/GithubRepoAssistant.ApiService/Program.cs#L23)

timheuer commented 1 year ago

/cc @sayedihashimi as FYI

timheuer commented 1 year ago

Similar, but not purely 1:1 on the scenario, but Aspire in a DevContainer also suffers from some of the port tunneling issues here.

samsp-msft commented 1 year ago

YARP could be another solution here. https://github.com/microsoft/reverse-proxy/issues/1618

timheuer commented 1 year ago

YARP could be another solution here. https://github.com/microsoft/reverse-proxy/issues/1618

Dev Tunnels are a managed service providing secure tunnels and direct tools integration. We have to enable it to work with those scenarios and ngrok would be similar here.

Riff451 commented 1 year ago

Hi, thanks for the amazing work with Aspire!

I think it would be nice to make this working with any other similar tool such as ngrok (as already mentioned). This is a quite common situation where a third party running remotely needs to communicate with a local component running in Aspire.

I think it would be also useful to show the public urls in the dashboard if possible.

Thanks.

davidfowl commented 9 months ago

We want to do this but it's out of scope for the initial release. FWIW, it'll be possible to build and experiment with this with the primitives being built in the first version.

Riff451 commented 8 months ago

Hi, I've got something working here with ngrok: https://github.com/Riff451/aspire/tree/luca/ngrok-attempt-container

The api would look something like this:

var ngrok = builder.AddNgrok("ngrok-tunnels", builder.Configuration["Ngrok:AuthToken"]!, 54097, "eu");

var apiA = builder.AddProject<Projects.NgRokEndToEnd_ApiServiceA>("test-api-a")
    .WithNgrokTunnel(ngrok);
builder.AddProject<Projects.NgRokEndToEnd_ApiServiceB>("test-api-b")
    .WithReference(apiA)
    .WithNgrokTunnel(ngrok);

And in the dashboard you'd have something like:

Where you can open the ngrok inspection ui (localhost:54097 above) and check the opened tunnels:

Would it be helpful? I'm not entirely sure I've followed the right path :)

Thanks.

SteveSandersonMS commented 7 months ago

I filed #3813 but then realised it duplicated this issue. Just for tracking, here's the description I posted:

Currently I can't see any sensible way to use VS Dev Tunnels with Aspire, for several reasons:

The thing that causes a dev tunnel to start is launching the project associated with that tunnel (e.g., Ctrl+F5). If you instead launch the Aspire orchestrator, this does not start the dev tunnels associated with the orchestrated projects.
Each time you use the VS UI to enable a dev tunnel for any of the projects in your solution, it turns this on for all the projects. And so you'll now be serving the Aspire dashboard through the dev tunnel, which often is very much not what you want (as it will then be reachable publicly)
In order to turn it off for a project, you need to use the Dev Tunnels config menu (below), but this appears for projects that use the Microsoft.NET.Sdk.Web SDK, but .AppHost uses Microsoft.NET.Sdk. So you can't control it directly for .AppHost.

Rationale

If you want to develop a Teams bot or expose APIs to Power Apps (etc), you would need to expose a dev tunnel for the corresponding API project. It would make sense to be able to do that in one of your Aspire-orchestrated projects.

timheuer commented 7 months ago

@SteveSandersonMS you can manage the tunnel as well using the Dev Tunnels menu and turn off the other ports that are presently getting auto-mapped...this is what I do presently (yes, not awesome, but possible):

What we need (@sayedihashimi @BillHiebert) I believe is:

[ ] Ability to specify a dev tunnel for a specific project (we have this)
[ ] Upon F5 only the specified project(s) gets the tunnel forwarded (right now every project is, including AppHost as Steve points out)
[ ] Maybe a question here -- do we add an additional endpoint to the resource specifying the tunnel URI so that it also shows in the dashboard?

Right now this is possible (sans dashboard display) but manual process. As Steve and @maryamariyan note, this would be good to get the coordination better like what we have in an individual project today.

SteveSandersonMS commented 7 months ago

turn off the other ports that are presently getting auto-mapped...this is what I do presently (yes, not awesome, but possible):

Does those changes persist? I was able to make it work like that once, but then after reopening VS it kindly forgot my preferences and I would have to do a series of things like:

Set the TeamsBot project as startup, and start it, to make the devtunnel start
Shut down that running project
Change the startup project back to .AppHost
Go into the UI for Dev Tunnels and turn off all the unwanted ports

If this is a flow that I'd have to go through each time, I'd rather not use Dev Tunnels and Aspire together😄

It's also unclear to me where these preferences are stored. I think they need to be something you can commit to source control otherwise you have to educate everyone on your team how to configure things in order to run the project.

SteveSandersonMS commented 7 months ago

The most usable workaround right now, I think, is just to launch the dev tunnel from the CLI command:

builder.AddExecutable("my-dev-tunnel", "devtunnel", builder.AppHostDirectory, "host", YourTunnelName);

This is pretty much ideal as it then starts up and shuts down corresponding to the AppHost, and gives you the logs in the same place.

timheuer commented 7 months ago

Does those changes persist?

No they do not (well kinda, but as the ports change with DCP new ports get added)

It's also unclear to me where these preferences are stored. I think they need to be something you can commit to source control otherwise you have to educate everyone on your team how to configure things in order to run the project.

DevTunnels, by design, are individual for the developer. You may not possess the permissions to create/use one in a shared environment. So that is the reason these re per-user right now (and by default when created a private to the individualunless changed)

Yes it's clear we have work here to do to ensure these steps aren't needed and can have a smoother experience as we have with single projects. Just need to think it through and design the expected interactions.

swegele commented 7 months ago

I got DevTunnel working (fairly convenient - the url lasts 30 days).

Thanks to the comment above from @SteveSandersonMS

The most usable workaround right now, I think, is just to launch the dev tunnel from the CLI command

and to this video showing how to use the devtunnel CLI.

Repro steps here to help someone else accomplish this quicker without all the trial and error: I assume you:

are on windows
only have one dev tunnel running with one port (if that is not true for you this should get you going)

Download the "devtunnel.exe" however you want...ms link here
Place the exe in a nice convenient spot like "C:\tools\"
Add the following to your windows system path in Environment Variables "c:\tools"
Now you should be able to simply open powershell or command window and type "devtunnel" and the following will work.
Run devtunnel create -a -l mydevtunnel (the label 'mydevtunnel' is totally optional)
Run devtunnel port create -p <port #> --protocol https (the https is important especially with aspire pre 5)
Now put this in your aspire AppHost Program.cs builder.AddExecutable("my-dev-tunnel", "c:/tools/devtunnel.exe", builder.AppHostDirectory, "host"); (the name 'my-dev-tunnel' can be whatever)

That last line number 7 - simply tells aspire to run the devtunnel.exe with the "host" argument. DevTunnel.exe will see your one tunnel and use it by default.

When you run the aspire project - you can look at the console logs for "my-dev-tunnel" and you will see the URL. Copy the URL wherever you need to have your app use that as the base URL. Example:

var apiService = builder.AddProject<Projects.AspireApiService>("apiservice")
    .WithEnvironment("WEBHOOK_BASEURL", "https://r2s....usw3.devtunnels.ms/")

For the next 30 days (as long as the tunnel you defined is active) you can re-use that same URL. Then after than, probably just create a new tunnel or refresh the existing? and get the new url and viola.

HTH someone

joperezr commented 5 months ago

@davidfowl is this something we want to do in 8.1? If so, we need to understand what we are enabling and sync with @timheuer and @vijayrkn to see what work will flow from this.

timheuer commented 5 months ago

Assigning to @bradygaster for the experience design needs first before we assign any engineering work.

bradygaster commented 5 months ago

Will work through a few sample iterations and understand the end-to-end on this and have some ideas for a design by eow. Right now I don't hate the idea @SteveSandersonMS and @swegele discuss. I'd just need to know if this would be a component or an extension we'd want to ship in Aspire. CC @joperezr for opinions on that.

davidfowl commented 5 months ago

High level commentary:

The design should take into account the fact that it needs to work in all sorts of environments (VS, VS Code, CLI, WSL). We can choose to stage them, but we don't want any dead ends.
It would be interesting to understand if we want deeper integration with DCP here. Could the dev tunnel be spun up by DCP itself? DCP already spins up proxies and this is another proxy. Maybe worth investigating there.
Making the urls in dashboard show the proxy urls means the data is available to the app host (this is the problem with have today with code spaces).

The last bullet is what pushes me to make this a feature of DCP.

cc @karolz-ms @danegsta

bradygaster commented 5 months ago

The BASIS underpinning should enable us to party in non-VS areas. So I think the goal is possible. I'd just need to update my understanding of the underlying APIs with @sayedihashimi and @timheuer.

swegele commented 5 months ago

@bradygaster let me know if you need help testing or whatev.

andrew-vdb commented 5 months ago

I have it working with Aspire, its not intuitive Steps

Create and enable dev tunnel by selecting aspnet core project
Go back to Aspire dashboard project, run it will run aspire dashboard with dev tunnel url https://{dynamic}-{port}.euw.devtunnels.ms/ the url has certain pattern, simply change the port to what you see in Aspire dashboard that's it

mattleibow commented 5 months ago

You can control the name of everything:

# start dev tunnel
devtunnel delete exciting-tunnel --force
devtunnel create exciting-tunnel --allow-anonymous
# add the REST API port
devtunnel port create exciting-tunnel --port-number 7258 --protocol https
# add the Open Telemetry port (analytics and logs to the dashboard)
devtunnel port create exciting-tunnel --port-number 21031 --protocol https
# add the Aspire Dashboard
devtunnel port create exciting-tunnel --port-number 17048 --protocol https
devtunnel host exciting-tunnel

In my code, I have this:

public static IConfigurationBuilder AddAspireApp(this IConfigurationBuilder builder, Dictionary<string, string> settings, string? devTunnelId = null)
{
    var copy = new Dictionary<string, string>(settings);

    if (!string.IsNullOrWhiteSpace(devTunnelId))
    {
        foreach (var setting in copy)
        {
            if (setting.Value.Contains("localhost"))
            {
                // source format is `http[s]://localhost:[port]`
                // tunnel format is `http[s]://exciting-tunnel-[port].devtunnels.ms`
                var newVal = Regex.Replace(
                    setting.Value,
                    @"://localhost\:(\d+)(.*)",
                    $"://{devTunnelId}-$1.devtunnels.ms$2");

                copy[setting.Key] = newVal;
            }
        }
    }

    builder.AddInMemoryCollection(copy);

    return builder;
}

The usage will be:

builder.Configuration.AddAspireApp(AspireAppSettings.Settings, "exciting-tunnel");

Basically, I create a tunnel with my desired ID and then replace all cases of localhost with the dev tunnel. Now if aspire could auto set up that tunnel... and then with @BretJohnson and the maui-aspire integration have an option for providing the tunnel ID....

This is all working very well and the specific ID I set really helps make things "just work".

karolz-ms commented 5 months ago

@davidfowl @danegsta @bradygaster I think it makes a lot of sense to introduce dev tunnels support at app model + DCP level.

More specifically, we could introduce a new kind of Endpoint into the Aspire app model: a "DevTunnelEndpoint", When DCP encounters this kind of endpoint, instead of (or maybe in addition to-) running a local proxy, it would set up an ephemeral dev tunnel to expose the backend set. Everything else would work as today: once the tunnel is up and running, DCP would publish address & port information about it like for any other endpoints; clients would have these injected through environment variables or startup parameters; Aspire dashboard would display the tunnel address as the endpoint info etc. etc. At shutdown DCP could tear these ephemeral dev tunnels down. If necessary, we could support persistent tunnels too, in a fashion similar to persistent containers.

We would have to figure out how to obtain the credentials to manage dev tunnels in all use cases (various IDEs, command line, various OSes), but there are various possibilities, and prior art, to deal with it.

Also note that exposing a DevTunnelEndpoint does not preclude exposing a regular Endpoint that leverages the same set of backend instances. Which means the clients that need the tunnel would use it, but other clients that are running locally could connect directly (or via local proxy).

CC @BillHiebert @fiveisprime @mitchdenny

mitchdenny commented 5 months ago

I think that having a separate annotation for dev tunnels makes sense which then references the endpoint name.

In the UX both endpoints would be shown and the dev tunnel endpoint might also show a QR code if you hover on top of it.

Whether the AppHost or DCP launches the devtunnel executable is debatable. I'm leaning towards DCP in consideration of future persistent container features.

mitchdenny commented 5 months ago

OK I've had a bit more of a chance to think about this over the weekend. Here is what I think the requirements should be (for DevTunnels at least):

The AppHost launcher (VS, VSCode, dotnet watch, Rider, whatever) should be responsible for creating the DevTunnel "tunnel". This would be set as some kind of environment variable.
We have an AppModel extension something along the lines of `WithDevTunnel("endpointName", options => {}) (second argument optional). This would add an annotation that is used later.
When the AppHost launches DCP etc is started up in the usual way with ports assigned etc.
When we get the allocatedendpoints added we execute devtunnel to forward the ports for endpoints where we have a related devtunnel annotation.
For each port we forward through the tunnel we add a URL via the resource notification service (like we do for Bicep deployments).
We update the Portal to allow developers to hover their mouse cursor over a particular endpoint on a resource, this brings up a tool tip which shows a QR code that you can point your phone at if you happen to be testing a mobile experience (pending usability/accessibily review obviously).

mattleibow commented 5 months ago

For 6, if I have a mobile app that is using REST endpoints, will the tunnel ID be configurable and predictable?

swegele commented 5 months ago

For 6, if I have a mobile app that is using REST endpoints, will the tunnel ID be configurable and predictable?

I am curious about the same. Personally, I'm ok if it follows the current pattern of lasting 1 month...so long as it doesn't totally reset to random for every run/debug operation. That would be a pain for external testers.

timheuer commented 5 months ago

Here is what I think the requirements should b

I think we need to design this a bit more and have some discussions about various models. Specifically, nearly every type of 'tunnel' tech (assuming this might be extensible in @mitchdenny's proposal to those beyond DevTunnel) requires some auth, has some business model, etc. Modeling it in the appmodel would have to take those into account. A portion of these concerns are generally handled by higher level tools (e.g., VS carries current auth for DevTunnel for example). /cc @bradygaster

mitchdenny commented 5 months ago

@timheuer yeah the reason I think it needs to do auth in the higher level tools is because that is how these things seem to operate.

Even ngrok does auth separately.

I think in time we could look at doing auth inside the dashboard somehow or via the dashboard. But I think we'd want to have some of the extensibility story fleshed out for that.

bradygaster commented 5 months ago

Chatted with @sayedihashimi this morning about some ideas here; we're thinking a good place to surface the "enlistment" would be in the App Host Program.cs, so folks could maybe have a RunWithDevTunnel() extension on projects. That could shell out and setup/manage tunnels using the Dev Tunnels CLI. Would folks dig that approach? It would open it up to not-just-VS experiences.

karolz-ms commented 5 months ago

I think it makes a lot of sense @bradygaster. In particular, dev tunnel CLI has experience for authentication and creating tunnels, that would allow app host runtime/DCP to leverage existing tunnels, or create ephemeral ones with minimal complexity and without needing to deal with authentication flows.

A couple of points in response to what @mitchdenny said earlier. As far as I understand his proposal, it puts the code that sets up dev tunnel use for particular Aspire app/service in the app host runtime. My original thinking was that functionality would go into DCP, where all other network management/traffic proxying/port allocation currently resides. Both can be made to work and there is pros/cons to both approaches; we should have a more detailed discussion about these alternatives.

The AppHost launcher (VS, VSCode, dotnet watch, Rider, whatever) should be responsible for creating the DevTunnel "tunnel". This would be set as some kind of environment variable.

I think a good design rule of thumb is to be careful about, and try to minimize, relying on many tools to provide a key piece of the experience, each with its own implementation. This can lead to inconsistent experience and issues. This is also one of the reasons why I like the proposal to rely on dev tunnels CLI, which is maintained by the dev tunnels team and provides a single way to manage tunnels. That is not to say that there is no value in some tools providing "premium" experience, but being able to say that with dev tunnels CLI the basic needs are met, no matter the OS, or how the Aspire app is launched, there is a lot of value in this IMO.

timheuer commented 5 months ago

shell out and setup/manage tunnels using the Dev Tunnels CLI

Is the assumption that this API is a part of Aspire core, but an would fail without the prereq tools (similar to not finding docker/podman)?

mitchdenny commented 5 months ago

I've done some experimentation:

https://github.com/dotnet/aspire/pull/4771

This is by no means a forcing function on direction. I decided to play with this because I'm off this week and was working on a hobby project and decided I needed tunnel support in Aspire :)

I've learned a couple of things:

Using devtunnel.exe is kind of chatty. Lots of slow command invocations to get the tunnels into the state we want.
--allow-anonymous can only be set at the tunnel level, not the port level. The implication here is that if we want to expose multiple endpoints, but only make some allow anonymous access - then we are going to have a tunnel per endpoint.
We may run into some issues with certain containers/resource types where the container needs configuration data on what the actual tunnel endpoint is ... its possible that some resource types just won't like to be tunneled where they are simultaneously accessible via the tunnel and locally.

mitchdenny commented 3 months ago

Moving this to Backlog, but @maddymontaquila @sayedihashimi @karolz-ms we should talk about whether we can put this in 9.0. We've got ideas on how to make it work but it will cut across a few areas.

dotnet / aspire