Allow serving requests from a subpath (for reverse proxying)

jamietanna commented 5 months ago

I'm looking at using this project as part of another (https://dmd.tanna.dev) in which I'm looking to reverse proxy this project under a sub-path.

This can be seen in i.e. https://dependency-management-data-example.fly.dev/datasette/dmd?sql=select+*+from+metadata which reverse proxies from the datasette process' HTTP server under a sub-path.

I believe this will require changes in this project to enable it, by injecting in the subpath to i.e. /assets URLs, which would then be i.e. /sql-studio/assets

jamietanna commented 5 months ago

Asked in Discord for more of a hand https://discord.com/channels/804011606160703521/1254325457977999402

jamietanna commented 5 months ago

@frectonz any thoughts on how we may be able to achieve this? This would be a huge feature for me, and I'd then be able to start using it alongside my project's hosted web application

I'm considering - for the meantime - setting up a fork of the project / steps how to configure a local build that can be hosted at a subpath, but would love if it were first-class support

frectonz commented 5 months ago

This is a good idea, I think it would good to implement. I just have a couple of questions.

What's best way for me to setup a reverse proxy, to test this feature?
Do you have an idea of the changes that would need to be made to implement?

I don't think i have completely understood the problem.

jamietanna commented 5 months ago

Re:

What's best way for me to setup a reverse proxy, to test this feature?

Of course!

So to give a demo of the issue I'm facing, which leads to why we'd want to implement this, you can run the following:

sql-studio sqlite preview
# then, in another terminal
go run .

For the following Go code:

Code

(Sorry it's in Go :stuck_out_tongue:) `go.mod`: ``` module example go 1.20 ``` `main.go`: ```go package main import ( "log" "net/http" "net/http/httputil" "net/url" ) func main() { u, err := url.Parse("http://localhost:3030") if err != nil { log.Fatal(err) } proxy := httputil.NewSingleHostReverseProxy(u) mux := http.NewServeMux() mux.Handle("/sql-studio", proxy) log.Fatal(http.ListenAndServe(":8080", mux)) } ```

You'll notice if you browse to http://localhost:8080/sql-studio that we don't receive any response, just a blank page.

If you check the browser's console, you'll notice i.e.

sql-studio Loading module from “http://localhost:8080/assets/index-Dc4xAj-D.js” was blocked because of a disallowed MIME type (“text/plain”).
sql-studio The resource from “http://localhost:8080/assets/index-DpDMcPjm.css” was blocked due to MIME type (“text/plain”) mismatch (X-Content-Type-Options: nosniff).

You can see when viewing the source for http://localhost:8080/sql-studio that the URLs for static assets are set to i.e. /assets/index-Dc4xAj-D.js

view-source:http://localhost:8080/sql-studio

```html SQL Studio

```

However, in the case that we're proxying the app, we would want to have all URLs prefixed with /sql-studio.

jamietanna commented 5 months ago

Re:

Do you have an idea of the changes that would need to be made to implement?

I believe we need to make two changes:

modify the sql-studio Rust code to provide a new argument, i.e. base-url which would allow specifying i.e. --base-url /sql-studio to match the above example
pass that base URL, if provided, to the frontend HTML

This would effectively make the following change to the returned HTML:

 <!doctype html>
 <html lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
-    <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
-    <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
-    <link rel="manifest" href="/site.webmanifest" />
+    <link rel="apple-touch-icon" sizes="180x180" href="/sql-studio/apple-touch-icon.png" />
+    <link rel="icon" type="image/png" sizes="32x32" href="/sql-studio/favicon-32x32.png" />
+    <link rel="icon" type="image/png" sizes="16x16" href="/sql-studio/favicon-16x16.png" />
+    <link rel="manifest" href="/sql-studio/site.webmanifest" />

     <title>SQL Studio</title>

     <style>
       @font-face {
         font-family: "JetBrains Mono";
         font-weight: 100 800;
-        src: url(/JetBrainsMono.ttf);
+        src: url(/sql-studio/JetBrainsMono.ttf);
       }
     </style>
-    <script type="module" crossorigin src="/assets/index-Dc4xAj-D.js"></script>
-    <link rel="stylesheet" crossorigin href="/assets/index-DpDMcPjm.css">
+    <script type="module" crossorigin src="/sql-studio/assets/index-Dc4xAj-D.js"></script>
+    <link rel="stylesheet" crossorigin href="/sql-studio/assets/index-DpDMcPjm.css">
   </head>

   <body>
     <div id="root"></div>
   </body>
 </html>

I'm not sure if we'd want to do that by adding something into the UI (asked in the Vite Discord, no answer yet) or make it so Warp performs pre-processing on the built UI code and injects that parameter

jamietanna commented 5 months ago

I've just tried to do some (fairly cursed) server-side rewriting in the reverse proxy, but unfortunately that doesn't quite work.

I've used the following code:

```go package main import ( "bytes" "io" "log" "net/http" "net/http/httputil" "net/url" "strconv" "strings" ) func main() { u, err := url.Parse("http://localhost:3030") if err != nil { log.Fatal(err) } proxy := httputil.NewSingleHostReverseProxy(u) originalDirector := proxy.Director proxy.Director = func(r *http.Request) { r.URL.Path, _ = strings.CutPrefix(r.URL.Path, "/sql-studio") originalDirector(r) } proxy.ModifyResponse = func(r *http.Response) error { b, err := io.ReadAll(r.Body) if err != nil { return err } defer r.Body.Close() // this is pretty hacky, but as a step towards https://github.com/frectonz/sql-studio/issues/16 in the meantime of upstream support b = bytes.Replace(b, []byte("href=\"/"), []byte("href=\"/sql-studio/"), -1) b = bytes.Replace(b, []byte("src=\"/"), []byte("src=\"/sql-studio/"), -1) b = bytes.Replace(b, []byte("url(/"), []byte("url(/sql-studio/"), -1) b = bytes.Replace(b, []byte("/api"), []byte("/sql-studio/api"), -1) r.Body = io.NopCloser(bytes.NewReader(b)) r.ContentLength = int64(len(b)) r.Header.Set("Content-Length", strconv.Itoa(len(b))) return nil } mux := http.NewServeMux() mux.Handle("/sql-studio/", proxy) log.Fatal(http.ListenAndServe(":8080", mux)) } ```

However, this doesn't work as a number of the URLs are rendered client-side by React.

So by running this code, we can see a 404 page rendered from SQL studio (as the URL doesn't match a known path) and clicking i.e. Overview works but the URL is then set to / instead of /sql-studio/, and clicking Tables returns in an error.

jamietanna commented 5 months ago

I believe this "only" requires two places for this URL change to happen:

/api URLs need to be amended to i.e. /new-path/api
any UI assets need to be amended to i.e. /new-path/site.webmanifest

The UI side of things is going to be more complex

jamietanna commented 5 months ago

[!CAUTION] Drastic proposal

A rather drastic solution, but one that would simplify the implementation is to always serve off a subpath, i.e. so we would always be on /sql-studio/tables or /sql-studio/api.

This would solve this issue, and make it so it's easier to front by other apps, as well as being clearer on URLs.

But it is also going to break anyone who's got a URL from the previous version.

(We could hopefully add some redirects from i.e. /api to /sql-studio/api, or at least bump the minor version to make folks aware of possible breakage)

And I'm not sure if that's something you'd want to do? Naturally, this is a self-serving ask of me, and I'm happy if we find an alternative, but thought I'd share it as an option, in case it was something you'd be up for.

frectonz commented 5 months ago

334a2a1 is a potential solution to this issue. I tried to do the following two things.

Modify the rust code to serve the app at a specified path configured via the CLI. For example,
```
sql-studio --base-url=http://localhost:8080/sql-studio sqlite preview
```
This will parse out the sql-studio path from the base url and will start the server at localhost:3030/sql-studio/. But i have later removed this implementation since the common way to do this is not to modify where the server host the app but to modify the server view of the path via the reverse proxy. So that the server can host all of the UI at the root level and the reverse proxy will handle adding the /sql-studio prefix.

Take a look at the following Go code to see an example of this.

```golang package main import ( "log" "net/http" "net/http/httputil" "net/url" "strings" ) // errorHandler handles errors in the proxy func errorHandler(w http.ResponseWriter, r *http.Request, err error) { http.Error(w, "Something went wrong", http.StatusBadGateway) log.Printf("Proxy error: %v", err) } func main() { // Parse the URL to proxy targetURL, err := url.Parse("http://localhost:3030") if err != nil { log.Fatalf("Failed to parse target URL: %v", err) } // Create a reverse proxy proxy := httputil.NewSingleHostReverseProxy(targetURL) // Modify the default error handler proxy.ErrorHandler = errorHandler // Custom director for modifying the request proxy.Director = func(req *http.Request) { req.URL.Scheme = targetURL.Scheme req.URL.Host = targetURL.Host req.Host = targetURL.Host // Modify the request path to strip the /sql-studio prefix req.URL.Path = strings.TrimPrefix(req.URL.Path, "/sql-studio") if req.URL.Path == "" { req.URL.Path = "/" } // Add additional request modifications if needed } // Create a ServeMux and handle the proxy mux := http.NewServeMux() mux.Handle("/sql-studio/", proxy) // Start the server log.Printf("Starting reverse proxy server on :8080, forwarding to %s", targetURL) if err := http.ListenAndServe(":8080", mux); err != nil { log.Fatalf("Failed to start server: %v", err) } } ```

Modify the client UI code to work in cases where it's being served at a subpath.
```
sql-studio --base-url=http://localhost:8080/sql-studio sqlite preview
```
After running the above and setting up a reverse proxy that's able to rewrite the path. The client UI will fully be accessible and functional at http://localhost:8080/sql-studio.

frectonz commented 5 months ago

Also will just providing a --base-path=/sql-studio be better than providing a full url --base-url=http://localhost:8080/sql-studio.

jamietanna commented 5 months ago

Amazing thank you @frectonz :rocket: That looks really good!

I think having --base-path would be preferable, as does the --base-url need to be the public URL i.e. I'd be looking at deploying sql-studio at https://dependency-management-data-example.fly.dev/sql-studio, which would mean I'd need to invoke:

sql-studio --base-url=https://dependency-management-data-example.fly.dev/sql-studio sqlite preview

Right?

frectonz commented 5 months ago

which would mean I'd need to invoke:

sql-studio --base-url=https://dependency-management-data-example.fly.dev/sql-studio sqlite preview

Right?

Yes.

If this implementation works for you i will implement the --base-path version and create a new release (0.1.17).

jamietanna commented 5 months ago

Yes, this looks + sounds great, thank you :rocket:

I'll get my own codebase ready to use the new release :grin:

jamietanna commented 5 months ago

Hmm, I've been having a play with this and haven't quite been able to get it working - will retry tomorrow and let you know (likely user error)

frectonz commented 5 months ago

Remember that sql-studio is still serving the files at the root level. For example if you run

sql-studio --base-path /sql-studio sqlite preview

the ui will be served at http://localhost:3030/ not http://localhost:3030/sql-studio . The reverse proxy is gonna have to rewrite the path from for example http://localhost:8080/sql-studio/... to http://localhost:3030/..... Checkout the go code i sent above as an example.

I am gonna close this issue as solved, unless any new problems come up.

jamietanna commented 5 months ago

Thank you, that makes sense, and it's now ready for prime time :clap: https://gitlab.com/tanna.dev/dependency-management-data/-/merge_requests/268 / https://gitlab.com/tanna.dev/dependency-management-data/-/releases/v0.98.0

frectonz / sql-studio

Allow serving requests from a subpath (for reverse proxying) #16