Can not use any other path but root for the frontend ingress

[michaelajr]

Describe the bug Ingress paths do not work, the frontend MUST be run at the root.

To Reproduce Create an ingress with a path other than /... e.g., /kubecost. you'll get a 404.

If you add a rewrite annotation to the ingress, like /$2, you'll get the main page - but it will spin because all the links and api calls use / as the app base.

Expected behavior I would expect there be a place to configure the frontend to have a base path for the app and all the subpaths and JS calls would work off that.

[michaelajr]

Maybe there is a docker arg to specify the app root?

[dwbrown2]

@michaelajr, thanks for reporting! this is definitely a use case that should be supported. we will investigate, any chance you have an example ingress definition for us to test against?

[michaelajr]

Here's a snippet of what I was doing. The ingress rule worked - as I was able to hit the container. It just returned a 404.

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hosts:
    - mycluster.mycompany.com
  paths:
    - /kubecost
  tls:
  - hosts:
    - mycluster.mycompany.com
    secretName: kubecost-ingress-tls

Then I added a re-write rule so /kubecost would hit the container as /.

...
annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
...

That worked - I got the first page. But all the links on the page, and all the JS calls, were constructed off the root. So the page was calling /api instsead of /kubecost/api. Which makes sense as the page has no understanding of my ingress path.

Usually when UI apps want to support running off a path - they take a flag or setting where the user can specify the basePath, and all links and JS calls are constructed off that basePath.

[michaelajr]

Btw, this is an important issue for us as we run another app at the root.

[dwbrown2]

Thanks, this is helpful. We are investigating now.

As a potential immediate workaround, I believe you should be able to add this endpoint directly on /index (screenshoot below) from any kubecost deployment where the frontend is accessible/functioning.

[michaelajr]

Well - we do not have a centralized installation of Kubecost. So unfortunately this will not help. We are in talks for an enterprise license and would probably move to a centralized set up soon. But for now - we are putting kubecost into one of our clusters to evaluate it. Would really like to run this off our existing FQDN - but it needs to run off a path. Thanks for your help.

[dwbrown2]

We were able to reproduce and a fix is now in progress. This is actually an issue with our cluster “autodiscovery." We confirmed that in our repro environment you can go to the UI and add this same URL as a new cluster and it should work fine.

Also, cluster endpoints can be directly added at install time as well: https://github.com/kubecost/cost-analyzer-helm-chart/blob/master/cost-analyzer/values.yaml#L280

[michaelajr]

So to be clear - I need to run the Kubecost frontend at a path. E.g. /kubecost. Specifying that path for the frontend ingress, should take me to the frontend app (right now I get a 404 from your nginx). Everything then has to work off that path - including all the links and JS API calls in the frontend html.

This seems to be more of an issue with the app's nginx set up. It knowns nothing about the /kubecost path. So it returns a 404. And all links and API calls are to the root /.

We have a single FQDN for our clusters. So all the admin apps that are exposed on the cluster are exposed via paths. We do not want to create a new fqdn for Kubecost to work.

thanks!

[AjayTripathy]

Hi @michaelajr , this makes sense. We're thinking about the solution here in two phases:

1. Use the full path correctly, instead of the root URL, once the application frontend has been loaded. You would still need an annotation for rewrite on your ingress to get the frontend to load, as you've already done.
2. Support adding arbitrary rootURLs to our application without adding annotations/rewrites like nginx.ingress.kubernetes.io/rewrite-target: /$2 We'd expect the first is a quick change and is already in progress, but the second would take longer. Does part 1 solve most of your issue here?

[michaelajr]

Well I can add the rewrite annotation - and get to the index page. But none of the links work on that page as they are all constructed using the root not the path I specified in the ingress. Which makes sense since the app does not know about my ingress and it is running at the root.

The app has to 1) run at the same nginx location as the ingress path, and 2) make all links and calls using that path.

[mrtnfchs]

Hi @AjayTripathy

I think we should not only focus on kubernetes ingress resources only but any ingress gateway (we're using Ambassador). I managed to create a custom prefix path (eg: https://[my-domain]/kubecost/) and reach the cost-analyzer frontend and most of the pages work fine but some of them don't and result in a blank screen (eg: abandoned-workloads.html)

This because, as mentioned earlier, the href's are hard-coded on the path / (eg: <link rel="stylesheet" href="/body.fa6886bd.css">) and regardless of the ingress path configured, it will not be able to load/find these css or js files. I believe better would be to update the href's with ./ (eg: <link rel="stylesheet" href="./body.fa6886bd.css">) or remove the prefix as a whole (eg: <link rel="stylesheet" href="body.fa6886bd.css">)

The pages that do work, have a full external path configured (eg: <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">) or reference local files without / (eg: <script src="configs.js"></script>) which is an option too of course

PS: I love your tool, great work!

[dwbrown2]

@mrtnfchs, great feedback! thanks a ton. niko's pr referenced above will address this. It's expected to be in our next minor release.

For reference, we just confirmed this is resolved using this example Ingress: https://github.com/kubecost/docs/blob/master/ingress-examples.md#non-root-path-example

[michaelajr]

@dwbrown2 @AjayTripathy I have this working. But had to add the following to get Grafana linking correctly.

grafana:
  grafana.ini:
    server:
      domain: mycluster.net
      root_url: https://mycluster.net/kubecost/grafana
      serve_from_sub_path: true

[michaelajr]

@AjayTripathy Also I get errors in the console. And the cost indicator is a warning icon.

[dwbrown2]

Thanks for sharing, Michael. Looks like something else may be going on here given that /model/clusterInfo is returning data but not /model/clusterCosts

Any chance you can share any more info on the response from the latter? If it's a 500 a bug report may help us determine what's going on.

[michaelajr]

@dwbrown2 will do. I noticed this before. Before putting it on a path. It happens using an ingress, but not when using kube proxy. Kinda weird.

[maeghan-porter]

I am wanting to run kubecost on a /kubecost path on an EKS cluster using the AWS ALB Controller. The example you provide here: https://github.com/kubecost/docs/blob/master/ingress-examples.md#non-root-path-example is specific to nginx, if I set up an ingress with the AWS ALB Controller to use the /kubecost path how do I get the app to accept this path? Or has that not been implemented?

[AjayTripathy]

Hi @maeghan-porter , we would need the equivalent of a rewrite on the flavor of Ingress you are using. This may not yet be supported by the AWS ALB ingress. https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1571

[maeghan-porter]

Hi @maeghan-porter , we would need the equivalent of a rewrite on the flavor of Ingress you are using. This may not yet be supported by the AWS ALB ingress. kubernetes-sigs/aws-load-balancer-controller#1571

Thank you for your response. That is a shame. AWS doesn't have plans to support path rewrites on ALBs until Q2 of 2022. I can see that, given the comment above (https://github.com/kubecost/cost-analyzer-helm-chart/issues/322#issuecomment-604362985), there are others wanting to use kubecost on a non-root path that use other types of ingress controllers, so it would be lovely if y'all would consider supporting non-root paths from your application without relying specifically on the nginx ingress controller.

[dwbrown2]

@maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

[maeghan-porter]

@maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

Well it would be ideal if instead of having the ingress controller rewrite the path, which lots of them can't, if we could tell kubecost to accept a non-root path instead.

[AjayTripathy]

Hi @maeghan-porter thanks for that feedback. Feel free to open an issue and we'll look to get that prioritized!

[sivakumark88]

@michaelajr Along with this configuration, do we need to add any ingress routing rules for Grafana ? Just added these in values.yml and update helm chart but grafana not acceesible. Kubecost is accessible from non-root url. http://domain/kubecost but grafana not working in http://domain/kubecost/grafana.

grafana:
  grafana.ini:
    server:
      domain: mycluster.net
      root_url: https://mycluster.net/kubecost/grafana
      serve_from_sub_path: true

[blmeira]

@maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

Well it would be ideal if instead of having the ingress controller rewrite the path, which lots of them can't, if we could tell kubecost to accept a non-root path instead.

This is still an issue today on multiple levels. For security purposes, snippets are blocked on the ingress. Thusly blocking the "solution" from working.

Base app roots should always be configurable, especially in a Kubernetes context (which Kubecost specifically aims for)

[AjayTripathy]

As an update, this works now except for with certain cases of OIDC, which we're investigating.

[jessegoodier]

@maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

Well it would be ideal if instead of having the ingress controller rewrite the path, which lots of them can't, if we could tell kubecost to accept a non-root path instead.

which ingress controller are you using? I may be able to provide a workaround.

[maeghan-porter]

.> > > @maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

Well it would be ideal if instead of having the ingress controller rewrite the path, which lots of them can't, if we could tell kubecost to accept a non-root path instead.

which ingress controller are you using? I may be able to provide a workaround.

The AWS ALB Controller in an EKS cluster. Our workaround was to create the ingress separately outside of the helm install of kubecost.

[jessegoodier]

.> > > @maeghan-porter what specifically are you looking to support? I'm not aware of any place this is blocked on the Kubecost product side, but let me know if see any place where you feel it is.

Well it would be ideal if instead of having the ingress controller rewrite the path, which lots of them can't, if we could tell kubecost to accept a non-root path instead.

which ingress controller are you using? I may be able to provide a workaround.

The AWS ALB Controller in an EKS cluster. Our workaround was to create the ingress separately outside of the helm install of kubecost.

Sounds good, that is exactly what I see most often. Thanks for the quick reply.