[BUG] Port-Forwarding not Working

ivanthewebber commented 1 year ago

To Reproduce

Follow these steps: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/

## Commands I ran

## Deploy Basic Example
# https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/
# https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic.yaml
kubectl create -f basic.yaml

# check logs
kubectl logs -f deploy/basic-example

# forward Flink UI
kubectl port-forward svc/basic-example-rest 8081
kubectl port-forward svc/basic-example-rest 8081:8091
kubectl port-forward svc/basic-example-rest 8091:8081

# forward port with cloudshell UI
# browse port

# clean up
kubectl delete flinkdeployment/basic-example

## Deploy Rest-Configured Example
### ADD the following to basic.yaml under flinkConfiguration:
    # # ## Flink Web UI
    # # The port to which the REST client connects to. If rest.bind-port has
    # # not been specified, then the server will bind to this port as well.
    # #
    # rest.port: "8091"

    # # The address to which the REST client will connect to
    # #
    # rest.address: localhost

    # # Port range for the REST and web server to bind to.
    # #
    # #rest.bind-port: 8080-8090

    # # The address that the REST & web server binds to
    # # By default, this is localhost, which prevents the REST & web server from
    # # being able to communicate outside of the machine/container it is running on.
    # #
    # # To enable this, set the bind address to one that has access to outside-facing
    # # network interface, such as 0.0.0.0.
    # rest.bind-address: 0.0.0.0

kubectl create -f rest.yaml

# check logs
kubectl logs -f deploy/basic-example

# forward Flink UI
kubectl port-forward svc/basic-example-rest 8091

# attempt access at proxy port

# try this instead
kubectl port-forward --address 0.0.0.0 svc/basic-example-rest 8091:8091

# attempt access at proxy port

# clean up
kubectl delete flinkdeployment/rest-example

Observed Behavior

Unable to load anything more than the favicon. Port is not being forwarded correctly. Screenshot (373) Screenshot (374)

GET https://gateway14.westus.console.azure.com/n/cc-e48447c7/cc-e48447c7/proxy/8091/assets/favicon/manifest.json 403 (Forbidden)

<html>
    <body>
        Unauthorized. <a href='javascript:window.open("https://shell.azure.com", "_blank", "toolbar=no,scrollbars=yes,resizable=yes,menubar=no,location=no,status=no")'>Sign-in</a>
        to authenticate and refresh this page.
    </body>
</html>

Expected behavior

Port should be forwarded similar to behavior on local device or ws2. Should be able to access Flink UI dashboard.

Is this specific to Cloud Shell?

Yes

Interface information

Edge on Windows 11. Shell on https://ms.portal.azure.com/#home.

Additional context

Setting kubernetes.rest-service.exposed.type: LoadBalancer still works, but is an unacceptable solution for our production AKS.

theJasonHelmick commented 1 year ago

Hi @ivanthewebber -- One option is to open the required port through the graphical using the toolbar button furtherest to the right "Web Preview" --

Let me ask @robins1212 if there is an automated or command way to do this.

robins1212 commented 1 year ago

@ivanthewebber do you see any cookies included on the request to the forwarded port?

The error you are seeing indicates that the port is open, but we cannot authenticate your request. If you follow the sign-in link and after the shell loads in the new tab refresh the web preview it should come up. If you have cookies disabled the port forwarding will not work.

@theJasonHelmick can we make sure the docs have guidance on this?

ivanthewebber commented 1 year ago

Thanks @theJasonHelmick, the web preview was what I was trying. @robins1212, as my screenshot shows nothing renders, the page is blank, there is no sign-in link to follow. I can see the same cookie that you showed. Screenshot (375)

ivanthewebber commented 1 year ago

I can access the minified javascript sources by clicking on the links in the error messages, but I think they are not loading or executing due to the MIME type error (shown in one of the screenshots)

theJasonHelmick commented 1 year ago

Hi @ivanthewebber -- We need to further investigate this and report back. I'll follow up with @robins1212

ivanthewebber commented 1 year ago

Thanks, it's been a couple weeks, any update? I'm glad this is getting investigated, it's important for my team.

theJasonHelmick commented 1 year ago

Hi @ivanthewebber - We have begun an investigation and believe we understand the issue. We are currently investigating a proper solution. I don't have a timeline yet for a fix.

ivanthewebber commented 9 months ago

I noticed some updates (like being able to use PowerShell or Bash), so I tested it again but it still doesn't work. I also noticed this related issue: https://github.com/Azure/CloudShell/issues/372

Azure / CloudShell