VS Code is broken for Workbench 2023.06.0 for Azure ML

[ianpittwood]

code-server introduced a security change for CORS checks, but unfortunately this change has caused issues for VS Code in Workbench for GCW image since it operates behind two reverse proxies.

Blocked request error:

2023-06-23T19:36:11.670Z] debug host "localhost:80" does not match origin "80-workstation-lj8ypj22.cluster-qq2nsyizsnhkksfpere36vh65o.cloudworkstations.dev"; blocking request to /stable-704ed70d4fd1c6bd6342c436f1ede30d1cff4710?reconnectionToken=07478307-3250-49d0-a766-efbdaff25795&reconnection=false&skipWebSocketFrames=false

code-server bug related to our issue: https://github.com/coder/code-server/issues/6064

We should be able to resolve the issue by setting the X-Forwarded-Host header to the URL of the workstation for requests to code-server.

EDIT: As of 11/21/23 this issue has been fixed by Google in Workbench for GCW. The issue still appears to be present in 2023.09 Workbench for Azure ML. The above error snippet should still correlate to the errors appearing in Azure ML.

[ianpittwood]

The same bug is present in Workbench for Azure ML after upgrading.

[kfeinauer]

@blairj09 Is going to speak with the Workbench team to see if they can look into what headers need to be propagated to get this working.

[kfeinauer]

@blairj09 Any updates on this?

[kfeinauer]

@MariaSemple Do we know what the state of Workbench + CORS/VSCode is for reverse-proxy environments like GCW and Azure ML? @bschwedler raised this with the Workbench team a few days ago, so want to make sure we follow up.

[jlang-311]

@zachhannum @MariaSemple our automated tests would be sufficient but worth stating we don't have a set up as far as we know with reverse proxies. If we wanted to move forward now then we would have to see if we needed that set up and what it would take to get it.

[MariaSemple]

Could we add the header and just validate that our tests as-is pass, and then ask the platform team if it solves the GCW issue?

[jlang-311]

@MariaSemple that makes sense!

[ianpittwood]

Good news, we can no longer replicate this issue in 2023.09.1! It's unclear whether this was a fix we made in Workbench, a change to code-server, or some sort of change to the GCW system. I would lean the latter since versions previously noted as not working (2023.06.0) are now working. Regardless, I'm closing this issue as fixed and plan to upgrade the Workbench for GCW image to the latest Workbench release.

[shalutiwari]

@MariaSemple @jlang-311 Can someone please verify that workbench tests are passing for this change?

[ianpittwood]

@shalutiwari I don't believe any actual code changes were required. I will try to get with @zachhannum today to check this problem's presence in Azure ML. I believe this bug only happens in very specific situations where we're negotiating through more than one reverse proxy, such is the case in Azure ML and GCP.

[zachhannum]

Ian is correct, there have been no code changes introduced to Workbench at this point to address this. Ian and I worked to try reproducing this in GCW yesterday and the issue did not manifest as it did before.

Based on the code-server issue linked, the requirement is that X-Forwarded-Host needs to be set in order for code-server to accept these requests. In my understanding, X-Forwarded-Host should be set by the proxy itself, so any code change we would have applied in Workbench would be a bandaid fix.

My guess is that something was changed or updated in GCW to resolve the issue with the proxies themselves.

[ianpittwood]

Reopening this issue. It only affects Azure ML now, but it's still something we should fix for the customers utilizing the Workbench for Azure ML image.

[kfeinauer]

This is blocked as it needs more investigation with the Workbench team, and they are busy in preparation for their Dec release.

[costrouc]

If VSCode opened and worked then we think this issue is now resolved. Worth testing again

[costrouc]

Closing since this is resolved. cc @blairj09