search

rstudio / rstudio-docker-products

Docker images for RStudio Professional Products
https://hub.docker.com/u/rstudio
MIT License
66 stars 55 forks source link

Workbench for GCW session calls intermittently return `405 Method Not Found` errors #583

Closed ianpittwood closed 1 year ago

ianpittwood commented 1 year ago

Recently @jthomasmock noticed that calls to retrieve, create, or modify sessions in RStudio on Workbench for GCW were returning 405 Method Not Found errors. After further investigation, we found multiple calls similar to the following in rserver.log:

2023-07-07T15:36:00.089430Z [rserver] ERROR Invalid method OST requested for uri: /events/get_events; LOGGED FROM: bool rstudio::core::http::AsyncServerImpl<ProtocolType>::onHeadersParsed(rstudio_boost::shared_ptr<rstudio::core::http::AsyncConnectionImpl<typename ProtocolType::socket> >, rstudio::core::http::Request*) [with ProtocolType = rstudio_boost::asio::local::stream_protocol  typename ProtocolType::socket = rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol>] src/cpp/server/ServerInitLocalStream.cpp:466

For some reason, requests reaching Workbench are partially cut off, yielding calls to methods such as OST, ET, or ST. Since these are not valid methods, the server does as expected and returns 405 Method Not Found errors.

We are unsure why client requests are being cut off as they are sent correctly from the client, but sometimes reach the Workbench server as fragments. This issue has been escalated to Google to further investigate the Workstation intermediary networking and servers for a cause.

kfeinauer commented 1 year ago

There is more troubleshooting we can do on our side to ensure that the issue is with Google's upstream proxy, but their development team said they were looking into this.

If they don't get back with us soon and do not respond, we should look into running tshark within the development container to ensure that the incoming traffic is truncated - if it isn't, that would imply a Workbench issue.

However, we have seen this in the access logs on nginx that this is already truncated, so this is very likely an upstream issue.

blairj09 commented 1 year ago

This was identified as a bug on the Google side and they report to have rolled out a fix last week. We need to verify the fix and then we can close the issue.

ianpittwood commented 1 year ago

Google appears to have resolved this issue. We'll close it and reopen if the error appears in the future.