`rsconnect` auth does not work on Colorado

[akgold]

I can publish to Colorado RSC with accounts registered to rsconnect, but it does not work to auth in pins. I can pin using API Key auth.

The error is thrown in pins:::rsc_GET. I'm not sure what other diagnostic information to supply, but happy to do so...

> pins::board_rsconnect()
Error: Found multiple matching RStudio Connect servers
ℹ Please disambiguate with `server` and/or `account`
Run `rlang::last_error()` to see where the error occurred.

> rsconnect::accounts()
       name                         server
1 alex.gold           colorado.rstudio.com
2 alex.gold rsc.soleng.rstudioservices.com

> board <- pins::board_rsconnect(server = "colorado.rstudio.com")
Connecting to RSC ??? at <https://colorado.rstudio.com/rsc>
> board
Pin board <pins_board_rsconnect>
Cache size: 4.49K
Error in rsc_check_status(req) : Internal Server Error (HTTP 500).
> str(board)
List of 10
 $ board               : chr "pins_board_rsconnect"
 $ api                 : num [1:2] 0 1
 $ cache               : 'fs_path' chr "~/.cache/pins/rsc-e62371cfd77db754024f9c5ed3556a73"
 $ name                : chr "rsconnect"
 $ url                 : chr "https://colorado.rstudio.com/rsc"
 $ account             : chr "alex.gold"
 $ server_name         : chr "colorado.rstudio.com"
 $ auth                :List of 5
  ..$ username   : chr "alex.gold"
  ..$ accountId  : chr "58"
  ..$ token      : chr "T000e187685ad3cb8e1fe596a7d5b559c"
  ..$ server     : chr "colorado.rstudio.com"
  ..$ private_key: <hidden>
 $ versioned           : logi TRUE
 $ use_cache_on_failure: logi TRUE
 - attr(*, "class")= chr [1:2] "pins_board_rsconnect" "pins_board"

# Add CONNECT_SERVER and CONNECT_API_KEY envvars
> source("~/.Rprofile")
> pins::board_rsconnect()
Connecting to RSC 2021.09.0 at <https://colorado.rstudio.com/rsc>
Pin board <pins_board_rsconnect>
Cache size: 4.49K

[hadley]

Given that it's a 500 error, it might be useful to get the logs off the server.

[akgold]

@colearendt -- where would these logs be? Access log?

[colearendt]

A couple of places - not in DataDog yet, so I poked around on the server:

# from connect3
127.0.0.1 - - [16/Nov/2021:11:10:48 +0000] "GET /bike_model_performance_app/ HTTP/1.1" 500 744

Do you know about what time it was that you got this 500?

[sellorm]

I got the same thing on the same server at roughly 2021-11-18 10:27:55 UTC

There might be one or two more in the prior 10-15 minutes as I was testing.

[colearendt]

I have unfortunately reached a bit of a dead-end here without a better way to get debug logging / information about what is being used to generate the signature. RSC is rejecting the signature:

2021/11/18 10:27:40 Warning: authentication failed for token Tb84764a081b2d99b6f1dce158bbd4f63: invalid signature for request '"GET\n/rsc/__api__/applications/\nThu, 18 Nov 2021 10:27:39 GMT\n1B2M2Y8AsgTpgAmY7PhCfg=="': crypto/rsa: verification error

✗ ssh soleng-connect3 -- sudo cat /var/log/rstudio-connect.access.log | grep ' 500'
127.0.0.1 - - [18/Nov/2021:10:15:44 +0000] "GET /__api__/server_settings HTTP/1.1" 500 824
127.0.0.1 - - [18/Nov/2021:10:16:14 +0000] "GET /__api__/applications/?filter=content_type%3Apin&count=1000 HTTP/1.1" 500 822
127.0.0.1 - - [18/Nov/2021:10:27:40 +0000] "GET /__api__/applications/?filter=content_type%3Apin&count=1000 HTTP/1.1" 500 822

I have a packet capture and enough to test / see what signature is/should have been generated (i.e. the request, the Date header, etc.), but our tooling is very tricky to do that, and we were not able to use the rsconnect.* options to generate more verbose logs. Happy to provide some of this information or help test more if there is an easy way to either generate signatures from given inputs or increase logging verbosity.

From the error message and past experience, my only guesses are:

• the :443 port is getting injected into the signed url, which then fails on the server validation because :443 has been stripped (we saw this before)
• the /rsc/ part of the path is getting added to the "file" that we md5, when it should not be? This works in rsconnect though, so not sure what is going on with pins
• Vice versa, the /rsc/ part of the path is not getting added to the "file" that we md5 when it should be. (This may be the most likely scenario... I just tested a shell script that has this behavior and it reports the same error. Fixing the "file" by adding "/rsc" (the full URL path) resolves)

Basically, token auth is not working on RSC at present. Either because of https or because of the subpath that we host Connect at.

[hadley]

@colearendt can you tell me more about the 443 port? I don't see that port used anywhere in pins.

And are you saying the path passed to the signature is relative to the server root? Currently we're sending "rsc/__api__/server_settings". Are you saying that should be "__api__/server_settings"?

[hadley]

Also it works for pins::board_rsconnect(server = "connect.rstudioservices.com") so what's the difference between these servers?

[hadley]

After pairing with @jmcphers for 30 minutes and considering many exotic possible causes we discovered that it's a missing / 😬

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.