Azure tests fail locally

[hadley]

@hongooi73 can you give me some pointers as to what I need to do here?

pins:::board_azure_test(path = "", type = "file")
#> Pin board <pins_board_azure>
#> Cache size: 0
#> Error in process_storage_response(response, match.arg(http_status_handler), : Forbidden (HTTP 403). Failed to complete Storage Services operation. Message:
#> AuthenticationFailed
#> Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
#> RequestId:b303f52d-801a-0078-1bd1-ed4306000000
#> Time:2021-12-10T14:25:27.6299922Z
#> Signature fields not well formed.

pins:::board_azure_test(path = "", type = "dfs")
#> Pin board <pins_board_azure>
#> Cache size: 0
#> Error in list_adls_files(container, ...): Conflict (HTTP 409). Failed to complete Storage Services operation. Message:
#> This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
#> RequestId:a716848a-801f-000a-68d1-ed4449000000
#> Time:2021-12-10T14:25:28.0030145Z.

^{Created on 2021-12-10 by the reprex package (v2.0.1)}

[hadley]

Oh and MS365 too:

pins:::board_ms365_test_charpath(delete_by_item = TRUE)
#> Using authorization_code flow
#> Creating Microsoft Graph login for tenant 'consumers'
#> Using authorization_code flow
#> Error in initialize(...): Failed to create server

^{Created on 2021-12-10 by the reprex package (v2.0.1)}

If I run interactively, I see a bit more:

Using authorization_code flow
Creating Microsoft Graph login for tenant 'consumers'
Using authorization_code flow
createTcpServer: address already in use

[hongooi73]

For Azure, how are you getting the SAS? As noted in my comment in the PR, the SAS from the Azure portal can't be used. You can use AzureStor instead:

library(AzureStor)
endp <- storage_endpoint("https://xxxx", key="key")
sas <- get_account_sas(endp, permissions="rwcdl")

For MS365, do you have httpuv installed? I assume you're running this in RStudio?

Does this work:

AzureAuth::get_azure_token("https://graph.microsoft.com/user.read", tenant="consumers",
    app=Microsoft365R:::.microsoft365r_app_id, version=2)

[hadley]

I have a PINS_AZURE_SAS env var that I forgot to record how I generated 😬 Sounds like I should switch to using a key and then generate the SAS from that.

[hadley]

I switched board_azure_test() to this:

board_azure_test <- function(path, type = c("blob", "file", "dfs"), ...) {
  skip_if_missing_envvars("board_azure()", "PINS_AZURE_KEY")

  type <- arg_match(type)
  acct_name <- Sys.getenv("PINS_AZURE_ACCOUNT", "pins")
  acct_url <- sprintf("https://%s.%s.core.windows.net/test-data", acct_name, type)
  endp <- AzureStor::storage_endpoint(acct_url, key = Sys.getenv("PINS_AZURE_KEY"))
  sas <- AzureStor::get_account_sas(endp, permissions="rwcdl")

  container <- AzureStor::storage_container(acct_url, sas = sas)
  board_azure(container, path = path, cache = tempfile(), n_processes = 2, ...)
}

And now I get slightly different errors:

pins:::board_azure_test(path = "", type = "file")
#> Pin board <pins_board_azure>
#> Cache size: 0
#> Error in process_storage_response(response, match.arg(http_status_handler), : Not Found (HTTP 404). Failed to complete Storage Services operation. Message:
#> ShareNotFound
#> The specified share does not exist.
#> RequestId:ed125707-701a-006c-7ff6-ed0b69000000
#> Time:2021-12-10T18:47:11.6868114Z.
pins:::board_azure_test(path = "", type = "dfs")
#> Pin board <pins_board_azure>
#> Cache size: 0
#> Error in list_adls_files(container, ...): Conflict (HTTP 409). Failed to complete Storage Services operation. Message:
#> This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
#> RequestId:c4cab980-e01f-006e-26f6-edb5d1000000
#> Time:2021-12-10T18:47:12.0474873Z.

^{Created on 2021-12-10 by the reprex package (v2.0.1)}

I'm using the key from "Access keys" section of my "pins" storage account. Do they need to come from somewhere else? Or do I need to config something?

[hongooi73]

Ah, forgot to add that. You need to create the test-data file share beforehand, to match the test-data blob container. You should be able to do this in the portal.

I suggest running this once, to generate a long-term SAS. Then you can save that and use it in the tests, rather than keeping a key lying around.

endp <- AzureStor::storage_endpoint("https://pins.blob.core.windows.net", key = "key")
sas <- AzureStor::get_account_sas(endp, permissions="rwcdl", expiry=as.Date("2030-01-01"))

[hadley]

Ok, that fixes board_azure_test(path = "", type = "file"), and thanks for the hint on how to generate the SAS.

For type dfs, I still get the same error:

Error in list_adls_files(container, ...) : 
Conflict (HTTP 409). Failed to complete Storage Services operation. Message:
This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
RequestId:3e631205-001f-0059-4d01-ee677d000000
Time:2021-12-10T20:09:22.1540790Z.

[hongooi73]

I did a search and it looks like that error can occur when your storage account is of the wrong kind. In the portal, is it "general purpose v2"?

You can't change the account kind, but if this is just a testing account, you can easily delete and recreate it. You can do this in the portal, or use AzureRMR:

library(AzureStor)
library(AzureRMR)

az <- create_azure_login(tenant="yourtenant")
rg <- az$get_subscription("your-sub-id")$get_resource_group("pins_rg_name")
rg$delete_storage_account("pins")

# wait a few minutes

rg$create_storage_account("pins", hierarchical_namespace_enabled=TRUE)

[hadley]

Hmmm, that looks ok:

[hadley]

Maybe I need to do this?

[hongooi73]

Yeah, probably need to do that. I don't think there's any cost involved to upgrade.

Did you manage to sort out the auth issue with MS365?

[hadley]

@hongooi73 oh yeah, that was solved by a reboot 😬 I guess I must've had a rogue process binding to the port needed for OAuth.

[hadley]

Conversion to data lake gen 2 isn't going well — it keeps freezing on the validation step, which seems weird given how small my containers are.

[hadley]

But deleting and re-creating the container seemed to do the trick. Thanks for all the help!

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.