Azure Storage Account connections via proxy fail in an Azure environment.

yamashin55 commented 6 months ago

Do you already have an issue opened with F5 support?

No.

Description

Storage Account connections via proxy fail in an Azure environment.

Environment information

Cloud Failover Extension Version: 2.1.0
BIG-IP version: 17.1.1.3
Cloud provider: Azure

Severity Level

Severity: <3>

Log Detailed

It has already been confirmed that connections without proxy settings (direct) work correctly.

Failure logs when using proxies. /var/log/restnoded/restnoded.log

Thu, 02 May 2024 20:29:50 GMT - finest: socket 425 opened
Thu, 02 May 2024 20:29:50 GMT - fine: [f5-cloud-failover] HTTP Request - POST /declare
Thu, 02 May 2024 20:29:50 GMT - fine: [f5-cloud-failover] Successfully validated declaration
Thu, 02 May 2024 20:29:50 GMT - info: [f5-cloud-failover] Global logLevel set to 'silly'
Thu, 02 May 2024 20:29:50 GMT - finest: [f5-cloud-failover] Modifying existing data group f5-cloud-failover-state with body {"name":"f5-cloud-failover-state","type":"string","records":[{"name":"state","data":"eyJjb25maWciOnsiY2xhc3MiOiJDbG91ZF9GYWlsb3ZlciIsImVudmlyb25tZW50IjoiYXp1cmUiLCJjb250cm9scyI6eyJjbGFzcyI6IkNvbnRyb2xzIiwibG9nTGV2ZWwiOiJzaWxseSJ9LCJyZXRyeUZhaWxvdmVyIjp7ImVuYWJsZWQiOnRydWUsImludGVydmFsIjoyfSwiZXh0ZXJuYWxTdG9yYWdlIjp7InNjb3BpbmdUYWdzIjp7ImY1X2Nsb3VkX2ZhaWxvdmVyX2xhYmVsIjoiQklHSVBfRkFJTE9WRVJfU1RPUkFHRSJ9fSwiZmFpbG92ZXJBZGRyZXNzZXMiOnsic2NvcGluZ1RhZ3MiOnsiZjVfY2xvdWRfZmFpbG92ZXJfbGFiZWwiOiJCSUdJUF9GQUlMT1ZFUl9BRERSRVNTMiJ9LCJyZXF1aXJlU2NvcGluZ1RhZ3MiOmZhbHNlfSwic2NoZW1hVmVyc2lvbiI6IjIuMS4wIn19"}]}
Thu, 02 May 2024 20:29:51 GMT - finest: socket 426 opened
Thu, 02 May 2024 20:29:54 GMT - info: [f5-cloud-failover] Successfully wrote Failover trigger scripts to filesystem
Thu, 02 May 2024 20:29:54 GMT - finest: [f5-cloud-failover] Device initialization complete
Thu, 02 May 2024 20:29:55 GMT - finest: [f5-cloud-failover] Fetched proxy settings: 
Thu, 02 May 2024 20:29:55 GMT - finest: [f5-cloud-failover] {"protocol":"http","host":"x.x.x.x","port":"3128","username":"","password":"********"}
Thu, 02 May 2024 20:29:55 GMT - finest: [f5-cloud-failover] Subscriptions:  {"0":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
Thu, 02 May 2024 20:29:55 GMT - finest: [f5-cloud-failover] Listing Storage Accounts
Thu, 02 May 2024 20:29:56 GMT - finest: [f5-cloud-failover] Storage Account Information:  f5cfe12345
Thu, 02 May 2024 20:29:56 GMT - finest: socket 426 closed
Thu, 02 May 2024 20:29:57 GMT - finest: [f5-cloud-failover] Status: "HTTP request failed: 502 \"<!DOCTYPE html PUBLIC \\\"-//W3C//DTD HTML 4.01//EN\\\" \\\"http://www.w3.org/TR/html4/strict.dtd\\\">\\n<html><head>\\n<meta type=\\\"copyright\\\" content=\\\"Copyright (C) 1996-2020 The Squid Software Foundation and contributors\\\">\\n<meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=utf-8\\\">\\n<title>ERROR: The requested URL could not be retrieved</title>\\n<style type=\\\"text/css\\\"><!-- \\n /*\\n * Copyright (C) 1996-2022 The Squid Software Foundation and contributors\\n *\\n * Squid software is distributed under GPLv2+ license and includes\\n * contributions from numerous individuals and organizations.\\n * Please see the COPYING and CONTRIBUTORS files for details.\\n */\\n\\n/*\\n Stylesheet for Squid Error pages\\n Adapted from design by Free CSS Templates\\n http://www.freecsstemplates.org\\n Released for free under a Creative Commons Attribution 2.5 License\\n*/\\n\\n/* Page basics */\\n* {\\n\\tfont-family: verdana, sans-serif;\\n}\\n\\nhtml body {\\n\\tmargin: 0;\\n\\tpadding: 0;\\n\\tbackground: #efefef;\\n\\tfont-size: 12px;\\n\\tcolor: #1e1e1e;\\n}\\n\\n/* Page displayed title area */\\n#titles {\\n\\tmargin-left: 15px;\\n\\tpadding: 10px;\\n\\tpadding-left: 100px;\\n\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\n}\\n\\n/* initial title */\\n#titles h1 {\\n\\tcolor: #000000;\\n}\\n#titles h2 {\\n\\tcolor: #000000;\\n}\\n\\n/* special event: FTP success page titles */\\n#titles ftpsuccess {\\n\\tbackground-color:#00ff00;\\n\\twidth:100%;\\n}\\n\\n/* Page displayed body content area */\\n#content {\\n\\tpadding: 10px;\\n\\tbackground: #ffffff;\\n}\\n\\n/* General text */\\np {\\n}\\n\\n/* error brief description */\\n#error p {\\n}\\n\\n/* some data which may have caused the problem */\\n#data {\\n}\\n\\n/* the error message received from the system or other software */\\n#sysmsg {\\n}\\n\\npre {\\n}\\n\\n/* special event: FTP / Gopher directory listing */\\n#dirmsg {\\n    font-family: courier, monospace;\\n    color: black;\\n    font-size: 10pt;\\n}\\n#dirlisting {\\n    margin-left: 2%;\\n    margin-right: 2%;\\n}\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\n    border-bottom: groove;\\n}\\n#dirlisting td.size {\\n    width: 50px;\\n    text-align: right;\\n    padding-right: 5px;\\n}\\n\\n/* horizontal lines */\\nhr {\\n\\tmargin: 0;\\n}\\n\\n/* page displayed footer area */\\n#footer {\\n\\tfont-size: 9px;\\n\\tpadding-left: 10px;\\n}\\n\\n\\nbody\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\n:lang(he) { direction: rtl; }\\n --></style>\\n</head><body id=ERR_READ_ERROR>\\n<div id=\\\"titles\\\">\\n<h1>ERROR</h1>\\n<h2>The requested URL could not be retrieved</h2>\\n</div>\\n<hr>\\n\\n<div id=\\\"content\\\">\\n<p>The following error was encountered while trying to retrieve the URL: <a href=\\\"https://f5cfe12345.blob.core.windows.net/?\\\">https://f5cfe12345.blob.core.windows.net/?</a></p>\\n\\n<blockquote id=\\\"error\\\">\\n<p><b>Read Error</b></p>\\n</blockquote>\\n\\n<p id=\\\"sysmsg\\\">The system returned: <i>[No Error]</i></p>\\n\\n<p>An error condition occurred while reading data from the network. Please retry your request.</p>\\n\\n<p>Your cache administrator is <a href=\\\"mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_READ_ERROR&amp;body=CacheHost%3A%20ip-10-0-1-12%0D%0AErrPage%3A%20ERR_READ_ERROR%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Thu,%2002%20May%202024%2020%3A29%3A57%20GMT%0D%0A%0D%0AClientIP%3A%2020.243.120.254%0D%0AServerIP%3A%20f5cfe12345.blob.core.windows.net%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2F%3Fcomp%3Dlist%20HTTP%2F1.1%0AAccept%3A%20application%2Fjson,%20text%2Fplain,%20*%2F*%0D%0AAuthorization%3A%20Bearer%20XXXXXXXXXXXXXXXXXXXXXXXXXXXX%0D%0Ax-ms-version%3A%202017-11-09%0D%0AUser-Agent%3A%20axios%2F0.21.4%0D%0AConnection%3A%20keep-alive%0D%0AHost%3A%20f5cfe12345.blob.core.windows.net%0D%0A%0D%0A%0D%0A\\\">webmaster</a>.</p>\\n<br>\\n</div>\\n\\n<hr>\\n<div id=\\\"footer\\\">\\n<p>Generated Thu, 02 May 2024 20:29:57 GMT by ip-10-0-1-12 (squid/5.7)</p>\\n<!-- ERR_READ_ERROR -->\\n</div>\\n</body></html>\\n\"" Retries left: 60

Access Loged at Proxy Server(Squid).


"2024/05/02 20:01:50.724" 20.243.120.254 1825 4.150.240.10 2474 443 application/json "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Storage/storageAccounts?api-version=2023-01-01 HTTP/1.1" 200 "-" "axios/0.21.4" "TCP_REFRESH_MODIFIED:HIER_DIRECT"
"2024/05/02 20:01:51.537" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"
"2024/05/02 20:01:53.345" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"
"2024/05/02 20:01:55.148" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"
"2024/05/02 20:01:56.950" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"
"2024/05/02 20:01:58.756" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"

mikeshimkus commented 6 months ago

Hi @yamashin55, can you provide the following:

Your squid.conf
RBAC roles assigned to identity used by the BIG-IP instances

It looks like you have a password set in the proxy db variables, but no username. Is that correct or is it redacted?

Also, how did you deploy CFE? Did you use the ARM templates? Thanks

yamashin55 commented 6 months ago

@mikeshimkus, Thank you for replay.

Below is /etc/squid/squid.conf

ubuntu@ip-10-0-1-12:~$ grep -v '^\s*#' /etc/squid/squid.conf |grep -v '^\s*$'
acl localnet src 0.0.0.1-0.255.255.255  # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8             # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10          # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16         # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12          # RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16         # RFC 1918 local private network (LAN)
acl localnet src fc00::/7               # RFC 4193 local private network range
acl localnet src fe80::/10              # RFC 4291 link-local (directly plugged) machines
acl allallow src 0.0.0.0/0
acl CONNECT method CONNECT
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
include /etc/squid/conf.d/*.conf
http_access allow localhost
http_access allow allallow
http_access deny all
http_port 3128
logformat mycombined   "%{%Y/%m/%d %H:%M:%S}tl.%03tu" %>a %>st %<a %<st %>rP %mt "%rm %>ru HTTP/%rv" %>Hs "%{Referer}>h" "%{User-Agent}>h" "%Ss:%Sh"
access_log daemon:/var/log/squid/access.log mycombined
coredump_dir /var/spool/squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern \/(Packages|Sources)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims
refresh_pattern \/Release(|\.gpg)$ 0 0% 0 refresh-ims
refresh_pattern \/InRelease$ 0 0% 0 refresh-ims
refresh_pattern \/(Translation-.*)(|\.bz2|\.gz|\.xz)$ 0 0% 0 refresh-ims
refresh_pattern .               0       20%     4320
ubuntu@ip-10-0-1-12:~$

Below is RBAC roles

{
"id": "/subscriptions/XXXXXXXXXX/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXXXXX",
"properties": {
    "roleName": "F5-BIGIP-FailOver-Role",
    "description": "F5 Networks BIG-IP",
    "assignableScopes": [
        "/subscriptions/XXXXXXXXXX/resourceGroups/f5-rsg-vm",
        "/subscriptions/XXXXXXXXXX/resourceGroups/f5-rsg-vnet"
    ],
    "permissions": [
        {
            "actions": [
                "Microsoft.Network/*/join/action",
                "Microsoft.Network/networkInterfaces/write",
                "Microsoft.Network/publicIPAddresses/write",
                "Microsoft.Network/routeTables/*/read",
                "Microsoft.Network/routeTables/*/write",
                "Microsoft.Storage/storageAccounts/read",
                "Microsoft.Storage/storageAccounts/blobServices/containers/read",
                "Microsoft.Storage/storageAccounts/blobServices/containers/write",
                "Microsoft.Authorization/*/read",
                "Microsoft.Compute/locations/*/read",
                "Microsoft.Compute/virtualMachines/*/read",
                "Microsoft.Compute/virtualMachineScaleSets/*/read",
                "Microsoft.Compute/virtualMachineScaleSets/networkInterfaces/read",
                "Microsoft.Network/networkInterfaces/read",
                "Microsoft.Network/publicIPAddresses/read",
                "Microsoft.Resources/subscriptions/resourceGroups/read",
                "Microsoft.Compute/virtualMachines/extensions/*"
            ],
            "notActions": [],
            "dataActions": [
                "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
                "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write"
            ],
            "notDataActions": []
        }
    ]
}
}

It looks like you have a password set in the proxy db variables, but no username. Is that correct or is it redacted?

This proxy server has no authentication. I have not set a proxy ID and password on my BIG-IP. I have not been changed from the default.

I used fllowing command: tmsh modify sys db proxy.host value 54.85.XX.XX tmsh modify sys db proxy.port value 3128 tmsh save sys config

Also, how did you deploy CFE? Did you use the ARM templates?

No. I didn't use ARM templates. I deployed the BIG-IP manually. CFE Configuration is :

{
    "class": "Cloud_Failover",
    "environment": "azure",
    "controls": {
      "class": "Controls",
      "logLevel": "silly"
    },
    "retryFailover": {
        "enabled": true,
        "interval": 2
     },
    "externalStorage":{
        "scopingTags": {
            "f5_cloud_failover_label": "BIGIP_FAILOVER_ADDRESS2"
        }
    },
    "failoverAddresses": {
        "scopingTags": {
            "f5_cloud_failover_label": "BIGIP_FAILOVER_STORAGE"
        }
    }
}

*Below are the logs when no proxy is used. (Direct access is successful.)

Thu, 02 May 2024 20:21:18 GMT - fine: [f5-cloud-failover] HTTP Request - POST /declare
Thu, 02 May 2024 20:21:18 GMT - fine: [f5-cloud-failover] Successfully validated declaration
Thu, 02 May 2024 20:21:18 GMT - info: [f5-cloud-failover] Global logLevel set to 'silly'
Thu, 02 May 2024 20:21:18 GMT - finest: [f5-cloud-failover] Creating new data group f5-cloud-failover-state with body {"name":"f5-cloud-failover-state","type":"string","records":[{"name":"state","data":"eyJjb25maWciOnsiY2xhc3MiOiJDbG91ZF9GYWlsb3ZlciIsImVudmlyb25tZW50IjoiYXp1cmUiLCJjb250cm9scyI6eyJjbGFzcyI6IkNvbnRyb2xzIiwibG9nTGV2ZWwiOiJzaWxseSJ9LCJyZXRyeUZhaWxvdmVyIjp7ImVuYWJsZWQiOnRydWUsImludGVydmFsIjoyfSwiZXh0ZXJuYWxTdG9yYWdlIjp7InNjb3BpbmdUYWdzIjp7ImY1X2Nsb3VkX2ZhaWxvdmVyX2xhYmVsIjoiQklHSVBfRkFJTE9WRVJfU1RPUkFHRSJ9fSwiZmFpbG92ZXJBZGRyZXNzZXMiOnsic2NvcGluZ1RhZ3MiOnsiZjVfY2xvdWRfZmFpbG92ZXJfbGFiZWwiOiJCSUdJUF9GQUlMT1ZFUl9BRERSRVNTMiJ9LCJyZXF1aXJlU2NvcGluZ1RhZ3MiOmZhbHNlfSwic2NoZW1hVmVyc2lvbiI6IjIuMS4wIn19"}]}
Thu, 02 May 2024 20:21:21 GMT - info: [f5-cloud-failover] Successfully wrote Failover trigger scripts to filesystem
Thu, 02 May 2024 20:21:21 GMT - finest: [f5-cloud-failover] Device initialization complete
Thu, 02 May 2024 20:21:21 GMT - finest: [f5-cloud-failover] Fetched proxy settings: 
Thu, 02 May 2024 20:21:21 GMT - finest: [f5-cloud-failover] {"protocol":"http","host":"","port":"8080","username":"","password":"********"}
Thu, 02 May 2024 20:21:21 GMT - finest: [f5-cloud-failover] Subscriptions:  {"0":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
Thu, 02 May 2024 20:21:21 GMT - finest: [f5-cloud-failover] Listing Storage Accounts
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Storage Account Information:  f5cfe12345
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Container {"Name":"f5cloudfailover"} was found...
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Container f5cloudfailover already exists, continuing...
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Cloud Provider initialization complete
Thu, 02 May 2024 20:21:22 GMT - fine: [f5-cloud-failover] Performing failover - initialization
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Device initialization complete
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Fetched proxy settings: 
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] {"protocol":"http","host":"","port":"8080","username":"","password":"********"}
Thu, 02 May 2024 20:21:22 GMT - fine: [f5-cloud-failover] config: 
Thu, 02 May 2024 20:21:22 GMT - fine: [f5-cloud-failover] {"class":"Cloud_Failover","environment":"azure","controls":{"class":"Controls","logLevel":"silly"},"retryFailover":{"enabled":true,"interval":2},"externalStorage":{"scopingTags":{"f5_cloud_failover_label":"BIGIP_FAILOVER_STORAGE"}},"failoverAddresses":{"scopingTags":{"f5_cloud_failover_label":"BIGIP_FAILOVER_ADDRESS2"},"requireScopingTags":false},"schemaVersion":"2.1.0"}
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] proxySettings: 
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] {"protocol":"http","host":"","port":"8080","username":"","password":"********"}
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Telemetry submitted successfully
Thu, 02 May 2024 20:21:22 GMT - finest: [f5-cloud-failover] Telemetry payload: {"customerId":"d3563489-ec7f-4117-830b-b5ed2ef02816","failover":{"event":false,"success":true},"product":{"version":"2.1.0","locale":"en-US","installDate":"2024-05-02T20:21:22.065Z","installationId":"","environment":"azure","region":"japaneast"},"featureFlags":{"ipFailover":false,"routeFailover":false},"operation":{"clientRequestId":"e7855baa-7140-41e1-ab6f-f4903a70cf43","userAgent":"f5-cloud-failover/2.1.0","result":"SUCCESS","resultSummary":"Configuration Successful"}}
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Device initialization complete
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Fetched proxy settings: 
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] {"protocol":"http","host":"","port":"8080","username":"","password":"********"}
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Subscriptions:  {"0":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Listing Storage Accounts
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Storage Account Information:  f5cfe12345
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Container {"Name":"f5cloudfailover"} was found...
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Container f5cloudfailover already exists, continuing...
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Cloud Provider initialization complete
Thu, 02 May 2024 20:21:23 GMT - finest: [f5-cloud-failover] Failover initialization complete

yamashin55 commented 6 months ago

I thought that CONNECT Method is normally used for HTTPS communication in case of Explicit Proxy.

CONNECT abc.com

Use proxy Web access from BigiP.

[admin@10-0-1-15:Standby:In Sync] ~ # curl --proxy http://54.85.112.57:3128 https://httpbin.org/ip
{
"origin": "54.85.112.57"
}
[admin@10-0-1-15:Standby:In Sync] ~ #

Squid Access Logs

"2024/05/03 01:34:20.325" 20.78.32.69 987 3.211.223.136 5838 443 - "CONNECT httpbin.org:443 HTTP/1.1" 200 "-" "curl/7.47.1" "TCP_TUNNEL:HIER_DIRECT"

However, when accessing from CFE, it appears that the GET method is being used. GET https://management.azure.com/... GET https://f5cfe12345.blob.core.windows.net/?comp....

"2024/05/02 20:01:50.724" 20.243.120.254 1825 4.150.240.10 2474 443 application/json "GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Storage/storageAccounts?api-version=2023-01-01 HTTP/1.1" 200 "-" "axios/0.21.4" "TCP_REFRESH_MODIFIED:HIER_DIRECT"
"2024/05/02 20:01:51.537" 20.243.120.254 1687 20.150.85.196 5535 443 text/html "GET https://f5cfe12345.blob.core.windows.net/?comp=list HTTP/1.1" 502 "-" "axios/0.21.4" "TCP_MISS:HIER_DIRECT"

However, the access log of Azure Resource Manager(management.azure.com) shows a normal 200 response. The Storage Account (f5cfe12345.blob.core.windows.net) access log is a 502 response.

mikeshimkus commented 6 months ago

@yamashin55 I created internal issue EC-510 for this.

Can you also share the JSON config of the storage account (click on the account overview in the Azure portal and then the JSON View link). Do you have any ACLs or NSG rules applied to the storage account that would deny traffic from the proxy IP/vNET but allow it from the BIG-IP instance IPs?

Regarding the method, the curl command is using the proxy db settings directly, while CFE passes them to the Axios client as proxy options. Regardless of that difference, this was successfully tested with identical db var and squid configuration, so I suspect something blocking access from the proxy server specifically.

yamashin55 commented 6 months ago

@mikeshimkus,
Thank you for comment about the difference of method.

JSON config of the storage account.

{
"sku": {
    "name": "Standard_LRS",
    "tier": "Standard"
},
"kind": "StorageV2",
"id": "/subscriptions/d3563489-ec7f-4117-830b-xxxxxxxxxxxx/resourceGroups/f5-rsg-vm/providers/Microsoft.Storage/storageAccounts/f5cfe12345",
"name": "f5cfe12345",
"type": "Microsoft.Storage/storageAccounts",
"location": "japaneast",
"tags": {
    "f5_cloud_failover_label": "BIGIP_FAILOVER_STORAGE"
},
"properties": {
    "dnsEndpointType": "Standard",
    "defaultToOAuthAuthentication": false,
    "publicNetworkAccess": "Enabled",
    "keyCreationTime": {
        "key1": "2024-05-02T18:56:52.9341396Z",
        "key2": "2024-05-02T18:56:52.9341396Z"
    },
    "allowCrossTenantReplication": false,
    "privateEndpointConnections": [],
    "minimumTlsVersion": "TLS1_2",
    "allowBlobPublicAccess": false,
    "allowSharedKeyAccess": true,
    "networkAcls": {
        "resourceAccessRules": [],
        "bypass": "AzureServices",
        "virtualNetworkRules": [
            {
                "id": "/subscriptions/d3563489-ec7f-4117-830b-xxxxxxxxxxxx/resourceGroups/f5-rsg-vnet/providers/Microsoft.Network/virtualNetworks/10.0.0.0_16/subnets/10.0.1.0_24",
                "action": "Allow",
                "state": "Succeeded"
            }
        ],
        "ipRules": [
            {
                "value": "54.85.112.57",
                "action": "Allow"
            }
        ],
        "defaultAction": "Deny"
    },
    "supportsHttpsTrafficOnly": true,
    "encryption": {
        "requireInfrastructureEncryption": false,
        "services": {
            "file": {
                "keyType": "Account",
                "enabled": true,
                "lastEnabledTime": "2024-05-02T18:56:52.9497731Z"
            },
            "blob": {
                "keyType": "Account",
                "enabled": true,
                "lastEnabledTime": "2024-05-02T18:56:52.9497731Z"
            }
        },
        "keySource": "Microsoft.Storage"
    },
    "accessTier": "Hot",
    "provisioningState": "Succeeded",
    "creationTime": "2024-05-02T18:56:52.8560186Z",
    "primaryEndpoints": {
        "dfs": "https://f5cfe12345.dfs.core.windows.net/",
        "web": "https://f5cfe12345.z11.web.core.windows.net/",
        "blob": "https://f5cfe12345.blob.core.windows.net/",
        "queue": "https://f5cfe12345.queue.core.windows.net/",
        "table": "https://f5cfe12345.table.core.windows.net/",
        "file": "https://f5cfe12345.file.core.windows.net/"
    },
    "primaryLocation": "japaneast",
    "statusOfPrimary": "available"
}
}

Do you have any ACLs or NSG rules applied to the storage account that would deny traffic from the proxy IP/vNET but allow it from the BIG-IP instance IPs?

I changed the "Public network access" setting to "Enabled from selected virtual networks and IP addresses".

Azure portal -> StorageAccout -> Networking -> Firewalls and virtual networks, for Selected networks.

And at Firewall rules section, I added the global address of squid proxy server. Other than that, no specific ACLs have been changed.

mikeshimkus commented 6 months ago

Thanks. It might be helpful to configure the storage account like the ARM templates do just for testing, for example: https://github.com/F5Networks/f5-azure-arm-templates-v2/blob/9efd07d357ef01e35e5db0a95a7ac6debca15a57/examples/modules/bigip-standalone/bigip.json#L424

I have placed this issue in the queue and will update here with the outcome.

yamashin55 commented 6 months ago

I tested using StorageAccount of ARM standalone BigIP Template with the same result. This 502 error symptom did not improve.

I found a similar problem was raised with axios. issues-5078 I have no programming knowledge and don't know...

mikeshimkus commented 5 months ago

@yamashin55 We just released https://github.com/F5Networks/f5-cloud-failover-extension/releases/tag/v2.1.1 with a fix for this issue.

yamashin55 commented 5 months ago

@mikeshimkus It's working fine with the new version(v2.1.1)! Thank you for your help! I couldn't find the rpm package file, so I replaced the following two files with v2.1.1 and checked the reproducibility in my lab env. It works fine.

src/nodejs/providers/azure/cloud.js src/nodejs/util.js

Could you upload the rpm file when you have time. I think I have solved the problem.

F5Networks / f5-cloud-failover-extension