search

shibayan / appservice-acmebot

Automated ACME SSL/TLS certificates issuer for Azure App Service (Web Apps / Functions / Containers)
Apache License 2.0
392 stars 60 forks source link

Azure Functions' ZipFS fails to deploy appservice-acmebot due to HTTP 401 #458

Closed daiplusplus closed 4 months ago

daiplusplus commented 4 months ago

Describe the bug

I've been running appservice-acmebot without any problems for the past 3 years, but it stopped renewing certs about a month ago; I didn't notice until recently when I had to manually renew a bunch of certs.

I'm not aware of any changes made to my Azure Subscription or the Function Apps that run appservice-acmebot.

All of the following errors/messages happen when trying to access my already-deployed appservice-acmebot.

This is what appears in C:\home\LogFiles\eventlog.xml:

These events appear repeatedly, in the same order, over-and-over:

    <Event>
        <System>
            <Provider Name="ZipFS"/>
            <EventID>0</EventID>
            <Level>1</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:27Z"/>
            <EventRecordID>81540906</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Failed to download zip. See the text file located in the wwwroot folder for more details. Error: 0x80070003</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="ZipFS"/>
            <EventID>0</EventID>
            <Level>1</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:28Z"/>
            <EventRecordID>81540953</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Failure zip was generated to \\?\d:\local\SitePackages\81495046.zip</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="ZipFS"/>
            <EventID>0</EventID>
            <Level>1</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:28Z"/>
            <EventRecordID>81541015</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Failed to copy zip from remote source.</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="ZipFS"/>
            <EventID>0</EventID>
            <Level>1</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:28Z"/>
            <EventRecordID>81541093</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Adding virtual directory from failure zip file \\?\d:\local\SitePackages\81495046.zip</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module V2"/>
            <EventID>1032</EventID>
            <Level>4</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:28Z"/>
            <EventRecordID>81541890</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Application 'C:\Program Files (x86)\SiteExtensions\Functions\3.22.0\32bit\' started successfully.</Data>
            <Data>Process Id: 5724.</Data>
            <Data>File Version: 13.1.22321.32. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 3eeb12e106b9e913c3a4dec1a7d16da7b74149eb</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module V2"/>
            <EventID>1033</EventID>
            <Level>4</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:13:30Z"/>
            <EventRecordID>81543703</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-7-176</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Application 'MACHINE/WEBROOT/APPHOST/FUNC-DVI-ACME2-LF7A' has shutdown.</Data>
            <Data>Process Id: 5724.</Data>
            <Data>File Version: 13.1.22321.32. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 3eeb12e106b9e913c3a4dec1a7d16da7b74149eb</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module V2"/>
            <EventID>1032</EventID>
            <Level>4</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:14:09Z"/>
            <EventRecordID>81887125</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-8-197</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Application 'C:\Program Files (x86)\SiteExtensions\Functions\3.22.0\32bit\' started successfully.</Data>
            <Data>Process Id: 5344.</Data>
            <Data>File Version: 13.1.22321.32. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 3eeb12e106b9e913c3a4dec1a7d16da7b74149eb</Data>
        </EventData>
    </Event>
    <Event>
        <System>
            <Provider Name="IIS AspNetCore Module V2"/>
            <EventID>1033</EventID>
            <Level>4</Level>
            <Task>0</Task>
            <Keywords>Keywords</Keywords>
            <TimeCreated SystemTime="2024-06-21T14:14:09Z"/>
            <EventRecordID>81887484</EventRecordID>
            <Channel>Application</Channel>
            <Computer>10-30-8-197</Computer>
            <Security/>
        </System>
        <EventData>
            <Data>Application 'MACHINE/WEBROOT/APPHOST/FUNC-DVI-ACME2-LF7A' has shutdown.</Data>
            <Data>Process Id: 5344.</Data>
            <Data>File Version: 13.1.22321.32. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 3eeb12e106b9e913c3a4dec1a7d16da7b74149eb</Data>
        </EventData>
    </Event>

Normally my Azure sites' EventLog.xml files have events going back at least a few months, but in this case both EventLog.xml and EventLog.prev.xml all contain events logged in the past hour.

Re: "See the text file located in the wwwroot folder for more details.", the file is named FAILED TO INITIALIZE RUN FROM PACKAGE.txt and contains this:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0   402    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 401

And a web.config containing only this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name = "Site Unavailable" stopProcessing = "true">
                    <match url = ".*" />
                    <action type = "CustomResponse" statusCode = "503" subStatusCode = "0" statusReason = "Site Unavailable" statusDescription = "Could not download zip" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Additionally, there are no new logs under C:\home\LogFiles\Application\ (such as C:\home\LogFiles\Application\Functions\Function\AddCertificate_HttpPoll) since 2022.

The "Failure zip was generated" file just contains a copy of the above 2 files.

The current appservice-acmebot package URI is https://stacmebotprod.blob.core.windows.net/appservice-acmebot/v4/latest.zip and I'm able to download that just fine from my desktop.

UPDATE: I found my current WEBSITE_RUN_FROM_PACKAGE value, and it's https://shibayan.blob.core.windows.net/azure-appservice-letsencrypt/v3/latest.zip - I get a HTTP 404 response using my desktop, but the logs show that Azure gets a 401 response.

...and I see you commented about this here. Welp.

To Reproduce

Unknown steps.

Environment (please complete the following information):

Additional context

daiplusplus commented 4 months ago

https://github.com/shibayan/keyvault-acmebot/discussions/500#discussioncomment-8957541