HttpClient / HttpWebRequest cannot download more than 2372B of file

vtelenak commented 2 weeks ago

Target name(s)

ESP32_REV0, ESP32_REV3

Firmware version

No response

Was working before? On which version?

No response

Device capabilities

System Information HAL build info: nanoCLR running @ ESP32 built with ESP-IDF e7771c7 Target: ESP32_REV0 Platform: ESP32

Firmware build Info: Date: Jun 17 2024 Type: MinSizeRel build, chip rev. >= 0, without support for PSRAM CLR Version: 1.9.1.243 Compiler: GNU ARM GCC v12.2.0

OEM Product codes (vendor, model, SKU): 0, 0, 0

Serial Numbers (module, system): 00000000000000000000000000000000 0000000000000000

Target capabilities: Has nanoBooter: NO IFU capable: NO Has proprietary bootloader: YES

AppDomains:

Assemblies: NFAppAPTest, 1.0.0.0 mscorlib, 1.15.6.0 System.IO.FileSystem, 1.1.47.0 nanoFramework.System.Collections, 1.5.31.0 System.Device.Gpio, 1.1.41.0 System.Net.Http, 1.5.138.0 nanoFramework.Logging, 1.1.100.0 nanoFramework.Runtime.Native, 1.6.12.0 System.Threading, 1.1.32.63105 nanoFramework.System.Runtime, 1.0.27.0 nanoFramework.Runtime.Events, 1.11.18.0 nanoFramework.System.Text, 1.2.54.0 System.IO.Streams, 1.1.59.0 System.Net, 1.10.79.0

Native Assemblies: mscorlib v100.5.0.19, checksum 0x445C7AF9 nanoFramework.Runtime.Native v100.0.9.0, checksum 0x109F6F22 nanoFramework.Hardware.Esp32 v100.0.10.0, checksum 0x6A20A689 nanoFramework.Hardware.Esp32.Rmt v100.0.4.0, checksum 0x608C5658 nanoFramework.Device.OneWire v100.0.4.0, checksum 0xB95C43B4 nanoFramework.Networking.Sntp v100.0.4.4, checksum 0xE2D9BDED nanoFramework.ResourceManager v100.0.0.1, checksum 0xDCD7DF4D nanoFramework.System.Collections v100.0.1.0, checksum 0x2DC2B090 nanoFramework.System.Text v100.0.0.1, checksum 0x8E6EB73D nanoFramework.System.IO.Hashing v100.0.0.1, checksum 0xEBD8ED20 nanoFramework.System.Security.Cryptography v100.0.0.2, checksum 0xF4AEFE6C nanoFramework.Runtime.Events v100.0.8.0, checksum 0x0EAB00C9 EventSink v1.0.0.0, checksum 0xF32F4C3E System.IO.FileSystem v1.1.0.0, checksum 0xCC556D24 System.Math v100.0.5.5, checksum 0x9F9E2A7E System.Net v100.2.0.1, checksum 0xD82C1452 System.Device.Adc v100.0.0.0, checksum 0xE5B80F0B System.Device.Dac v100.0.0.6, checksum 0x02B3E860 System.Device.Gpio v100.1.0.6, checksum 0x097E7BC5 System.Device.I2c v100.0.0.2, checksum 0xFA806D33 System.Device.I2c.Slave v1.0.0.0, checksum 0x4238164B System.Device.I2s v100.0.0.1, checksum 0x478490FE System.Device.Pwm v100.1.0.4, checksum 0xABF532C3 System.IO.Ports v100.1.6.1, checksum 0xB798CE30 System.Device.Spi v100.1.2.0, checksum 0x3F6E2A7E System.Runtime.Serialization v100.0.0.0, checksum 0x0A066871 System.Device.Wifi v100.0.6.4, checksum 0x00A058C6 Windows.Storage v100.0.3.0, checksum 0xF0C37E1B

++++++++++++++++++++++++++++++++ ++ Memory Map ++ ++++++++++++++++++++++++++++++++ Type Start Size ++++++++++++++++++++++++++++++++ RAM 0x3ffe46bc 0x0001b000 FLASH 0x00000000 0x00400000

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ++ Flash Sector Map ++ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Region Start Blocks Bytes/Block Usage +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 0 0x00010000 1 0x1B0000 nanoCLR 1 0x001C0000 1 0x1E0000 Deployment 2 0x003C0000 1 0x040000 Configuration

+++++++++++++++++++++++++++++++++++++++++++++++++++ ++ Storage Usage Map ++ +++++++++++++++++++++++++++++++++++++++++++++++++++ Start Size (kB) Usage +++++++++++++++++++++++++++++++++++++++++++++++++++ 0x003C0000 0x040000 (256kB) Configuration 0x00010000 0x1B0000 (1728kB) nanoCLR 0x001C0000 0x1E0000 (1920kB) Deployment

Deployment Map Empty

Description

HttpWebRequest downloads a 2372B file and then stream.Read returns only 0.

HttpClient never finishes downloading.

How to reproduce

example below

DownloadFileA - HttpClient - https://www.nuget.org/packages/nanoFramework.System.Net.Http.Client/1.5.113

DownloadFileB - HttpWebRequest - https://github.com/nanoframework/Samples/blob/main/samples/HTTP/HttpWebRequest/Program.cs

Expected behaviour

No response

Screenshots

No response

Aditional information

public class Program {

public static void Main() {
    DownloadFileB("https://letsencrypt.org/docs/integration-guide/", "I:\\test.txt");
}

private const string LetsEncryptCARootCertificate = "-----BEGIN CERTIFICATE-----\r\nMIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw\r\nTzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh\r\ncmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4\r\nWhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu\r\nZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY\r\nMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc\r\nh77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+\r\n0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U\r\nA5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW\r\nT8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH\r\nB5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC\r\nB5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv\r\nKBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn\r\nOlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn\r\njh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw\r\nqHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI\r\nrU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV\r\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq\r\nhkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL\r\nubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ\r\n3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK\r\nNFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5\r\nORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur\r\nTkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC\r\njNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc\r\noyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq\r\n4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA\r\nmRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d\r\nemyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=\r\n-----END CERTIFICATE-----";

internal static void DownloadFileA(string url, string filePath) {
    var httpClient = new HttpClient();
    httpClient.HttpsAuthentCert = new X509Certificate(LetsEncryptCARootCertificate);
    httpClient.SslProtocols = System.Net.Security.SslProtocols.Tls12;

    HttpResponseMessage response = httpClient.Get(url);
    response.EnsureSuccessStatusCode();

    using FileStream fs = new FileStream(filePath, FileMode.Create, FileAccess.Write);
    response.Content.ReadAsStream().CopyTo(fs);
}

internal static void DownloadFileB(string url, string filePath) {

    var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
    httpWebRequest.Method = "GET";

    //////////////////////////////////////////////////////////////////////
    // need to set the SSL protocol that the connection is going to use //
    // *** this MANDATORY otherwise the authentication will fail ***    //
    //////////////////////////////////////////////////////////////////////
    httpWebRequest.SslProtocols = System.Net.Security.SslProtocols.Tls12;

    // if the request is to a secured server we need to make sure that we either:
    // 1. provide the root CA certificate 
    // 2. the device has already stored a root CA bundle that will use when performing the authentication
    httpWebRequest.HttpsAuthentCert = new X509Certificate(LetsEncryptCARootCertificate);

    int totalBytesRead = 0;

    // get the response as a HttpWebResponse
    // wrap the response object with a using statement to make sure that it's disposed
    using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse()) {
        // wrap the response stream on a using statement to make sure that it's disposed
        using (var stream = httpWebResponse.GetResponseStream()) {
            // read response in chunks of 1k

            byte[] buffer = new byte[1024];
            int bytesRead = 0;

            Debug.WriteLine("Http response follows");
            Debug.WriteLine(">>>>>>>>>>>>>");

            using FileStream fs = new FileStream(filePath, FileMode.Create, FileAccess.Write);

            do {
                bytesRead = stream.Read(buffer, 0, buffer.Length);

                if (bytesRead > 0) {
                    totalBytesRead += bytesRead;
                    //Debug.Write(Encoding.UTF8.GetString(buffer, 0, bytesRead));

                    fs.Write(buffer, 0, bytesRead);
                }

                Debug.WriteLine("bytes read " + bytesRead);
            }
            while (totalBytesRead < httpWebResponse.ContentLength);

            fs.Flush();
            fs.Close();
        }

        Debug.WriteLine(">>>>>>>>>>>>>");
        Debug.WriteLine("End of Http response");
        Debug.WriteLine($"Read {totalBytesRead} bytes");
    }
}

}

Ellerbach commented 2 weeks ago

You are in a limited environment, constraints with CPU and memory. You are most likely hitting the memory allocation for those kind of tasks. If you need to read a very large file, then read by chunks like you are doing but write by chunks as well, don't feel up the memory like you are currently doing.

vtelenak commented 2 weeks ago

And how do I perceive memory now? It is read into a block that is 1KB and it is immediately stored on the Flash memory. First, it is an approach published in nanoframework samples. I need to do an OTA update and I am based on the article https://www.nanoframework.net/over-the-air-net-nanoframework-code-update-using-azure-iot/ I understand that I cannot work with large files, but over 2KB surely pe file with update will have.

Ellerbach commented 2 weeks ago

Adjust how you read the file, download chunk of the file and save it, download the next part. Exactly as you would do when you have a long list and you break it into pages.

nanoframework / Home