urequests.get ignores content-length header

[GM-Script-Writer-62850]

When connecting a server running this php sample

<?php
set_time_limit(0);
ob_start();
header('Content-type: text/plain');
header("Content-Encoding: none");
echo "OK";
header('Content-Length: '.ob_get_length());
header('Connection: close');
ob_end_flush();
@ob_flush();
flush();
if(session_id()) session_write_close();
sleep(10);
die("Client should never see this");
?>

if you view this page in a web browser or in curl you will only get OK, however urequests.get ignores the content length and waits for sleep to finish and get the die message before returning content, urequests.head has the same issue except you can't get the status_code

i managed to fix this, but my fix does nothing when i import it even though it works if i copy/paste the library inside my script this was how i fixed it

    @property
    def content(self):
        if self._cached is None:
            try:
                if(self.headers["Content-Length"]):
                    self._cached = self.raw.read(int(self.headers["Content-Length"]))
                else:
                    self._cached = self.raw.read()
            finally:
                self.raw.close()
                self.raw = None
        return self._cached

[jimmo]

I think this probably makes sense to fix!

i managed to fix this, but my fix does nothing when i import it even though it works if i copy/paste the library inside my script this was how i fixed it

How are you copying your fixed script to the device. Which directory are you putting it in. Note that sys.path by default is ["", ".frozen", "/lib"] so if you put it in "/lib" the frozen copy will take precedence. Place the file in the root directory instead (alongside your main.py if you want it to override the frozen one).

[GM-Script-Writer-62850]

i was putting it in /lib/urequests.py i got it to work by using /lib/urequest.py

i guess urequests in included on the stock pico firmware, thought it was not but i must have made a typo in the lib name when i tried to import it the 1st time

Now i can just use import urequest as urequests or just use import urequest instead of import urequests as urequest

it took deleting /lib/urequests.py for me to figure out what was going on

[bmidgley]

The problem arises for me with a service that sends a payload without a final newline. But I couldn't use the workaround. I ended up using a terrible hack where I read(1) until I notice the ending } in the payload.

[jonnor]

Is this still a problem in the latest versions of requests library?

[GM-Script-Writer-62850]

Yes this is present in MicroPython v1.24.0-preview.224.g6c3dc0c0b on 2024-08-22; Raspberry Pi Pico W with RP2040 PICO W Test Code:

#!/usr/bin/python3
import requests as urequest
import uasyncio
sleep=uasyncio.sleep
sleep_ms=uasyncio.sleep_ms

class GPIO:
    wlan=None

async def wifi():
    from wifi_auth import ssid, password
    import network

    GPIO.wlan = network.WLAN(network.STA_IF)
    wlan = GPIO.wlan
    wlan.active(True)
    #wlan.config(pm = 0xa11140)# Power management is very very bad, ping time is around 1000x worse and packet loss insane
    wlan.connect(ssid, password)

    while True:
        while True:
            wstat=wlan.status()
            if wstat < 0 or wstat >= 3:
                break
            print('Waiting for WiFi connection...')
            await sleep(1)

        if wlan.status() == 3:
            status = wlan.ifconfig()
            print('Wifi Connected; ip =',status[0])
            while wlan.isconnected():
                await sleep(30)
            print('WiFi Down')
        else:
            print("Failed to connect to wifi; retry in 30 seconds")
            await sleep(30)
        wlan.connect(ssid, password)

async def wait4wifi():
    uasyncio.create_task(wifi())
    while GPIO.wlan is None:
        await sleep(1)
    while not GPIO.wlan.isconnected():
        await sleep(1)
    print('wifi up')

uasyncio.run(wait4wifi())
print("ifconfig:",GPIO.wlan.ifconfig())
print("status:",GPIO.wlan.status())

r=urequest.get("http://10.0.0.69:8080/die.php")
print(r.status_code)
print(r.content)
print(r.text)
r.close()

Ouput:

Wifi Connected; ip = 10.0.0.190
wifi up
ifconfig: ('10.0.0.190', '255.255.255.0', '10.0.0.1', '10.0.0.1')
status: 3
200
b'OKClient should never see this'
OKClient should never see this

EDIT: Note that aiohttp does not have this issue