urequests.request(): unique headers only (dict keys)

[gnbl]

A web site I am attempting to scrape sends multiple "set-cookie"s.

These appear to be written to a dictionary, i.e. previous keys ("Set-Cookie") are overwritten: https://github.com/micropython/micropython-lib/blob/master/python-ecosys/urequests/urequests.py#L153

A custom function can be supplied - here's one that keeps a list:

def ph(l, resp_d):
    l = str(l, "utf-8")
    k, v = l.split(":", 1)
    if k in resp_d:
        resp_d[k] = [resp_d[k], v.strip()]
    else:
        resp_d[k] = [v.strip()]

But this of little use since the headers dict parameter of the request function has the same limitation (only one can be specified).

[andrewleech]

Do you mean you also want to be able to make a request with multiple set-cookie headers? Not just parse them from a response?

This duplicate header issue is discussed pretty regularly, both in regular python and here (https://github.com/micropython/micropython-lib/pull/217#issuecomment-1382895613)

[gnbl]

It is my understanding that for multiple cookies, multiple cookie headers are required. So to be able to handle sessions, both, parsing multiple set-cookie from a response and requesting multiple cookies is needed. But it seems you are well aware of this limitation - sorry I did not see that. Feel free to close as duplicate.

[gnbl]

Ah, it seems cookies are concatenated via semicolon (Wikipedia), so for the request, one cookie header is sufficient

[gnbl]

This concatenates the above array of response set-cookies into the request cookie:

cookie = "; ".join( c.split(";")[0] for c in r.headers["Set-Cookie"] )

"Works for me!" ;-) Now I need to figure out how to parse 50 kB of HTML..

[gnbl]

redirect drops arguments, if I see that correctly: https://github.com/micropython/micropython-lib/blob/master/python-ecosys/urequests/urequests.py#L163

[jonnor]

It seems that a fix was found. So I believe this issue can be closed?

[projectgus]

Agreed. @gnbl I'm not sure about the meaning of your last comment, but if it looks like a bug then please open a new issue for it and we can take a look. :pray: