search

zopefoundation / zope.publisher

Map requests from HTTP/WebDAV clients, web browsers, XML-RPC and FTP clients onto Python objects
Other
3 stars 13 forks source link

Should I see byte strings in BrowserRequest.form on Python 3? #40

Closed mgedmin closed 5 years ago

mgedmin commented 5 years ago

I'm porting an old Zope 3 application to Python 3. While running my functional tests on Python 3.7 I've been surprised to discover that BrowserRequest.form[key] sometimes contains byte strings.

The request in question has Content-Type: application/x-www-form-urlencoded, and wsgi.input is a BytesIO that has myfield=...%xx%yy... with UTF-8 bytes correctly urlencoded.

All form fields that are pure ASCII come out as native str objects.

(I'm testing with the latest version of every package from PyPI, barring any unexpected releases in the last couple of days. zope.publisher is version 5.0.1. The HTTP request was generated by zope.testbrowser/WebTest.)

mgedmin commented 5 years ago

Turns out there are two problems here: one is caused by the same root cause as #41 (buggy charset handling), the other one is that when the request data cannot be decoded using any of the autodetected encodings, zope.publisher gives up and leaves byte strings in request.form, letting the application handle them as it wills.

It's because this loop: https://github.com/zopefoundation/zope.publisher/blob/02e5f31a8e1950c5bd24f5a60d77025dd06bb330/src/zope/publisher/browser.py#L260-L266

does not handle the case when none of the loop iterations succeeded the .decode().