urllib.request example should use "with ... as:"

[40e92aad-fc19-4daa-8468-b436a15f1b56]

BPO	12955
Nosy	@terryjreedy, @ncoghlan, @orsenthil, @ezio-melotti, @merwok, @berkerpeksag, @vadmium
Files	urlopen-with.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/orsenthil' closed_at = created_at = labels = ['type-feature', 'docs'] title = 'urllib.request example should use "with ... as:"' updated_at = user = 'https://bugs.python.org/ValeryKhamenya' ``` bugs.python.org fields: ```python activity = actor = 'berker.peksag' assignee = 'orsenthil' closed = True closed_date = closer = 'berker.peksag' components = ['Documentation'] creation = creator = 'Valery.Khamenya' dependencies = [] files = ['38049'] hgrepos = [] issue_num = 12955 keywords = ['patch'] message_count = 15.0 messages = ['143836', '143841', '144141', '144142', '144146', '144165', '144167', '144168', '144195', '152574', '152594', '235580', '235581', '240550', '240551'] nosy_count = 11.0 nosy_names = ['terry.reedy', 'ncoghlan', 'orsenthil', 'ezio.melotti', 'eric.araujo', 'docs@python', 'Valery.Khamenya', 'python-dev', 'javawizard', 'berker.peksag', 'martin.panter'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue12955' versions = ['Python 3.4', 'Python 3.5'] ```

[40e92aad-fc19-4daa-8468-b436a15f1b56]

The following intuitive construction

with urllib2.build_opener().open() as:
    ...

leads to AttributeError: addinfourl instance has no attribute '__exit__'

http://docs.python.org/library/urllib2.html says almost nothing about concept of closing.

Could it be possible to add a canonical "with ... as:" example for urllib2.build_opener() ?

Thanks, Valery

[40e92aad-fc19-4daa-8468-b436a15f1b56]

JFYI, the attempt to use closing() manager from contextlib have revealed to the following issue:

https://bugs.pypy.org/issue867

[terryjreedy]

As I understand this, you are asking that 2.7 urllib2.build_opener().open(), which in 3.x is urllib.request.build_opener().open(), be upgraded to return an object that works as a context manager. Unless the docs say that this should already be the case, this is a feature request for 3.3.

I am unable to test whether this feature is already present (in 3.2.2). Your example line "with urllib2.build_opener().open() as:" has an obvious syntax error. When I correct that and adjust for 3.x

import urllib.request as ur
with ur.build_opener().open() as f:
    pass
#
TypeError: open() takes at least 2 arguments (1 given)

The doc for build_opener says it returns an OpenerDirector instance. help(ur.OpenerDirector.open) just says it needs a 'fullurl'. But when I add 'http:www.python.org' as an argument, I get urllib.error.URLError: \<urlopen error no host given> I do not know what else is needed.

Please copy and paste the ACTUAL (minimal) code you ran to get the AttributeError.

[orsenthil]

Just as a quick guideline. If you are talking about context manager support for urlopen, it is available and present in 3.x but not on 2.7. And I fear, it is late to make it available on 2.7, because it is a feature.

In any case, as Terry requested, a simple snippet would help us understand the problem and original poster's expectation.

[40e92aad-fc19-4daa-8468-b436a15f1b56]

Terry, Senthil, thanks, for replying to this ticket. OK, to the question:

1. @Terry, here is the full example as for CPython 2.7 I am talking about and the output:

#################

from urllib2 import Request, build_opener

request = Request('http://example.com')

with build_opener().open(request) as f:
    txt = f.read()
    print '%d characters fetched' % len(txt)
####################

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: addinfourl instance has no attribute '__exit__'

2. @Senthil, regarding the statement being a feature. I assume, that to open a connection, to read from a connection and to close it -- are the fundamental functions for what urllib2 was in particular created for. I was looking in docs for some hints of "canonical" way of doing this using urllib2.opener. After I have failed to find any guidance in docs, I've created this ticket. That is, I assume that no new feature is needed, but just a documented 5-lines example about a typical way of doing the above operations, especially *closing* the connection.

regards, Valery

[terryjreedy]

On 3.2.2, your example (adapted) produces
>>> 
2945 characters fetched

So, as Senthil said, the requested feature already exists. But it cannot be added to the 2.7 series; the Python 2.7 *language* is feature frozen. 2.7.z bug fixes serve to make the implementation better match the intended language.

Prior to the addition of with..., the standard way to close things was with explicit .close() or implicit closing when the function returned or the program exited. In the 3.2 docs, "20.5.21. Examples", there are several examples with f = \<some opener> \<do something with f> (closing ignored) Where possible, these could and should be changed in 3.2+ docs to with \<some opener> as f: \<do something with f> to promote the new idiom. This had been done in other chapters. Perhaps there is also a need for 'X supports the context manager protocol' to be added here or there. But showing that in the examples would make the point.

I have changed the title of this issue to match.

[40e92aad-fc19-4daa-8468-b436a15f1b56]

Guys, in my item 2 the simplistic goal was stated clearly: open, read and close.

Do you confirm that this basic sequence is not supported by urllib2 under 2.7 ?

(I just requested for a tiny documentation update entry)

regards, Valery

[orsenthil]

Valery, yes. I shall update 2.7 documentation with this known limitation and 3.x documentation with the example usage scenarios.

Thanks!

[merwok]

[Terry]

But when I add 'http:www.python.org' as an argument, I get urllib.error.URLError: \<urlopen error no host given>

Your URI lacks a host (netloc, in RFC parlance) component:
>>> urllib.parse.urlparse('http:python.org')
ParseResult(scheme='http', netloc='', path='python.org', params='', query='', fragment='')

[terryjreedy]

Senthil: http://docs.python.org/dev/library/contextlib.html#contextlib.closing currently has this example:

from urllib.request import urlopen
with closing(urlopen('http://www.python.org')) as page:

which is misleading in that the object returned from urlopen \<class 'http.client.HTTPResponse'> has __enter and __exit methods and therefore is a c.m. in itself and does not need to be wrapped in closing(). I did not really understand from your comment whether there is any need to use closing() with anything returned in urllib.

At the moment, shelves are not C.M.s, and would make better examples, but bpo-13896 suggests 'fixing' that also.

[merwok]

Either we find a commonly used stdlib class that is not a context manager but has a close method and is not going to become a context manager (I can’t see why such a thing would be), or we can add something like: “closing is useful with code that you can’t change to add context management, for example urllib.urlopen before Python 3.3”.

[vadmium]

bpo-22755 is about the example arms race for contextlib.closing().

[vadmium]

Here is a patch to change the urlopen() examples to use context managers where appropriate.

There were also a few examples of handling HTTPError which I didn’t touch, because the whole file object versus exception object thing is probably a separate can of worms.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 6d5336a193cc by Berker Peksag in branch '3.4': Issue bpo-12955: Change the urlopen() examples to use context managers where appropriate. https://hg.python.org/cpython/rev/6d5336a193cc

New changeset 08adaaf08697 by Berker Peksag in branch 'default': Issue bpo-12955: Change the urlopen() examples to use context managers where appropriate. https://hg.python.org/cpython/rev/08adaaf08697

[berkerpeksag]

Great patch. Thanks Martin.