search

eftsung / pygr

Automatically exported from code.google.com/p/pygr
0 stars 0 forks source link

Make download-progress messages in downloader.py more terse #5

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Right now, if one tries to download genome assemblies from UCSC, urllib 
module prints out too often progress messages. Thus, it is not possible to 
see any other mmessages. We have two choices.

(1) prints out dots per 0.1% not per packets from urllib
(2) prints out information lines per 1% progress

Anyway, we need to get file size information prior to downloading any 
files.

Original issue reported on code.google.com by deepr...@gmail.com on 22 May 2008 at 6:58

GoogleCodeExporter commented 8 years ago

Original comment by deepr...@gmail.com on 31 Mar 2009 at 8:43

Attachments:

GoogleCodeExporter commented 8 years ago 
Namshin,

Unfortunately your patch doesn't do exactly what it is supposed to do yet - 
while 
the number of output lines is indeed significantly smaller than without the 
patch, 
downloader can under appropriate circumstances still produce much more output 
than 
just one line every 10 percent. 

Example, assuming a file 1048576 bytes long and urrlib calling 
download_monitor() 
every 1000 bytes:

* downloaded 103865 B: int(100*103865./(1024*1024)) = 9, message suppressed
* downloaded 104865 B: int(100*104865./(1024*1024)) = 10, message "10 % done" 
is 
printed
* downloaded 105865 B: int(100*105865./(1024*1024)) = 10, message "10 % done" 
is 
printed again
[...]
* downloaded 114865 B: int(100*114865./(1024*1024)) = 10, message "10 % done" 
is 
printed again
* downloaded 115865 B: int(100*115865./(1024*1024)) = 11, message suppressed

To trigger this under natural circumstances, try downloading as large a data 
set as 
possible (e.g. hg18).

Original comment by mare...@gmail.com on 8 Apr 2009 at 1:47

GoogleCodeExporter commented 8 years ago

Original comment by deepr...@gmail.com on 9 Apr 2009 at 6:57

Attachments:

GoogleCodeExporter commented 8 years ago 
Still no dice, sorry - it seems printDict gets overwritten with ones every time 
download_monitor() is called so setting its appropriate fields to zero, while 
sound 
in theory, has no effect in practice, resulting in the same behaviour as with 
the 
first version of the patch.

Original comment by mare...@gmail.com on 9 Apr 2009 at 6:00

GoogleCodeExporter commented 8 years ago 
Earlier on today I looked into implementing persistent variables in Python and 
decided to take on this issue myself. Attached you will find a patch which, at 
least 
on leelab2, causes download-status messages to appear no more frequently than 
every 
10 percent.

Original comment by mare...@gmail.com on 9 Apr 2009 at 11:26

Attachments:

GoogleCodeExporter commented 8 years ago 
Whoops, forgotten to comment on tag changes. Given that my patch has been 
written 
from scratch, it makes sense for me to assume ownership of the issue - and 
Namshin's 
involvement makes him a perfect reviewer. Namshin, could you verify that my fix 
works?

Original comment by mare...@gmail.com on 9 Apr 2009 at 11:41

GoogleCodeExporter commented 8 years ago

Original comment by mare...@gmail.com on 9 Apr 2009 at 11:43

GoogleCodeExporter commented 8 years ago 
FYI, in accordance with our new policy I have created a new GitHub branch 
devoted to 
this issue: http://github.com/ctb/pygr/tree/issue005

Original comment by mare...@gmail.com on 10 Apr 2009 at 12:26

GoogleCodeExporter commented 8 years ago 
shouldn't we switch to using logger.info() instead of print >>sys.stderr?  I 
thought
that was now mandated as our standard policy...

Original comment by cjlee...@gmail.com on 10 Apr 2009 at 12:32

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Thanks Marek!

        print >>sys.stderr, 'downloaded %s bytes (%2.1f%%)...' \
                % (bytes, percentage)
        logger.info('downloaded %s bytes (%2.1f%%)...' % (bytes, percentage))

FYI. Pygr uses sys.stderr 71 times in pygr/ and 5 times in tests/oldtests.

Original comment by deepr...@gmail.com on 10 Apr 2009 at 12:43

GoogleCodeExporter commented 8 years ago 
Chris: this technically isn't a part of this issue but since pretty much every 
line 
of download_monitor() has changed comparing to the original, might as well get 
rid 
of the print statement... I've changed download_monitor() as suggested by 
Namshin 
(i.e. using the INFO level) plus done the same in download_unpickler() so that 
all 
download-related messages use the same output facility. Both changes have been 
committed to the issue005 branch on GitHub.

Namshin: just to clarify, does your comment mean you have successfully verified 
the 
fix?

Original comment by mare...@gmail.com on 10 Apr 2009 at 10:10

GoogleCodeExporter commented 8 years ago

Original comment by deepr...@gmail.com on 10 Apr 2009 at 10:22

GoogleCodeExporter commented 8 years ago 
Chris has pushed the code into the master and the dedicated branch has been 
deleted. 
Closing the issue.

Original comment by mare...@gmail.com on 14 Apr 2009 at 9:20