Closed dicortazar closed 10 years ago
mm sorry for the too-generic-title of the issue u_u
It is not odd if you consider that the content can be uncompressed on the fly (i.e. if you open the file with a web browser).
I think the issue is in: https://github.com/MetricsGrimoire/MailingListStats/blob/master/pymlstats/utils.py#L77
If the file comes uncompressed and ends in .gz, it tries to compress it; to uncompress it later. However, there might be more corner cases.
I think it's not a bug in mlstats. When I try to download the files from that mailing list using other tools such as curl or wget the server closes the connection and I get a corrupeted file.
Using curl:
LANG=C curl https://www.redhat.com/archives/rdo-list/2013-September.txt.gz > mbox
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
99 2550 99 2549 0 0 3383 0 --:--:-- --:--:-- --:--:-- 3380
curl: (18) transfer closed with 1 bytes remaining to read
The downloaded file is in plain text and doesn't contain all the messages for that month.
Downloading the file with wget it has to retry once and the file is also corrupted. The first part of the file is in plain text and the rest seems to be compressed.
sduenas@Guybrush:/tmp/rdo$ LANG=C wget https://www.redhat.com/archives/rdo-list/2013-September.txt.gz
--2013-11-28 19:11:44-- https://www.redhat.com/archives/rdo-list/2013-September.txt.gz
Resolving www.redhat.com (www.redhat.com)... 2.20.215.214
Connecting to www.redhat.com (www.redhat.com)|2.20.215.214|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2550 (2.5K) [application/x-gzip]
Saving to: '2013-September.txt.gz'
99% [=====================================================================================================================================================================================> ] 2,549 --.-K/s in 0s
2013-11-28 19:11:45 (470 MB/s) - Connection closed at byte 2549. Retrying.
--2013-11-28 19:11:46-- (try: 2) https://www.redhat.com/archives/rdo-list/2013-September.txt.gz
Connecting to www.redhat.com (www.redhat.com)|2.20.215.214|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 186553 (182K), 184004 (180K) remaining [application/x-gzip]
Saving to: '2013-September.txt.gz'
100%[++====================================================================================================================================================================================>] 186,553 701KB/s in 0.3s
2013-11-28 19:11:47 (701 KB/s) - '2013-September.txt.gz' saved [186553/186553]
I also tried with the mboxes from https://www.redhat.com/archives/rhos-list/ getting the same weird behaviour.
I'm gonna close and label it as invalid.
Shouldn't mlstats trigger an error pointing the request rather than trying to open a corrupt file?
@gpoo Totally agree. I will try to fix it ASAP.
As a data point, this works:
$ curl -r 0- -O http://www.redhat.com/archives/rdo-list/2013-April.txt.gz
The "-r 0-" gets curl to request the full range of the file, and does it a bit differently such that the complete + correct file contents are sent. No idea why the weird behaviour occurs without it though.
@berrange has pointed out that this works too:
$ wget --header="accept-encoding: gzip" http://www.redhat.com/archives/rdo-list/2013-April.txt.gz
Thanks @justinclift for provinding more information, I re-opened the bug.
With the change, RDO mailing lists still fails.
Please check with https://www.redhat.com/archives/rdo-list/
However, adding that header at https://github.com/MetricsGrimoire/MailingListStats/blob/master/pymlstats/utils.py#L106 makes that work.
But, if you add such header there, a typical mailing list that works, fails... such as http://lists.openstack.org/pipermail/foundation/
When you say "with the change", what are you meaning?
Oh sorry @justinclift, I was pointing the commit [1] by @gpoo fixing this issue.
With that change to the source code, I still reproduce this error (#11).
[1] https://github.com/MetricsGrimoire/MailingListStats/commit/9015abd83a96d2eaf9492379889a40315cfb34b2
No worries. If it's any help this command retrieves all of the RDO compress mbox files 100% correctly (for me anyway):
$ for list in 2014-May 2014-April 2014-March 2014-February 2014-January 2013-December 2013-November 2013-October 2013-September 2013-August 2013-July 2013-June 2013-May 2013-April; do curl -O -r 0- http://www.redhat.com/archives/rdo-list/$list.txt.gz; done
Tested using both curl 7.21.4 and 7.36.0. Both worked fine.
I guess you'd need to find the urllib2 way to request a range. :frowning:
Thanks @justinclift . However, we need something more generic. If a new archive is added (for instance new months), we would need to automatically update that list and not in a manual way.
In any case, as you mentioned, this needs to be fixed through the urllib2 library.
No worries. Try this: https://github.com/MetricsGrimoire/MailingListStats/pull/26
Adding that header worked for me through python shell. It should work for the RDO list too. Hopefully it won't stop the other mailing lists from working.
I don't think that solves the issue. Without that, htmlparser.py
works for me. See the following snippet:
import urllib2
import gzip
import cStringIO
import sys
def retrieve(url):
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.2 ' \
'(KHTML, like Gecko) Ubuntu/11.04 Chromium/15.0.871.0 ' \
'Chrome/15.0.871.0 Safari/535.2'
headers = { 'User-Agent': user_agent,
'Accept-Encoding': 'gzip, deflate' }
request = urllib2.Request(url, headers=headers)
response = urllib2.urlopen(request)
data = response.read()
if response.info().getheader('content-encoding') == 'gzip':
data = cStringIO.StringIO(data)
htmltxt = gzip.GzipFile(mode='r', compresslevel=0,
fileobj=data).read()
else:
htmltxt = data
response.close()
return htmltxt
if __name__ == '__main__':
print retrieve(sys.argv[1])
If you call it test.py
and try: python test.py http://www.redhat.com/archives/rdo-list/2013-April.txt.gz
, it will retrieve the document correctly.
I missed that utils also uses urllib2
to fetch documents. htmlparser.py
it is just part of the issue. Fixing utils.py
should be straightforward, though.
@dicortazar You need more than adding the header. You need to process the request if coming gzipped. Or did you already tried that one?
@gpoo Hmmm, it's not liking that (test.py) on a non-RDO mailing list:
$ python test.py http://lists.openstack.org/pipermail/foundation/2012-March.txt.gz > 2012-March.txt.gz
$ gunzip 2012-March.txt.gz
gzip: 2012-March.txt.gz: unexpected end of file
@justinclift You should do:
$ python test.py http://lists.openstack.org/pipermail/foundation/2012-March.txt.gz > 2012-March.txt
$ more 2012-March.txt
The content is already decompressed.
@gpoo What's the filesize on that 2012-March.txt.gz when it comes out of test.py for you?
$ ls -la
total 336
drwxr-xr-x 5 jc staff 170 12 May 19:02 .
drwxr-xr-x 19 jc staff 646 12 May 18:52 ..
-rw-r--r-- 1 jc staff 162823 12 May 19:02 2012-March.txt.gz
-rw-r--r-- 1 jc staff 839 12 May 18:52 test.py
For me, it's not already decompressed.
$ more 2012-March.txt.gz
"2012-March.txt.gz" may be a binary file. See it anyway? y
^_<8B>^H^H^E[<F6>O^B<FF>2012-March.txt^@<EC>}i<97><DB>Ʊ<F6>g<E3>Wt<E6>8W3^NIq<9B>5^NG<B3>Ʋ%K<91><E4>(ysr<EE>^AI<90><84>E^B4@^NE^?<B8><BF>
<FD><AD><A7><AA><BB><D1><E0>2ESC![<B6><E9>8<B6>E^R<8D>^ky<EA><A9><EA><EB>$^^<A9><91><9F>|P<FE>D<C5><E3> J'~<E7>C%N<FA>J<BD>^O<BA><U+A95F>(u
<A8><AA><8D><93>z<F5>d<BF><A6><EA><D5>Z<U+077B><A6><87>N<D6><<B5><FB>^R^__<C4><C3>a^X${ޥ? NԻiPR^G<DC>^X^ZP<F5><BA>n<AE>\<U+076F>V<D5><EE>
<D5><DB>w{<DE><DB>i<FB>Ǡ39Q<FF>yE-<BE>E<8B><EA>:<9E>F]^?^R<C6><D1>^?<D5>^Oc<FA><AF>@Mb5N<E2>q<9C><FA>C^UG<CE>/T:I<A6><9D><C9>4 <BC><97>A
<9A><FA><FD><A0><FC><FC><F2>D}]k4j<D5><E3>js<BF>V<A9>^^^^5j<F5><E3><E6><F1><B3>`<E4><87><C3>4<B9>I*<9D>x<D4><F2><BC>w^C?<FA><90><A2><F1><E0>&H<E6>q^T<A8><D9> V^C?U<9D>8<9A>$a{:<A1><E9><A0><EF>'<83>@uô3MS<BC><94><FE><C6>'<AB><BA><A1>bj<8A><BF>^]<FB><E9>D<U+14265A>^E<C1><87><B4>
<A2><CE>R<F5>\<A5>A<A0><C2>I<89>^?<D0>^K<82>nESC^C<C6>^K{<FE>p^XD*<8C><F0><B6>YL^S^]F<AA>=<ED>&i<B9><EC>4<DF><E3>g<CF>c?<E9>*?<EA><D2>^O?^Dj:6_<BC>^K:<83>(<EC><D0><]ģQ8<99>^DA<B9><EC>^O㨯f<E1>d<A0>|<EE>RL?<A5>^<FA><9D> ~Z<A2>^WMT8^Z<C7><C9>ď&<AA>ESCL0Q<D4>^^<AD><B4>O/<A5>
<B9>o<FB><ED><E1>\<B5>^Cj0Q<B3>8<F9>@^S^S<E3>)<99><8A>(<F8>8Qc^ZHP<F1><BC>W<91><D3>Ǭ<EB><F8>l^TӤ<D0><E4><A6>!O<8E><9E><80>Y<A0>^F^A~<<F3>y9xL<E1>D^E~JESCJ<F5><E2><84><FA>=<8A>^S,^BMo^Z<D0><N0b<DA>|~^T<FE><CC>k<C0>O<B6><E9><B9><E8>&^^<DE>P<F7>B<A7>ESC^U<F5>.k<96>[<F2>;^]<DA>0a{H
<BB>^T<EF><F7>o^B<D5>I^B^_+<EE><D3>xfj^T<8C><DA><F4><B2>A8V<9D><A1><9F><A6>'<B4><84>i<DC> <B1>!_<CA>w<95><E5><8F>hr<A6><C3>.<AD><FE>\<A5>a?
{4<C3>ф<E6>n^X<CF>0<96> HK<CA><EF><FE>8M<F1>&^L<8D>6<E3>؏<F0><F3><9F><A9>/XS<9A>^E_ީ<82>!^]^O<85><AD>9<A6>Ʊ<F1>bgr<F1>x<E0>w^F<EA><ED>$<A1>>
<F4><C3>^N͌?<A9><A8><E7>^Q<CD>T<U+061D><D2>><C8><F7><CB>^_<A6><F1>cESC}<C7>^K=[ZXi^Y[*<E8>aji^QjOESC<F2>^V^Zc{<BE><A2>;<A5>ş,M<A4>L^E~<E5>
<8F><C7>1^]^L<F9>]<D6>)<BB>^F<E8><D7>,<FC>^P<F2><F6><99><B2><C4><E0>SK<DD><E1><81>N^F!-<DE>`2^Y<9F><}<8A>^_Vr<E2><EB><E9><DF>qf#?<EA>^DO<B3>#<FD><F4><AD>^]<9F><E7><BD>^_<84><C3>@F<C8>^GBΏ<CC>C<ֻ^O<A7><85><B7>Q;<A0><93><DC>ESCƲ<97>^R4<A9>v<E9>|ӎ4<D3>@<A3><8B>b<F7><CF><X^VC<D4>[:
<AC>8<FD><C1><84>DG<B4>W<E2>^A<A8><C9>|<CC><C7>^?^T~^L<DC><D9>O'!5M<82>e<C4>'<80>{<80><C5>l'<B1><DF>^M蘥<D3>1Nv^E^B<CF>^L<80><B7>^AfA<FD>4
<A5><DF><F0>^@<FC>6^N4<9F>R^R"t<AE>:<E9>-R<A5>D<ED><D2>^^<C1><81><A5>^_$<F1><90>;<F7><FA><U+074B>^T<E2>^Q2n<86> <F9><89>^FGg^@ESCp7<EC><D1>(<E7>{"R<86>~^G"^Dg4NE<F0><CE><C2>t<80>!<DC>ē<C0><9C>^<DE>!<E8>^I^E<93>N<85><DE>*[5[<9D>n<E2><F7>&
<E7><89>^OxJ^S;<D1>r<92><DE><DF>#^Y<C7>;:<8A>gt,0<95>^Q^K<9C>t<C0><BB><F6>C^P<8C>嗎|<EF><C7>z <B8>^G^_IV<E1>ϓ^A<89><87>nj<A4><FF>$^^<D3>&IJ<BD>ESC<D0><F8><92><F9><93>T<F7>E?<C8>;<8D>^T<AA>"^A <FD>^K<A3>N<9C><D0>Z<B0>J<E3>wƝ<E9>(<88>&Z<EE><E7><C6>E3K<B2><AA>O<F2><9F>D^AMQ"<F2>^M=<E9><D2>ѕ<E5>t^D/<BD><8F><F6><C1><87><90><E4>iܓe<BB>Dw<F0><80>]5<85><B7><D1>^PE<<FA>$ك<9B>0<9E><92><9C><81>nDo<9F>P#t<80><C7><D3><F6><90>
<96>^C^O<FB><90>P<F4>8wt(^Z<80>w$^T<9B><96>W^_<F0>C^Qda<A0><CF>^B<8E>l<U+0515>^<D3>W<81>^Q<A8><F8>^D<A2>Q<A4>^N<FD>0<E8><F5>h<91>y<AA>D<94>
^L<C3>^O<AC><F3>!<E5>i<EE><FC>~?<81><AC><A6><8D>ݥ^U^X<86>$<B8>v<87>,*<FC><A4>3<D8><E3><F5>E<8B>>&.<85><94><C0><BB><E9>?q^@<B3>^<A5><FA>}3Y^R<BD><F4><AC>ޠ^KH/<DC><D0>0H<A6><F7>&Z<87><9B>ו<D6>M'<CF>9M^W<F5>`D<EF>b^U˲sL<9B><90>f<9A>N̢<C0><A2><D9>bŀ<99><A5><B1>M&q^R^Es^L^W_<D0>.9ESC
<86><9D>@}<E7>l<C1><AE>y<B1><U+EA48><BC>N<BE>^W<F8><B2>;<ED><E8>_ۣ;^L<FA><D4> <B3>ь<92><D4>˛iI<96>^E<98>G<92><F1>XV<BC><80><9E><E1><B7><D1>
<FE><CA><DA>0<A3><F6><E9><A4><C4>]9<CB>h<98>6<AB><D9><C9>8<D3>=<9A><B9><C4>X^Syya<A4>^Nz9<88><E9>uZ<98><CA>^V<A1>7cESC<D0>ESC<92><80>^V^Er"
^Qq2 ]@<A2><DB><F4><BA><83>}^_<F7>x";?M<C3>4<94>F^Sj(<E9>SCZ<86><AA><EC><DC><E9>MH<C7>G<F6><FA> <A6>7͂v^X<F9> <9F><ED>w<83>i<D2>M<BB><A4>
<BF><FF>G]'!<FE>#<DB><U+DEFC><E3><D4>^Q}w<BC><C7>C<F6>Y^U<F8><DD>.<BF><9A><A6><DA>X5^Uk`<FA>}^Xs<AC><EA><FD>^D<CB>HESCqēa%noj<AC>;k^G<FF>
<89><F4>^N<EC>j<EF>Y:<A6>^?<CD>;b^{<9D>`8<Q<FB><B5>z<F9><F0><B8>V<AE>6<F6>^O<<8F><AD><F3>u&=^MBL<FA>c<98><F4><8D><DA>^F&<FD>`Z<A2><B1><BB>&
<BD>4W<A4>I<EF>}<91><D9><F4>ϣ<F2><9B>`<<9C><97><DF><C5><F7>4<EA><DF>^D=:<F8><A4><CA><D3>{><B0><E4>6<D4>^O^ZǵZ<85>^^<AC><D7>^O<F6><8F>kKn<C3>7tZ<E7>OH<9F><C3>p|<C2>B<A9>^Sv<C5><E4><E0><DD><C4><C2><DF><EC>(^Vмw<B0><B7>h<DB>cC<D1>t7<C7>#աӔ<D0><F8>q<D8><C8><F1><8A><EC><97><C7>~<F6>%
<B5><DA><F1>'^P)A<^^<B2>b<EC><86>=^^%I~?Ɏ:<99><E5>C6^^g0<94>I<8C><C2>^C<99><84><A3><A0>^B<BF><EE> <ED><AF>^A<E9>,h29<AA>>٤lF<A3><95><B0>^S
<8E>ŗ<C1>0<C8>^Z<CF><CC><F3>^^6W/^^~pm^D<C8>_j^4^^<EB>SR<EB>]<E7><A8>,;IV<B7><89>i<A7>^U<AE><CC>^P<89><EA><B1>><94><D4>OQ<C0>"<AA>&<C9>\<B4>^PL<CF>8b<E9>?
^BQ<C9>q<<D4><D6>^Q^M<93>扜<A8><AE><F9><B6>=%<C1><9A><90><DC>!<B5>ESC<C0>*p<DC><^VCI<C8>?g<A1><CD>F^LI<FB><F9><C4>^_~P<E9><9C><FA>3°<A6>i
<90><9F><C8><E7><F8>ِ<EC>y6<B4>`đ<92>`^Y6<A3><83><99>^?<B7>^HJxK<E2>b<91><F2>$#<A1>^OE^Vb<FA>ih<^S$(ٛ"<D5><C3><EE>^Y<9B><80>,<AA>a˜x^mO<A9>^_"V<8B>oL'Kj^^OI(<AB><EF>_<BD><A3><B9><91><FD>ׅ<C6>a^Q<EC><93>^A^Xv<FB><81><BA> I<CD>^O0j<B3><B0>"9<87>C<B8> l <95><B2>9<E8>%^A<FB>F
<B4>jXV,^F<D6>Y<EC><9F><C0><B4><C9>㢵<98><F3>OG1<9F><E8>ھ,@<U+009A><96><B6><AA><CF>J<83><FB><97><F5><CB>i%<9A><B2><B0><9F>FCh_<8C><C6>o<A7>
$
This is what I get:
$ python test.py http://lists.openstack.org/pipermail/foundation/2012-March.txt.gz > 2012-March.txt
$ ls -l 2012-March.txt
-rw-rw-r-- 1 gpoo gpoo 1370 May 12 11:07 2012-March.txt
$ head -3 2012-March.txt
From nobody Mon May 12 11:07:16 2014
From: jgascon at gsyc.escet.urjc.es (Jorge Gascon Perez)
Date: Wed, 14 Feb 2007 19:46:10 -0000
``
Hmmm, it's weird that it's saying "unexpected end of file".
$ gunzip 2012-March.txt.gz-test.py
gzip: 2012-March.txt.gz: unexpected end of file
There's a 1 byte filesize difference compared to the curl method:
$ python test.py http://lists.openstack.org/pipermail/foundation/2012-March.txt.gz > 2012-March.txt.gz-test.py
$ curl -r 0- http://lists.openstack.org/pipermail/foundation/2012-March.txt.gz > 2012-March.txt.gz-curl
$ ls -la
total 656
drwxr-xr-x 6 jc staff 204 12 May 19:07 .
drwxr-xr-x 19 jc staff 646 12 May 18:52 ..
-rw-r--r-- 1 jc staff 162822 12 May 19:07 2012-March.txt.gz-curl
-rw-r--r-- 1 jc staff 162823 12 May 19:06 2012-March.txt.gz-test.py
This is confusing.
@gpoo It's probably due to different version dependencies or something. Not real sure.
I'll leave it to you. :smile:
No, my bad. I was in the wrong directory trying another test.py file :-P
I see the issue, in the script I was using print
. Anyway, if you change that for something like:
if __name__ == '__main__':
with open('output.gz', 'w') as f:
f.write(retrieve(sys.argv[1]))
should work. Beware with the name, it could be misleading depending of the content negotiation.
No worries, I'm sure you'll get it working. :smile:
(I've already moved on to other bits on my Task list for today)
@gpoo can you review this pull request? This one fixes the two problems that @dicortazar commented above:
Accept-Encoding
without including some code to uncompress the body of the request, caused that mboxes were compressed twice. I added the same code you included on htmlparser
to fix it.For further changes I have two proposals:
WRT Requests, I guess the main difference with urllib2 is in fact its more "reasonable" API. However, that would amount to adding another dependency to mlstats, since urlib2 is a part of Python standard libs, and Requests (AFAIK) is not. That's not a big issue, but well, i just wanted to mention it.
At some point I dismissed Requests
for the same reason explained by @jgbarah and because it is used only twice, which could be wrapped in once eventually. I don't have a strong opinion, though.
Regarding to magic numbers to identify the file type, I agree. I had the same idea, though for different purpose (just store the mboxes compressed and uncompress them on the fly, instead of duplicating the space used in ~/.mlstats
).
Beware, that you can be grabbing a file name foo.gz
, but receiving it deflated one, and viceversa.
I will check the patch in some hours, a first look at it seems ok though.
When running mlstats against this mailing list: https://www.redhat.com/archives/rdo-list, the tables are not filled.
One odd issue here is that the directory that should contain the compressed files, contain the uncompressed files but still with the original file. This is that files still conserve the original name ending with .gz, but if the command "file" is used, they are ASCII.
Output of the tool:
[13:11:14] /usr/local/bin/mlstats --no-report --db-user="xxx" --db-password="" --db-name="database" --db-admin-user="root" --db-admin-password="" "https://www.redhat.com/archives/rdo-list" Already downloaded https://www.redhat.com/archives/rdo-list/2013-April.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-May.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-June.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-July.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-August.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-September.txt.gz Already downloaded https://www.redhat.com/archives/rdo-list/2013-October.txt.gz Found substring 2013-November in URL https://www.redhat.com/archives/rdo-list/2013-November.txt.gz... Retrieving https://www.redhat.com/archives/rdo-list/2013-November.txt.gz... Unknown URL or directory: https://www.redhat.com/archives/rdo-list. Skipping. 0 messages analyzed 0 messages stored in database acs_mlstats_redhat_rdo_2437 0 messages ignored by the parser
In addition, removing the "try expect" at https://github.com/MetricsGrimoire/MailingListStats/blob/master/pymlstats/main.py#L300 the new error is the following one:
Traceback (most recent call last): File "/usr/local/bin/mlstats", line 37, in
pymlstats.start()
File "/usr/local/lib/python2.7/dist-packages/pymlstats/init.py", line 154, in start
web_user, web_password)
File "/usr/local/lib/python2.7/dist-packages/pymlstats/main.py", line 145, in init
t,s,np = self.__analyze_mailing_list(mailing_list)
File "/usr/local/lib/python2.7/dist-packages/pymlstats/main.py", line 297, in analyze_mailing_list
archives_to_analyze = self.set_archives_to_analyze(mailing_list, archives)
File "/usr/local/lib/python2.7/dist-packages/pymlstats/main.py", line 433, in __set_archives_to_analyze
mailing_list.mbox_dir)
File "/usr/local/lib/python2.7/dist-packages/pymlstats/utils.py", line 137, in uncompress_file
files = [extractor.gzExtraction(new_filepath)]
File "/usr/local/lib/python2.7/dist-packages/pymlstats/fileextractor.py", line 71, in gzExtraction
outputfileobj.write(gzipfile.read())
File "/usr/lib/python2.7/gzip.py", line 254, in read
self._read(readsize)
File "/usr/lib/python2.7/gzip.py", line 296, in _read
self._read_gzip_header()
File "/usr/lib/python2.7/gzip.py", line 190, in _read_gzip_header
raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file