downloadSubs: Error while writing subtitle file

[GoogleCodeExporter]

What steps will reproduce the problem?
1. not sure, using a mapped network drive to store the subs

Problem in the downloadSubs is that both the reading from the api and the 
writing to the file is in the same try / except block.
With this error message it's not clear which one failed.

In my case the srt file is created with a file size of 0kb.
Not all the subs in the queue display this behavior, most of them get 
downloaded just fine.
The 0kb created srt files do prevent me from retrying in the next batch run as 
it will detect a subtitle already there.

Original issue reported on code.google.com by nielswij...@gmail.com on 10 Jul 2012 at 12:17

• Merged into: #131

[GoogleCodeExporter]

If have seen some cases where the subtitle on bierdopje is corrupted. Could you 
please report which subtitle caused this problem? 

The API and writing might be separated, but there was a reason why I kept them 
in the same try catch.

Also please supply more debug information, try to reproduce it or find some 
consistance in when the problem occur. 

Original comment by romke.va...@gmail.com on 11 Jul 2012 at 5:29

[GoogleCodeExporter]

Have upped the # of log files / maximum log filesize as i don't have the logs 
for the incident anymore.
Also seperated the .read and .write into 2 different try/catches and increased 
the amount of info written to the logfile.

Results: in this test 4 subtitles consistently won't download with auto-sub and 
the download throws a socket.timeout exception.

Firefox can download them but with a huge delay; see the attached wireshark 
summary log.
Tried multiple times with both auto-sub and just using firefox but consistent 
results for subs in the following debug log:
So this problem is not in auto-sub but should be reported to bierdopje imho.
By seperating downloading & writing you do prevent writing 0kb srt files though.
You don't need 2 seperate try / catches though.

2012-07-11 20:28:32,588 DEBUG  downloadSubs: Trying to download the following 
subtitle 
http://www.bierdopje.com/downloads/sub/24268/Friends.S6E19.DVDRip/apikey/BB442E7
744E9B541
2012-07-11 20:29:02,619 ERROR  SRT READ: Error while fetching subtitle file. 
Destination: F:\Series\Friends\Season 6\Friends - S06E19 - The One With Joey's 
Fridge.autosub.nl.srt
2012-07-11 20:29:02,651 ERROR  SRT READ: <class 'socket.timeout'>

2012-07-11 20:29:02,846 DEBUG  downloadSubs: Trying to download the following 
subtitle 
http://www.bierdopje.com/downloads/sub/23932/Friends.S09E05.DVDRip/apikey/BB442E
7744E9B541
2012-07-11 20:29:32,875 ERROR  SRT READ: Error while fetching subtitle file. 
Destination: F:\Series\Friends\Season 9\Friends - S09E05 - The One With 
Phoebe's Birthday Dinner.autosub.nl.srt
2012-07-11 20:29:32,897 ERROR  SRT READ: <class 'socket.timeout'>

2012-07-11 20:29:33,216 DEBUG  downloadSubs: Trying to download the following 
subtitle 
http://www.bierdopje.com/downloads/sub/23939/Friends.S09E13.DVDRip/apikey/BB442E
7744E9B541
2012-07-11 20:30:03,256 ERROR  SRT READ: Error while fetching subtitle file. 
Destination: F:\Series\Friends\Season 9\Friends - S09E13 - The One Where Monica 
Sings.autosub.nl.srt
2012-07-11 20:30:03,289 ERROR  SRT READ: <class 'socket.timeout'>

2012-07-11 20:30:04,819 DEBUG  downloadSubs: Trying to download the following 
subtitle 
http://www.bierdopje.com/downloads/sub/23943/Friends.S09E17.DVDRip/apikey/BB442E
7744E9B541
2012-07-11 20:30:34,861 ERROR  SRT READ: Error while fetching subtitle file. 
Destination: F:\Series\Friends\Season 9\Friends - S09E17 - The One With The 
Memorial Service.autosub.nl.srt
2012-07-11 20:30:34,894 ERROR  SRT READ: <class 'socket.timeout'>

Original comment by nielswij...@gmail.com on 11 Jul 2012 at 6:56

Attachments:

• srtdownload.log

[GoogleCodeExporter]

Just realized that the api forbids default browsers to download using an api 
key and may apply rate limiting / bandwidth limiting.
So i tried downloading entirely manual using the site but still got the same 
timing results for all 4 of these files. 
Other subs from same season download ok.

Original comment by nielswij...@gmail.com on 11 Jul 2012 at 7:12

[GoogleCodeExporter]

Correct, api is rated limited for incorrect user agents to 0.1kb/s.

Sometimes subtitles on bierdopje.com are corrupted, when I tested I had a 
couple of files which resulted in a crash (i think it was) House.S01E01.avi. 
Spend hour debugging auto-sub, but it was caused by bierdopje.

To fix this I can set the socket.timeout to a more common http timeout (which 
is 300 seconds). This will result in 'slower' downloading but will download the 
corrupted subtitle. Also when it timesout it should remove the subtitle. What 
is your opinion?

Original comment by romke.va...@gmail.com on 12 Jul 2012 at 2:35

• Changed state: Accepted

[GoogleCodeExporter]

I wouldn't start writing until the entire subtitle has been read.
Just using a temporary variable to store the subtitle would serve that purpose; 
if a timeout of 300 is too long and it still throws an exception you won't have 
to deal with the existing 0byte subtitle file.

So change this:

if bierdopjeapi.resp:
    open(destsrt, 'wb').write(bierdopjeapi.resp.read())
bierdopjeapi.close()  

Into this:

if bierdopjeapi.resp:
    srtread = bierdopjeapi.resp.read()
    open(destsrt, 'wb').write(srtread)
bierdopjeapi.close()  

Original comment by nielswij...@gmail.com on 12 Jul 2012 at 4:07

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

In this situation it is possible indeed, but I don't really like keeping files 
in the memory. But this will do, still going to increase the time-out to 300 
seconds.

Original comment by romke.va...@gmail.com on 12 Jul 2012 at 7:35

[GoogleCodeExporter]

You could use a temp file for this but given the small size it's not really a 
problem to keep it in memory, just remember to trash the variable after using 
it (also when an exception is raised).

Original comment by nielswij...@gmail.com on 12 Jul 2012 at 7:50

[GoogleCodeExporter]

Original comment by romke.va...@gmail.com on 20 Sep 2012 at 2:43

• Added labels: Milestone-Release0.5.6

[GoogleCodeExporter]

Original comment by romke.va...@gmail.com on 4 Oct 2012 at 1:42

• Changed state: Duplicate