Impossible to download large files

[GoogleCodeExporter]

Hi!
Mollify is realy fantastic, but everytime I try to download large files (>10 
MB) the download ends immediately with the result of a 0 Byte file.

By downloading it "zipped" it is possible to download 50 MB and more. 
What could be the cause of it?

Thank's for answering!

Original issue reported on code.google.com by jens.kae...@web.de on 31 Aug 2010 at 12:14

[GoogleCodeExporter]

I'd need some log from server, could you turn on server debugging and get the 
logs after failed download (see instructions 
http://code.google.com/p/mollify/wiki/Troubleshooting)

Original comment by samuli.j...@gmail.com on 31 Aug 2010 at 12:31

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Ok,
I'll turn it on very soon and attach that file.

Original comment by jens.kae...@web.de on 31 Aug 2010 at 12:41

[GoogleCodeExporter]

my logs are attached.

Original comment by jens.kae...@web.de on 31 Aug 2010 at 1:44

Attachments:

• log.txt

[GoogleCodeExporter]

Unfortunately that log doesn't reveal anything, because you copied only the 
lines visible on the screen (there's plenty more if you scroll it down).

But can you access the server log? It could be better since download problems 
might not be visible to clients.

Original comment by samuli.j...@gmail.com on 31 Aug 2010 at 5:18

[GoogleCodeExporter]

Sorry for this,
here comes my next try. I just wanted to access the server log, but that's not 
as easy as I thought ...

Original comment by jens.kae...@web.de on 31 Aug 2010 at 9:06

Attachments:

• log.txt

[GoogleCodeExporter]

Hi, it seems I have the same issue here.  It seems also that the limit of the 
files is the same as the limit of the upload files (64Mo on my OVH server).

When I download a file less than 64Mo, I have no problem at all but when I try 
to download more (94Mo on my test files uploaded via ftp), I get an empty file 
(0 bytes of file size).

The log does not produce any error.  (But since nothing is printed when I click 
the download button, I guess no useful information will be printed here...)  
Here is the result :

--------------------------------------------------------------------------------
-
2010-09-03 23:02:47,410 [DEBUG] Get file details: Mjovc2tvbHZhbi5yYXI=
2010-09-03 23:02:47,411 [DEBUG] REQUEST (GET): 
http://nosmile.blogsite.org/files/backend/r.php/filesystem/Mjovc2tvbHZhbi5yYXI%2
C/details/
2010-09-03 23:02:47,649 [DEBUG] Request response: 200 
{"result":{"id":"Mjovc2tvbHZhbi5yYXI=","last_changed":"20100903225215","last_mod
ified":"20100903225215","last_accessed":"20100903225200","description":null,"per
mission":"RW"},"trace":["MySQL DB: 
mdejdrnosmil@mysql5-12.pro:mdejdrnosmil(files_molify_)","SERVER: 
{UNIQUE_ID:TIFidwoAQEAAAE48lgwAAABd, GEOIP_COUNTRY_CODE:BE, 
GEOIP_COUNTRY_NAME:Belgium, GEOIP_REGION:05, GEOIP_CITY:Opglabbeek, 
GEOIP_DMA_CODE:0, GEOIP_AREA_CODE:0, GEOIP_LATITUDE:51.049999, 
GEOIP_LONGITUDE:5.583300, 
SCRIPT_URL:\/files\/backend\/r.php\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/,

SCRIPT_URI:http:\/\/nosmile.blogsite.org\/files\/backend\/r.php\/filesystem\/Mjo
vc2tvbHZhbi5yYXI,\/details\/, PHP_VER:5, CONTENT_TYPE:text\/plain; 
charset=utf-8, 
HTTP_REFERER:http:\/\/nosmile.blogsite.org\/files\/client_debug\/E51FB2D3185C235
A5D10CF5CB6E19528.cache.html, 
HTTP_COOKIE:MOLLIFY-SESSION=379c0e709729100fe0b201470653bf4c, 
SCRIPT_FILENAME:\/homez.50\/mdejdr\/nos\/www\/files\/backend\/r.php, 
SERVER_PROTOCOL:HTTP\/1.1, REQUEST_METHOD:GET, QUERY_STRING:, 
REQUEST_URI:\/files\/backend\/r.php\/filesystem\/Mjovc2tvbHZhbi5yYXI%2C\/details
\/, SCRIPT_NAME:\/files\/backend\/r.php, 
PATH_INFO:\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/, 
PATH_TRANSLATED:\/homez.50\/mdejdr\/nos\/www\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/
details\/, UID:2753, PHP_SELF:\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/, 
REQUEST_TIME:1283547767, argv:{0:r.php}, argc:1}","SETTINGS: 
{host_public_address:http:\/\/nosmile.blogsite.org, enable_public_links:, 
new_folder_permission_mask:511, debug:1}","CONFIGURATION PROVIDER 
(MySQLConfigurationProvider): supported features={0:change_password, 
1:description_update, 2:administration, 3:permission_update, 4:user_groups} 
auth=1","FEATURES: {limited_http_methods:, file_upload:1, folder_actions:1, 
file_upload_progress:, zip_download:, change_password:1, description_update:1, 
permission_update:1, administration:1, user_groups:1, public_links:, 
mail_notification:, file_view:1, file_preview:1}","FILESYSTEM: 
allowed_file_upload_types={}","SESSION: {user_id:4, groups:{}, username:folk, 
default_permission:RW}","AUTH: is_authentication_required=1, 
is_authenticated=1","REQUEST: method=get, path={0:filesystem, 
1:Mjovc2tvbHZhbi5yYXI%2C, 2:details}, ip=87.67.1.64, params={}, data=","SERVICE 
(FilesystemServices): is_auth_required=1","DB: SELECT id, name, path FROM 
files_molify_folder where id='2'","{}","DB: SELECT permission FROM ((SELECT 
permission, user_id, 1 AS 'category', (IF(user_id = '4', 1, IF(user_id = '0', 
3, 2))) AS 'subcategory' FROM `files_molify_item_permission` WHERE item_id = 
'2:\/skolvan.rar' AND (user_id in (0,4))) UNION ALL (SELECT permission, 
user_id, 2 AS 'category', (((3 - CHAR_LENGTH(item_id)) * 10) + IF(user_id = 
'4', 0, IF(user_id = '0', 2, 1))) AS 'subcategory' FROM 
`files_molify_item_permission` WHERE (item_id REGEXP '^2:\/$') AND (user_id in 
(0,4))) ) AS u ORDER BY u.category ASC, u.subcategory ASC, u.permission 
DESC","DB: SELECT description FROM files_molify_item_description WHERE 
item_id='2:\/skolvan.rar'","{}","DB: SELECT permission FROM ((SELECT 
permission, user_id, 1 AS 'category', (IF(user_id = '4', 1, IF(user_id = '0', 
3, 2))) AS 'subcategory' FROM `files_molify_item_permission` WHERE item_id = 
'2:\/skolvan.rar' AND (user_id in (0,4))) UNION ALL (SELECT permission, 
user_id, 2 AS 'category', (((3 - CHAR_LENGTH(item_id)) * 10) + IF(user_id = 
'4', 0, IF(user_id = '0', 2, 1))) AS 'subcategory' FROM 
`files_molify_item_permission` WHERE (item_id REGEXP '^2:\/$') AND (user_id in 
(0,4))) ) AS u ORDER BY u.category ASC, u.subcategory ASC, u.permission 
DESC","RESPONSE success {id:Mjovc2tvbHZhbi5yYXI=, last_changed:20100903225215, 
last_modified:20100903225215, last_accessed:20100903225200, description:, 
permission:RW}"]}
2010-09-03 23:02:53,834 [DEBUG] Get file details: Mjovc2tvbHZhbi5yYXI=
2010-09-03 23:02:53,834 [DEBUG] REQUEST (GET): 
http://nosmile.blogsite.org/files/backend/r.php/filesystem/Mjovc2tvbHZhbi5yYXI%2
C/details/
2010-09-03 23:02:54,068 [DEBUG] Request response: 200 
{"result":{"id":"Mjovc2tvbHZhbi5yYXI=","last_changed":"20100903225215","last_mod
ified":"20100903225215","last_accessed":"20100903225200","description":null,"per
mission":"RW"},"trace":["MySQL DB: 
mdejdrnosmil@mysql5-12.pro:mdejdrnosmil(files_molify_)","SERVER: 
{UNIQUE_ID:TIFifgoAQEAAADyKWVYAAAAo, GEOIP_COUNTRY_CODE:BE, 
GEOIP_COUNTRY_NAME:Belgium, GEOIP_REGION:05, GEOIP_CITY:Opglabbeek, 
GEOIP_DMA_CODE:0, GEOIP_AREA_CODE:0, GEOIP_LATITUDE:51.049999, 
GEOIP_LONGITUDE:5.583300, 
SCRIPT_URL:\/files\/backend\/r.php\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/,

SCRIPT_URI:http:\/\/nosmile.blogsite.org\/files\/backend\/r.php\/filesystem\/Mjo
vc2tvbHZhbi5yYXI,\/details\/, PHP_VER:5, CONTENT_TYPE:text\/plain; 
charset=utf-8, 
HTTP_REFERER:http:\/\/nosmile.blogsite.org\/files\/client_debug\/E51FB2D3185C235
A5D10CF5CB6E19528.cache.html, 
HTTP_COOKIE:MOLLIFY-SESSION=379c0e709729100fe0b201470653bf4c, 
SCRIPT_FILENAME:\/homez.50\/mdejdr\/nos\/www\/files\/backend\/r.php, 
SERVER_PROTOCOL:HTTP\/1.1, REQUEST_METHOD:GET, QUERY_STRING:, 
REQUEST_URI:\/files\/backend\/r.php\/filesystem\/Mjovc2tvbHZhbi5yYXI%2C\/details
\/, SCRIPT_NAME:\/files\/backend\/r.php, 
PATH_INFO:\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/, 
PATH_TRANSLATED:\/homez.50\/mdejdr\/nos\/www\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/
details\/, UID:2753, PHP_SELF:\/filesystem\/Mjovc2tvbHZhbi5yYXI,\/details\/, 
REQUEST_TIME:1283547774, argv:{0:r.php}, argc:1}","SETTINGS: 
{host_public_address:http:\/\/nosmile.blogsite.org, enable_public_links:, 
new_folder_permission_mask:511, debug:1}","CONFIGURATION PROVIDER 
(MySQLConfigurationProvider): supported features={0:change_password, 
1:description_update, 2:administration, 3:permission_update, 4:user_groups} 
auth=1","FEATURES: {limited_http_methods:, file_upload:1, folder_actions:1, 
file_upload_progress:, zip_download:, change_password:1, description_update:1, 
permission_update:1, administration:1, user_groups:1, public_links:, 
mail_notification:, file_view:1, file_preview:1}","FILESYSTEM: 
allowed_file_upload_types={}","SESSION: {user_id:4, groups:{}, username:folk, 
default_permission:RW}","AUTH: is_authentication_required=1, 
is_authenticated=1","REQUEST: method=get, path={0:filesystem, 
1:Mjovc2tvbHZhbi5yYXI%2C, 2:details}, ip=87.67.1.64, params={}, data=","SERVICE 
(FilesystemServices): is_auth_required=1","DB: SELECT id, name, path FROM 
files_molify_folder where id='2'","{}","DB: SELECT permission FROM ((SELECT 
permission, user_id, 1 AS 'category', (IF(user_id = '4', 1, IF(user_id = '0', 
3, 2))) AS 'subcategory' FROM `files_molify_item_permission` WHERE item_id = 
'2:\/skolvan.rar' AND (user_id in (0,4))) UNION ALL (SELECT permission, 
user_id, 2 AS 'category', (((3 - CHAR_LENGTH(item_id)) * 10) + IF(user_id = 
'4', 0, IF(user_id = '0', 2, 1))) AS 'subcategory' FROM 
`files_molify_item_permission` WHERE (item_id REGEXP '^2:\/$') AND (user_id in 
(0,4))) ) AS u ORDER BY u.category ASC, u.subcategory ASC, u.permission 
DESC","DB: SELECT description FROM files_molify_item_description WHERE 
item_id='2:\/skolvan.rar'","{}","DB: SELECT permission FROM ((SELECT 
permission, user_id, 1 AS 'category', (IF(user_id = '4', 1, IF(user_id = '0', 
3, 2))) AS 'subcategory' FROM `files_molify_item_permission` WHERE item_id = 
'2:\/skolvan.rar' AND (user_id in (0,4))) UNION ALL (SELECT permission, 
user_id, 2 AS 'category', (((3 - CHAR_LENGTH(item_id)) * 10) + IF(user_id = 
'4', 0, IF(user_id = '0', 2, 1))) AS 'subcategory' FROM 
`files_molify_item_permission` WHERE (item_id REGEXP '^2:\/$') AND (user_id in 
(0,4))) ) AS u ORDER BY u.category ASC, u.subcategory ASC, u.permission 
DESC","RESPONSE success {id:Mjovc2tvbHZhbi5yYXI=, last_changed:20100903225215, 
last_modified:20100903225215, last_accessed:20100903225200, description:, 
permission:RW}"]}
--------------------------------------------------------------------------------
-

Original comment by djnosm...@gmail.com on 3 Sep 2010 at 10:13

[GoogleCodeExporter]

Just a post to give you some more information :

I was wrong when linking the size to the upload_size since I have a file of 59M 
which also produce a 0bytes fil when downloaded.

I beleived it was a server configuration issue (transparent gziped output that 
get out of memory and ) but after deactivating it via htaccess, nothing changed 
so, this is not this case.  (Still it can be another configuration which I did 
not think/find about...)

I also browsed a little the source code and did not find anything strange.  I'm 
not a php expert but I used a similar way of handling download on another OVH 
server which ran just fine (but it's not online anymore so I can't test on this 
server, sorry).

So, I'm stuck and have no more idea for now on finding the error...

Original comment by djnosm...@gmail.com on 4 Sep 2010 at 11:29

[GoogleCodeExporter]

djnosmile, the log you posted is from client, and unfortunately download errors 
(or results of any kind) don't end up in client at all. So I'd need the server 
log for this.

Downloading files should not have anything to do with memory limits, as the 
files are read in small chunks. There can of course be some PHP setting that 
controls this, but I'm not aware of any, but hopefully server log would reveal 
this.

Original comment by samuli.j...@gmail.com on 4 Sep 2010 at 1:30

[GoogleCodeExporter]

Ok, silly question then, how do I access them?

Something interesting maybe, I've looked at the header and it contains 
"Content-Length: 0" for these files...

Original comment by djnosm...@gmail.com on 4 Sep 2010 at 2:01

[GoogleCodeExporter]

It depends, some service providers don't allow this, but you might have, for 
example, ssh or ftp access to your account where these logs are (PHP error log 
is what I need). Or perhaps you have to ask admin for these, it's really 
impossible for me to say.

Content length header itself doesn't help right now, if the downloaded file is 
empty it is quite obvious content length is also zero.

Original comment by samuli.j...@gmail.com on 4 Sep 2010 at 2:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

ok, I can't have access to the php error log with OVH (only the apache log)...

I tried to add these on top of r.php :

ini_set('error_log', 'mylog.log');
error_reporting(E_ALL);
ini_set("display_errors", "stdout");

I've join a copy of the mylog.log file produced by 2 downloads.  Hope that 
could help.

Original comment by djnosm...@gmail.com on 4 Sep 2010 at 3:56

Attachments:

• mylog.log

[GoogleCodeExporter]

Unfortunately the logs don't reveal anything.

On the contrary, there is a event fired _after_ the download, which means that 
PHP completed the download without any errors (as any error would have caused a 
break, and no event would have been fired).

So judging from this, I'd suspect there is something in reading the file, you 
are sure the file is valid (ie you can download it properly, for example, with 
ftp)? And permissions are correct?

What I I could do, is to add some more debug logging into the download process, 
maybe we can get more info from there.

Original comment by samuli.j...@gmail.com on 6 Sep 2010 at 6:19

[GoogleCodeExporter]

Is it possible to download "unzipped" files in the way "zipped" files are 
downloaded?
I mean, if I download a file, the download-window shows me something like "2 of 
3 MB are downloaded". Using the zip-download I receive the message "remaining 
time unknown, 2 MB are downloaded" which allows me to download large files of 
an unknown size.
Could that be a solution to download large unzipped files?

Original comment by jens.kae...@web.de on 6 Sep 2010 at 10:40

[GoogleCodeExporter]

jens, unfortunately it doesn't make any difference.

The browser shows the progress based on the http header, where the content 
length is defined. If this value is not given, browser cannot tell how much 
there is still to come, but it does not affect the outcome (ie. browser takes 
all the data there is).

Zip feature does not give the size info since it cannot know it (except in the 
latest version where server zip packaging has been rewritten, it can report the 
size)

Original comment by samuli.j...@gmail.com on 6 Sep 2010 at 11:17

[GoogleCodeExporter]

Is there an easy way to configure the output file not to tell the content 
length for the download?
Maybe that's a way to tell my browser to reveive all the data as you wrote.

Original comment by jens.kae...@web.de on 6 Sep 2010 at 11:30

[GoogleCodeExporter]

Well, there is no option for this, but you could modify your copy of 
"include/OutputHandler.class.php" and modify line 108

       if ($size) header("Content-Length: ".$size);

into

       if (FALSE) header("Content-Length: ".$size);

Original comment by samuli.j...@gmail.com on 6 Sep 2010 at 11:37

[GoogleCodeExporter]

I correct myself, the line to be modified is 88 (although exact same content). 
Also, you could remove the "set_time_limit" in line 100.

Original comment by samuli.j...@gmail.com on 6 Sep 2010 at 11:40

[GoogleCodeExporter]

Thank's for this solution!

Now I'm able to download files up to 20 MB (instead of 8 MB).
Great!

Original comment by jens.kae...@web.de on 6 Sep 2010 at 12:02

[GoogleCodeExporter]

Could you try with attached version on OutputHandler, it will produce some more 
log for downloads.

Original comment by samuli.j...@gmail.com on 6 Sep 2010 at 7:19

Attachments:

• OutputHandler.class.php

[GoogleCodeExporter]

Hi,

Well, the file is correct since I can download it by typing its adress on my 
browser (btw, this is a security issue which is not documented, you should add 
some htaccess files...).

Anyway, here is the log file.  Unfortunately, I guess it's not really usefull 
either...

Original comment by djnosm...@gmail.com on 6 Sep 2010 at 10:23

Attachments:

• mylog.log

[GoogleCodeExporter]

Good news guys, here is the solution to this issue...

As I said above, I beleived that the data was transparently compressed by my 
provider with no ability to disable this function.  After some research, I 
finally found this php manual page : http://php.net/manual/en/function.flush.php

These two lines are exactly my case :
--------------------
Several servers, especially on Win32, will still buffer the output from your 
script until it terminates before transmitting the results to the browser.

Server modules for Apache like mod_gzip may do buffering of their own that will 
cause flush() to not result in data being sent immediately to the client.
-------------------

Linking this information to my previous idea, I guess that the function used by 
my provider to compress output just crash silently for receiving too much data 
without flushing it's buffer.

So, to solve the problem, you juste need to add a call to ob_flush() before 
flush() :

while (!feof($stream)) {
    set_time_limit(0);
    echo fread($stream, 1024);
    ob_flush();
    flush();
}

It was a try on a guess but here it is.  Now the script is perfectly working.

Original comment by djnosm...@gmail.com on 6 Sep 2010 at 11:00

[GoogleCodeExporter]

Great find, thanks a lot! I've seen this solution when trying to solve other 
issues, but didn't think it was about this.

I think I'll add a setting that uses ob_flush for those who have the same 
problem and cannot turn off server compression.

Original comment by samuli.j...@gmail.com on 7 Sep 2010 at 7:02

• Changed state: Accepted

[GoogleCodeExporter]

I just tried it - Absolutely perfect! It was no problem to download a file of 
140 MB.
Thank's for that solution.

Original comment by jens.kae...@web.de on 7 Sep 2010 at 7:28

[GoogleCodeExporter]

Made a new setting that does the same thing, will be in 1.7

Original comment by samuli.j...@gmail.com on 8 Sep 2010 at 4:51

• Changed state: Started

[GoogleCodeExporter]

Issue 86 has been merged into this issue.

Original comment by samuli.j...@gmail.com on 8 Sep 2010 at 4:52

[GoogleCodeExporter]

Available in 1.7

Original comment by samuli.j...@gmail.com on 7 Nov 2010 at 12:15

• Changed state: Fixed

[GoogleCodeExporter]

Well, I've done the update from 1.6.0 and it's not working...

Original comment by djnosm...@gmail.com on 4 Dec 2010 at 2:16

[GoogleCodeExporter]

Well, I've done the update from 1.6.0 and it's not working...  I had to comment 
the first line with ob_start() to get it work...

Don't know why.  I tried playing a little with the other fonction but as I 
don't have a lot of time now, I did not get on further research on what exactly 
does each of them so...

Original comment by djnosm...@gmail.com on 4 Dec 2010 at 2:47

[GoogleCodeExporter]

Still not working with mollify version 1.8.0.1 , even with the 
"support_output_buffer" to False)  Did the same changes and it worked...

Original comment by djnosm...@gmail.com on 17 May 2011 at 7:18

[GoogleCodeExporter]

I was under impression that you actually _had_ to call ob_flush to make it work 
(at least in comment #23 you say so), now you say you have to comment something 
out?

Anyway, if you actually do need to call ob_flush, then you have to have the 
setting "support_output_buffer" set to TRUE, not FALSE.

Original comment by samuli.j...@gmail.com on 17 May 2011 at 8:11

[GoogleCodeExporter]

Sorry, my mistake here.  I indeed tried with "support_output_buffer" to TRUE 
without success.

About comment #23, I said that I had to call ob_flush, not ob_start.  I agree 
that it's strange but if I add a call to ob_start, it doesn't work anymore.  
(Even if the doc specified the ob_start call...)  Don't know why...

Original comment by djnosm...@gmail.com on 17 May 2011 at 8:28

[GoogleCodeExporter]

So you just removed the ob_start from line 132 and it works (with 
"support_output_buffer" set to TRUE)?

Original comment by samuli.j...@gmail.com on 18 May 2011 at 6:07

[GoogleCodeExporter]

Exactly

Original comment by djnosm...@gmail.com on 18 May 2011 at 8:49

[GoogleCodeExporter]

Hi, I have just installed the 1.8.0.2 and everything worked properly.

Anyway, there is still a little problem.  It does not seem to be as easy.  I 
tried to activate the zip download option and, it said that ob_flush had 
nothing to flush...

Conclusion, when using zip download, the automatic buffer is deactivated (for 
some unknow reason) and it fails.  Si I added a check on line 138 of 
OutputHandler which is now :

if ($this->supportOutputBuffer and ob_get_length()!=FALSE ) @ob_flush();

With this code, everything work well...  (I also tried to re-introduce the 
ob_start with the opposite test without  success)

Original comment by djnosm...@gmail.com on 25 May 2011 at 4:33

[GoogleCodeExporter]

Thanks for the addition, I made the change.

Original comment by samuli.j...@gmail.com on 25 May 2011 at 5:35