Maximum .ics file size for import.php

[jberanek]

Seems that the import.php does not function for an .ics file > 1.4MB...or approximately 32,000 lines. Is there a way to increase this? My .ICS file is about 7MB and about 120,000 lines.

Thanks, Nick

Reported by: *anonymous

Original Ticket: mrbs/support-requests/562

[jberanek]

How does it fail? It's possible that it's failing because you've hit 'max_execution_time' (normally defaults to 30 seconds) or else 'upload_max_filesize' (normally 2M). You can change these in php.ini.

[I'm thinking of rewriting import.php so that it uses streams, which would avoid both these problems.]

Original comment by: campbell-m

[jberanek]

Also 'post_max_size' as the PHP manual[1] states:

post_max_size integer Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize. If memory limit is enabled by your configure script, memory_limit also affects file uploading. Generally speaking, memory_limit should be larger than post_max_size. When an integer is used, the value is measured in bytes. Shorthand notation, as described in this FAQ, may also be used. If the size of post data is greater than post_max_size, the $_POST and $_FILES superglobals are empty. This can be tracked in various ways, e.g. by passing the $_GET variable to the script processing the data, i.e.

, and then checking if $_GET['processed'] is set.

[1] - http://www.php.net/manual/en/ini.core.php#ini.post-max-size

Original comment by: jberanek

[jberanek]

I guess potentially also 'memory_limit' ...

Original comment by: jberanek

[jberanek]

I have tried the following inside php.ini

upload_max_filesize = 10M post_max_size = 10M max_execution_time = 300 memory_limit = 10M

I have also tried to add the following lines to import.php:

/ Sets some INI Variables / ini_set("post_max_size", "10M"); ini_set("upload_max_filesize", "10M"); ini_set("memory_limit", "10M" ); set_time_limit(300);

As far as errors go, I don't see any at all... its just a blank output screen that just shows the header of the MBRS page, but no data is shown after that.

Nick

Original comment by: *anonymous

[jberanek]

Turns out that my host provider at siteground will does not allow any changes to global settings... the one that I think is causing me the issue is max_execution_time as it is set to 45 seconds and I cannot change it.

So I guess I am going to have to find a way to write my .ics files for import to no more than 1500 entries at a time... or wait until your "streams" version of import.php is built.

Nick

Original comment by: *anonymous

[jberanek]

Yes, in the short term I think your best bet is to split the file up into smaller chunks and import them one at a time.

I'm looking at two ways of improving import.php. One is to import a file that has already been uploaded via FTP to the web server. The other is to allow a compressed file to be uploaded from the client and then get import.php to de-compress it. It's probably going to be a few weeks before I get a chance to do either of these. However the first one may be relatively easy for you to do by hacking import.php and hard-coding a filename.

Campbell

Original comment by: campbell-m

[jberanek]

Anything that still performs the processing in the web server could still hit 'max_execution_time' though. :(

If you have shell access to the web server it would be possible to make import.php run on the CLI I guess.

Original comment by: jberanek

[jberanek]

Good point. Maybe I'll also need to do some performance analysis and optimisation of import.php. The PHP manual says that

"Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real."

Off hand, I can't think that there's much processing outside of stream operations and database queries to be done, so maybe import.php is amenable to some optimisation. Nick if your file is not confidential would you be able to mail it to me off-list so that I can use it as a test file? If it is confidential, don't worry, I can easily construct some test files.

Original comment by: campbell-m

[jberanek]

Anything that still performs the processing in the web server could still hit 'max_execution_time' though. :(

If you have shell access to the web server it would be possible to make import.php run on the CLI I guess.

John

On 4 July 2014 10:08, Campbell Morrison cimorrison@users.sf.net wrote:

Yes, in the short term I think your best bet is to split the file up into smaller chunks and import them one at a time.

I'm looking at two ways of improving import.php. One is to import a file that has already been uploaded via FTP to the web server. The other is to allow a compressed file to be uploaded from the client and then get import.php to de-compress it. It's probably going to be a few weeks before I get a chance to do either of these. However the first one may be relatively easy for you to do by hacking import.php and hard-coding a filename.

Campbell

• [support-requests:#562] http://sourceforge.net/p/mrbs/support-requests/562/ Maximum .ics file size for import.php*

Status: open Group: Created: Thu Jul 03, 2014 05:49 AM UTC by Anonymous Last Updated: Thu Jul 03, 2014 02:47 PM UTC Owner: nobody

Seems that the import.php does not function for an .ics file > 1.4MB...or approximately 32,000 lines. Is there a way to increase this? My .ICS file is about 7MB and about 120,000 lines.

Thanks, Nick

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/mrbs/support-requests/562/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

-- John Beranek - MRBS Developer http://mrbs.sourceforge.net/

To generalise is to be an idiot. -- William Blake

Original comment by: *anonymous

[jberanek]

Its not confidential... now I just gotta figure out what email address I need to send the file to....

:-)

Nick

Original comment by: *anonymous

[jberanek]

[email address snipped]

Original comment by: campbell-m

[jberanek]

I am still waiting for the sourceforge folks to re-enable my account so I can PM you the file.

Nick

Original comment by: *anonymous

[jberanek]

I am still waiting for the sourceforge folks to re-enable my account so I can PM you the file.

Nick

Original comment by: *anonymous

[jberanek]

send to you email from mine.... [email address snipped]

Original comment by: *anonymous

[jberanek]

Thanks for the file. I tried it on my system and I think what's happening is that it is running out of memory and that's why you just get the blank screen. I've rewritten import.php so that it doesn't use so much memory and checked this into the trunk in Rev 2849. I then re-ran it on my system and successfully managed to import 7158 events. The other 1577 events were conflicts and couldn't be imported - and having checked the .ics file I can see that they are genuine conflicts. The script took ~130 seconds on my system using your test file already uploaded to the server using FTP.

To get it to run I had to increase the size of the room_name field in the SQL table from 25 characters (the MRBS default) to 255 as you have some quite long room names. Together with this I had to set

$maxlength['room.room_name']       = 255;

and also set_time_limit() to something larger than 130.

I'll have a look when I get more time at allowing compressed files to be uploaded as this should get round many cases where the file size is greater than the max upload size. I notice your 7MB file compressed to 482KB, so if that's a typical compression ratio for .ics files a 2MB file upload limit would allow files of up to about 25-30MB to be imported.

Also when I get more time I'll have a look at adding some better indications of progress.

Campbell

Original comment by: campbell-m