out of memory (importing >9000 lines)

[GoogleCodeExporter]

What steps will reproduce the problem?
1. new dict. empty -> import
2. select file -> press OK
3. error below

Fatal error: Out of memory (allocated 69468160) (tried to allocate 16 
bytes) in /home/.savanna/priskorn/priskorn.opia.dk/pelikano/lib/
class.domxml.php on line 199

What is the expected output? What do you see instead?
succesful import of 10000 lines from xml-file

What version of the product are you using? On what operating system?
newest

Please provide any additional information below.
ask

Suggestions:
I have 10 dictionaries with 10-120.000 entries (lines of xml) to import 
into GW.
how can I do this with minimum effort?

suggestion 1:
multiline import from URL option
or
import URL-folder option 

in folder I would place xml-files as small as is required
(I would split large xml-files with eg. sed)

suggestion 2:
code us out of handling 4-10+ MB xml-files (there must be a tool for this)

suggestion 3:
convert-script from xml->sql and upload this with phpmyadmin?

or:
something else?

Original issue reported on code.google.com by d.prisk...@gmail.com on 6 Apr 2008 at 10:10

Attachments:

• peco1-testo2-1-10000.xml

[GoogleCodeExporter]

Are you sure about this:
"Recommended number of terms which could be added at once is 500. It varies 
depending
on server performance. " se in the wiki.

Original comment by d.prisk...@gmail.com on 11 Apr 2008 at 3:32

[GoogleCodeExporter]

The process of importing terms is a complicated task. There are a lot of
transformations with text during the process. You can import 10.000 terms on a 
local
machine. But a web-server, especially shared hosting, allows to import 500 
terms or
less at best results.

Since 1.8.6, Glossword can bypass the restriction of maximum running time on 
server.
But the restriction of memory still exists. 69 megabytes for 10.000 terms in not
enough, it requires, perhaps, 128 megabytes or more.

Try to import 500 terms at once. Glossword provides information (filename etc) 
to
help you deal with uploading numerous XML/CSV-files.

Original comment by dmitry.s...@gmail.com on 12 Apr 2008 at 12:58

• Added labels: Type-Question
• Removed labels: Type-Defect

[GoogleCodeExporter]

> import URL-folder option 

Interesting option.

> convert-script from xml->sql and upload this with phpmyadmin?

The transformation from xml to sql is possible only for one dictionary and only 
for a
fresh installation. Search engine calculates keywords dynamically, depending on 
each
term.

Original comment by dmitry.s...@gmail.com on 12 Apr 2008 at 1:10

[GoogleCodeExporter]

Original comment by dmitry.s...@gmail.com on 13 Apr 2008 at 6:25

• Changed state: Solved

[GoogleCodeExporter]

To increase the limit, edit file `/inc/config.inc.php`
and change `$sys['max_lines_csv'] = 10000;` where `10000` is the current value.

Original comment by dmitry.s...@gmail.com on 29 Jan 2010 at 2:22