attardi
commented
9 years ago On 4/10/2015 09:00, agoyaliitk wrote:
Can you explain why this error occurs?
— Reply to this email directly or view it on GitHub https://github.com/attardi/wikiextractor/issues/2.
Because thre are template definitions that invoke themselves recursively. In the case in question the template invocation
{{Multiple sclerosis}}
expands to a body
{{Navbox | name = Demyelinating diseases of CNS | title = [[Multiple sclerosis]] and other [[demyelinating disease]]s of [[Centr al nervous system|CNS]]([[ICD-10 Chapter VI: Diseases of the nervous system#%28G3 5–G37%29 Demyelinating diseases of the central nervous system|G35–G37]], [[List of ICD-9 codes 320–359: diseases of the nervous system#Other disorders of the cent ral nervous system %28340–349%29|340–341]]) |bodyclass = hlist |{{Multiple sclerosis|state=expanded}}) | titlestyle = background: Silver; ...
and the template expansion procedure would keep expanding forever. Templates are to be considered as macros, in which recursion is not allowed.
I added a check on the depth of recursive expansion, similar to the one used in the official code from MediaWiki, to handle these malformed templates.
-- Beppe
agoyaliitk
sanja7s
cifkao
Can you explain why this error occurs? I used the updated version of the script uploaded yesterday. Now it's giving this error.
Traceback (most recent call last): File "./WikiExtractor.py", line 1797, in
main()
File "./WikiExtractor.py", line 1793, in main
process_data(input_file, args.templates, output_splitter)
File "./WikiExtractor.py", line 1621, in process_data
extract(id, title, page, output)
File "./WikiExtractor.py", line 132, in extract
text = clean(text)
File "./WikiExtractor.py", line 1256, in clean
text = expandTemplates(text)
File "./WikiExtractor.py", line 307, in expandTemplates
res += expandTemplate(text[s+2:e-2], depth+l)
File "./WikiExtractor.py", line 808, in expandTemplate
ret = expandTemplates(template, depth + 1)
File "./WikiExtractor.py", line 307, in expandTemplates
res += expandTemplate(text[s+2:e-2], depth+l)
File "./WikiExtractor.py", line 769, in expandTemplate
params = templateParams(parts[1:], depth)
File "./WikiExtractor.py", line 396, in templateParams
parameters = [expandTemplates(p, frame) for p in parameters]
File "./WikiExtractor.py", line 307, in expandTemplates
res += expandTemplate(text[s+2:e-2], depth+l)
File "./WikiExtractor.py", line 769, in expandTemplate
params = templateParams(parts[1:], depth)
File "./WikiExtractor.py", line 396, in templateParams
parameters = [expandTemplates(p, frame) for p in parameters]
File "./WikiExtractor.py", line 307, in expandTemplates
res += expandTemplate(text[s+2:e-2], depth+l)
File "./WikiExtractor.py", line 808, in expandTemplate
ret = expandTemplates(template, depth + 1)
File "./WikiExtractor.py", line 313, in expandTemplates
res += text[cur:]
MemoryError