CSV Convertor Fails Uploading Japan CSV File

[wendyrogers]

I have bee trying to upload a large CSV file of data fro Japan but the Convertor keeps failing.

[Bjwebb]

How large is the file?

[wendyrogers]

11mb Ben thanks and have just sent it across W

[Bjwebb]

Looks like this is due to the large size of the file. I should be able to convert this specific CSV for you by running the conversion locally.

Server log:

[Thu May 22 09:05:31 2014] [error] ERROR:csviatimodeleditor:Exception on /model/268/change_csv/ [POST]
[Thu May 22 09:05:31 2014] [error] Traceback (most recent call last):
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
[Thu May 22 09:05:31 2014] [error]     response = self.full_dispatch_request()
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Thu May 22 09:05:31 2014] [error]     rv = self.handle_user_exception(e)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
[Thu May 22 09:05:31 2014] [error]     reraise(exc_type, exc_value, tb)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
[Thu May 22 09:05:31 2014] [error]     rv = self.dispatch_request()
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
[Thu May 22 09:05:31 2014] [error]     return self.view_functions[rule.endpoint](**req.view_args)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/csviatimodeleditor/__init__.py", line 336, in model_change_csv
[Thu May 22 09:05:31 2014] [error]     result = chardet.detect(reopen_for_decoding.read())
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/chardet/__init__.py", line 24, in detect
[Thu May 22 09:05:31 2014] [error]     u.feed(aBuf)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/chardet/universaldetector.py", line 115, in feed
[Thu May 22 09:05:31 2014] [error]     if prober.feed(aBuf) == constants.eFoundIt:
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/chardet/charsetgroupprober.py", line 59, in feed
[Thu May 22 09:05:31 2014] [error]     st = prober.feed(aBuf)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/chardet/hebrewprober.py", line 217, in feed
[Thu May 22 09:05:31 2014] [error]     aBuf = self.filter_high_bit_only(aBuf)
[Thu May 22 09:05:31 2014] [error]   File "/home/pythonuser/CSV-IATI-Converter.modeleditor/pyenv/lib/python2.7/site-packages/chardet/charsetprober.py", line 51, in filter_high_bit_only
[Thu May 22 09:05:31 2014] [error]     aBuf = re.sub(r'([\\x00-\\x7F])+', ' ', aBuf)
[Thu May 22 09:05:31 2014] [error]   File "/usr/lib/python2.7/re.py", line 151, in sub
[Thu May 22 09:05:31 2014] [error]     return _compile(pattern, flags).sub(repl, string, count)
[Thu May 22 09:05:31 2014] [error] MemoryError

[Bjwebb]

Current approach to large files is to run the conversion locally on a machine with more resources. Therefore I am unassigning myself from this issue.