MemoryError

[emontgomery-usgs]

@esturdivant-usgs Looks like things were clicking along fine, then I got an OverflowError. Thought it was maybe because I didn't re-pip install pysb, so I went back and did that after activating sb_py3, and before running sb_automation. That didn't solve it. Maybe it is that size threshold is the problem- it has to be less than 2000.

I changed the max_MBsize to 2000 (from 2700), and it ran further without the overflow error, and skipped files. But it eventually got to a memory error: The original directory I'm uploading from is local to my computer, but it got aways before getting the error.

Here's the traceback: UPLOADING: files matching '2017029FA_U031_Sandwich*' TO DO File ['2017029FA_U031_Sandwich_f1.zip', '2017029FA_U031_Sandwich_f4.zi p', '2017029FA_U031_Sandwich_f7.zip'] is too big to upload here. Please manually upload afterward. Traceback (most recent call last): File "sb_automation.py", line 185, in data_item, bigfiles1 = upload_files_matching_xml(sb, data_item, xml_file, ma x_MBsize=max_MBsize, replace=True, verbose=verbose) File "C:\Users\emontgomery\python_progs\science-base-automation\autoSB.py", li ne 594, in upload_files_matching_xml item = sb.upload_files_and_upsert_item(item, up_files) # upsert should "crea te or update a SB item" File "c:\users\emontgomery\python_progs\science-base-automation\src\pysb\pysb\ SbSession.py", line 362, in upload_files_and_upsert_item ret = self._session.post(url, params=params, files=files, data=data) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\sessions.py", line 555, in post return self.request('POST', url, data=data, json=json, **kwargs) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\sessions.py", line 494, in request prep = self.prepare_request(req) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\sessions.py", line 437, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\models.py", line 308, in prepare self.prepare_body(data, files, json) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\models.py", line 496, in prepare_body (body, content_type) = self._encode_files(files, data) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\requests\models.py", line 165, in _encode_files body, content_type = encode_multipart_formdata(new_fields) File "C:\Users\emontgomery\AppData\Local\Continuum\Miniconda3\envs\sb_py3\lib\ site-packages\urllib3\filepost.py", line 86, in encode_multipart_formdata body.write(data) MemoryError

Ideas?

[esturdivant-usgs]

I haven't seen this before...

It might end up being a question for the pysb folks. I'll see if they have anything about it on their site.

I see that it's happening on this call: sb.upload_files_and_upsert_item()

[esturdivant-usgs]

@emontgomery-usgs Had you successfully completed the routine with these files and the same file size threshold before this happened? I wonder whether this a problem caused by lingering memory use or if we would have hit this issue anyway.

I haven't turned up anything so my next attempt would be to clear out everything - wipe the pages, start a new SB session and then try again, perhaps with a lower size threshold.

[emontgomery-usgs]

I think I'm going to have to treat all my files at large, and not try to upload them this way.