search

richq / folders2flickr

Upload files to flickr
Other
102 stars 38 forks source link

MemoryError on RaspberryPi #52

Open timmalich opened 8 years ago

timmalich commented 8 years ago

I didn't found an issue for this already:

I guess you are loading the whole file into ram before uploading. On small devices like raspberry you can run easily into an out of memory exception. At least that seams to be the issue on my device. I've attached the trace output:

2016-05-30 19:25:16,713 uploadr.py:532 - uploadImage() Upload failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 515, in uploadImage res = getResponse(url) File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 102, in getResponse data = flickr.unmarshal(minidom.parse(urllib2.urlopen(url))) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open context=self._context) File "/usr/lib/python2.7/urllib2.py", line 1194, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/usr/lib/python2.7/httplib.py", line 1057, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 895, in _send_output msg += message_body

MemoryError

BTW: Thanks for the good and easy tool. :+1: :)

richq commented 8 years ago

Wow, yeah, this is what happens. Who knew? The uploadImage method has an open().read() there to suck up all the file, then encodes using the multipart protocol. But the whole file is read in. Maybe I should do some fancy size limit thing based on machine RAM. Flickr itself I think has a ~1Gb limit for big movie files.

The solution to this seems tricky and would require using another 3rd party lib https://stackoverflow.com/questions/15973204 Not sure what to do here :-S

timmalich commented 8 years ago

I guess at least fancy memory thing would be nice. I didn't got the stackoverflow hint either. Furthermore I'm now facing the issue that the script stops after a while. I guess it's related to the same memory bug. It seams to stop one special movie but I it doesn't gimme a nice error msg. It just says:

2016-06-01 18:43:14,469 uploadr.py:498 - uploadImage() Uploading image /FOO/GOPR0315.MP4 with tags "#2015/2015 FOO.MP4" "2015" "2015 FOOO"

Killed

The error.log doesn't provide a timestamp so I only can guess that the last error message has something to do with it:

Upload failed Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 514, in uploadImage url = buildRequest(api.upload, d, (photo,)) File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 158, in buildRequest contenttype, body = encodeMultipartFormdata(fields, files) File "/usr/lib/python2.7/site-packages/f2flickr/uploadr.py", line 147, in encodeMultipartFormdata body = crlf.join(L)

MemoryError

en5ads commented 7 years ago

Does this help? https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder