search

FNNDSC / pman

A process management system written in python
MIT License
23 stars 33 forks source link

Keep image data from being read into memory #96

Closed betaredex closed 5 years ago

betaredex commented 6 years ago
betaredex commented 5 years ago

Just tested. This used about 36KB of memory when dealing with a 64MB file.

danmcp commented 5 years ago

@betaredex LGTM

@rudolphpienaar Any thoughts?

rudolphpienaar commented 5 years ago

I'm happy to merge ... you all good? @danmcp @betaredex ?

betaredex commented 5 years ago

I'll test a bit more tomorrow but then it should be good to go.

On Tue, Nov 13, 2018, 5:38 PM Rudolph Pienaar <notifications@github.com wrote:

I'm happy to merge ... you all good? @danmcp https://github.com/danmcp @betaredex https://github.com/betaredex ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FNNDSC/pman/pull/96#issuecomment-438465637, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUxSNC_Sq6yaFE1-Tf16EivKCFfd6sVks5uu0n0gaJpZM4VW0ns .

betaredex commented 5 years ago

@husky-parul

danmcp commented 5 years ago

LGTM, @rudolphpienaar Please review and merge when ready.

jbernal0019 commented 5 years ago

Hey @danmcp This PR has been merged but it (together with latest pfioh) is making the integration tests fail. Could you please revert this?

danmcp commented 5 years ago

@jbernal0019 @betaredex and @husky-parul were looking into it. What issue are you seeing?

husky-parul commented 5 years ago

@jbernal0019 we are working on this.

jbernal0019 commented 5 years ago

OK, I also see this error inside pfcon:

Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.6/dist-packages/pfcon/pfcon.py", line 441, in dataRequest_process d_dataComs = dataComs() File "/usr/local/lib/python3.6/dist-packages/pfurl/pfurl.py", line 1336, in __call__ d_ret = self.pathOp_do(self.d_msg, action = str_action) File "/usr/local/lib/python3.6/dist-packages/pfurl/pfurl.py", line 1209, in pathOp_do d_ret['compress'] = eval("self.%s_compress(d_msg, **kwargs)" % str_action) File "<string>", line 1, in <module> File "/usr/local/lib/python3.6/dist-packages/pfurl/pfurl.py", line 1111, in pushPath_compress 'str_error': d_ret['str_error'] KeyError: 'str_error'

which makes me think there is also a bug in pfurl at line 1111

jbernal0019 commented 5 years ago

@danmcp I'm not sure what is making the integration tests fail. Bu I see the above traceback inside pfcon logs. Everything was working before the latest pman and pfioh PRs were merged.