Keep Getting Memory Error when sending 5000x160 images.

[geyang]

Potential Memory Leak?

[PhilReinhold]

Sorry about the delay Ge, I've been looking into the problems with larger plots, and have determined that the qt native socket library is the problem. They lack the most basic configurability, and so (on mac at least) it was capped to a 8K receive buffer, which meant lots of short transfers for big datasets. It turns out ZMQ and Qt can play nice through the QSocketNotifier class to create events for zmq sockets, so I'm returning to this. I've got a test now for a huge 5000x5000 image plot. It's not fast to load (several seconds from send to display) but it doesn't crash anything on my end. Check out the test (labelled 'plot huge' in the test runner) and tell me if you're still having problems.

[geyang]

Hi Phil,

I just got back from miami, will look through these and let you know.

On Tue, Mar 18, 2014 at 8:39 PM, Phil Reinhold notifications@github.comwrote:

Sorry about the delay Ge, I've been looking into the problems with larger plots, and have determined that the qt native socket library is the problem. They lack the most basic configurability, and so (on mac at least) it was capped to a 8K receive buffer, which meant lots of short transfers for big datasets. It turns out ZMQ and Qt can play nice through the QSocketNotifier class to create events for zmq sockets, so I'm returning to this. I've got a test now for a huge 5000x5000 image plot. It's not fast to load (several seconds from send to display) but it doesn't crash anything on my end. Check out the test (labelled 'plot huge' in the test runner) and tell me if you're still having problems.

— Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38009459 .

[geyang]

Hi Phil,

I just checked out your code. Everything is working fine except the plot huge. The test crashes everytime on my desktop.

Is there a way to see error messages? right now when I fire the test client up there is no error message in the console.

[PhilReinhold]

No, and I doubt there are any python level errors to report. You might try updating zmq? On Mar 21, 2014 11:30 AM, "episodeyang" notifications@github.com wrote:

Hi Phil,

I just checked out your code. Everything is working fine except the plot huge. The test crashes everytime on my desktop.

Is there a way to see error messages? right now when I fire the test client up there is no error message in the console.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38287457 .

[geyang]

Okay, when I was trying to plot a very simple 3x20 image by doing c.append_z, I got this error message:

Cannot build msvcr library: "msvcr90d.dll" not found

and then after 20 seconds, the image showed up. weird problem.

[geyang]

Can you give me the zmq version number that you are using? Thanks.

[PhilReinhold]

14.1 On Mar 21, 2014 11:48 AM, "episodeyang" notifications@github.com wrote:

Can you give me the zmq version number that you are using? Thanks.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38289680 .

[geyang]

The after a little while the console spits out another message, saying:

Found executable C:\MinGW32-xy\bin\g++.exe

[PhilReinhold]

It appears to be compiling something. I think this might be pyqtgraph. Is this stuff repeatable? On Mar 21, 2014 11:53 AM, "episodeyang" notifications@github.com wrote:

The after a little while the console spits out another message, saying:

Found executable C:\MinGW32-xy\bin\g++.exe

-- Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38290338 .

[geyang]

Right, the 'msvcr90d.dll not found' error is repeatable.

I am still waiting on the MinGW32 to 'finish', but I think it is repeatable.

[geyang]

I am trying this:

Found executable C:\MinGW32-xy\bin\g++.exe

right now my zmq is version 14.0.1.

[PhilReinhold]

You're not using the executable are you? On Mar 21, 2014 11:57 AM, "episodeyang" notifications@github.com wrote:

I am trying this:

Found executable C:\MinGW32-xy\bin\g++.exe

right now my zmq is version 14.0.1.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38290758 .

[geyang]

i am not using the executable. I am running

python -m liveplot.

I just bumped zmq to 14.1.1. trying to see if the problem persist.

the problem persists.

[PhilReinhold]

Maybe this is relevant? On Mar 21, 2014 11:59 AM, "episodeyang" notifications@github.com wrote:

i am not using the executable. I am running

python -m liveplot.

I just bumped zmq to 14.1.1. trying to see if the problem persist.

the problem persists.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38291078 .

[geyang]

It looks like append_z is working now, and the clear command also works with imageView now.

Somehow this version tend to drop connection more easily, aka if I press the test buttons randomly eventually the plotter cease to respond.

[geyang]

It looks like if I try to do anything with a image plot, the plotter ceases responding.

[geyang]

I rebooted the computer, now image is working except the plot huge.

[PhilReinhold]

How does it fail exactly? On Mar 21, 2014 12:19 PM, "episodeyang" notifications@github.com wrote:

I rebooted the computer, now image is working except the plot huge.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38293389 .

[geyang]

when I press that plot huge button, the liveplot_test client window just crashes.

windows says: python has crashed.

Ge

[geyang]

ok in my experiment, it takes around 12 slices of length = 50k array to stall the plotter.

[geyang]

oh sorry, 7k in length, not 50k.

[PhilReinhold]

I'm going to play around today with trying a shared-memory architecture. This is probably as fast as it's going to get wrt large dataset transfer. But it probably requires user-specification of the buffer size in advance.

On Fri, Mar 21, 2014 at 12:46 PM, episodeyang notifications@github.comwrote:

oh sorry, 7k in length, not 50k.

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38296464 .

[PhilReinhold]

Ok, so the new approach with shared memory is, I think, much faster for large plots. However, I think the proximate cause of the 'plot huge' crash has to do with calling numpy.sinc rather than anything to do with the plotter. In either case, check it out, and let me know if it seems better.

There are still some corners to be worked out. Right now all of the array has to fit in the shared memory buffer, but soon that should be an optional performance consideration. Also on mac at least, if things crash, the memory doesn't get released, and has to be freed manually with ipcrm

[PhilReinhold]

Well, yeah, this is just a memory issue. Depending on the computer I'm using I get memory errors even creating 5000x5000 arrays. When I get the chance I'll test it thoroughly with 64-bit python, but until then, stick with arrays under a few hundred MB.

[geyang]

Hey, is your lab moving towards 64bit python? Dave is against for breaking things. But if your lab want to move there I would like to hear about your experiences.

Ge

[PhilReinhold]

No, we haven't, and I haven't had a need for it yet. I think there are a few people using it without issue, but I'm not sure. Python(x,y) should be available in 64-bit next release. I think I'll give it a go when that happens

On Mon, Mar 24, 2014 at 2:20 PM, episodeyang notifications@github.comwrote:

Hey, is your lab moving towards 64bit python? Dave is against for breaking things. But if your lab want to move there I would like to hear about your experiences.

Ge

Reply to this email directly or view it on GitHubhttps://github.com/PhilReinhold/liveplot/issues/3#issuecomment-38481053 .