search

vtsuperdarn / davitpy

DEPRECATED The DaViT Python project
http://vtsuperdarn.github.com/davitpy/
GNU General Public License v3.0
37 stars 59 forks source link

Help in decreasing the processing time of a real time davitpy #145

Closed simonmr closed 8 years ago

simonmr commented 9 years ago

I have been working with the pydarn scripts to allow them to read in and process real time json data.

To help redesign our real time processing java scripts I used davitpy code as a basis and then built/ modified the parts that receive, parse and cycle through the data. It then takes the generated graphs and saves them as a png on a local server. The html code, which is located on that server, periodically refreshes and displays the newest image. This was all done to prevent the websites users from needing to run a local client for every radar on their machine, the current process.

What I have discovered is the way I built everything causes all of the images to be about 1 minute slow. Could anyone help me stream line my code I'd like to get it as close to real time as possible.

The code that I am working from is here: https://github.com/simonmr/superDarnJson

The website that will allow you to access these images is here: http://superdarn.gi.alaska.edu/java/images/index.html

asreimer commented 9 years ago

Very nice! I'm happy to see this project coming along so well. I'm a bit busy right now (start of the term, etc, etc) but I'll have a look at the code in the next week or two. I'll have to learn some json too.

Is all of the code needed to get a real-time server up and running in your repository (doesn't appear to be, just looks like a client)? If not, what are you guys using for the server? If I can set one up here at usask it will be much easier for me to debug/optimize.

asreimer commented 9 years ago

Couldn't help myself, I just had a look at the code.

At first glance, I noticed that the geographic plots are being generated from scratch every time. The most expensive piece of the geo plots is the creation of the utils.mapObj, which is essentially just a basemap object. It take a long time to initialize and draw basemap objects, so instead of continuously creating new ones, perhaps it would be best to create one and then reuse it. One way to do this is to create the basemap object and then plot it into a figure with the radar field of view on it. Then to plot the data, use a separate routine and keep a list of the objects that have been plotted to the figure. When you want to plot the next set of data, you simply delete the objects for the previous data and then plot the new data and update the figure. Clear as mud?

aburrell commented 9 years ago

I don’t know if this works with the utils.mapObj or not, but I’m sure it does. If you don’t want to delete data (which can also take a while, especially if you have collections and patches), you can draw a map to a new figure or subplot using a saved basemap map handle. You can make a basemap object a utils.mapObj object just by adding a datetime to the handle as an attribute.

On 9 Sep 2015, at 18:35, asreimer notifications@github.com wrote:

Couldn't help myself, I just had a look at the code.

At first glance, I noticed that the geographic plots are being generated from scratch every time. The most expensive piece of the geo plots is the creation of the utils.mapObj, which is essentially just a basemap object. It take a long time to initialize and draw basemap objects, so instead of continuously creating new ones, perhaps it would be best to create one and then reuse it. One way to do this is to create the basemap object and then plot it into a figure with the radar field of view on it. Then to plot the data, use a separate routine and keep a list of the objects that have been plotted to the figure. When you want to plot the next set of data, you simply delete the objects for the previous data and then plot the new data and update the figure. Clear as mud?

— Reply to this email directly or view it on GitHub.

simonmr commented 9 years ago

Thanks for those suggestions I have noticed that regenerating the geographic plots are taking the most time and will look into keeping the basemap object and somehow updating the radar field of view.

If you decide to set up the server any ways it is located here: https://github.com/jspaleta/SuperDARN_MSI_ROS/tree/mcm-devel/linux/home/radar/ros.3.6/codebase/superdarn/src.bin/os/rtmichelle.0.1/rtserver_wrap

I've tried to make the read me as clear as I can but it seems like swig is very particular so if you have any questions feel free to ask me.

Thanks Michelle