Graph Widget? Plot Widget?

[ccoupe]

Does anyone need a challenge ( @passenger94 ?). Long ago I did some some time series analysis foolery in Java and wrote a "widget" that would graph multiple time series data, auto scale the content to the window size, zoom in/out and move the zoomed graph left or right.

Hard core data analysis folks would just call out to gnuplot but gnuplot is huge and not really something we want to include in Shoes. Still, some cross platform visualization capability might be useful. I imagine, without thinking much on my part, that It would be Shoes widget (canvas) and we'd use mostly Cairo to draw and C/Ruby to get values out of Ruby arrays. Could it be done with pure Shoes/Ruby? Probably. Performance? I don't know - the test data sets I have are 5K of observations and they drew surprisingly quick on a Windows 98 laptop in Java 1.02,

[ccoupe]

Wow! That old Java code still works and just as painful to setup. Screen shot of graphing ^SPX (S&P 500) and ^IRX.R (3 month US Treasury Bill rate). The question is how to do it in Shoes and not what the plot tells you.

[ccoupe]

csv test data

[passenger94]

foolery in Java

sounds like a pleonasm :-D

haha, thanks for challenge :-), will look at it, if only for curiosity, i was interested in this when looking at the profiler thing and ways to get visual feedback lack of time lately, some opportunities maybe next month

[dredknight]

this looks really awesome! I have to get together and finish that popup thing and I can hop over and check this too :)

[ccoupe]

Still thinking out loud about this. It should be easy enough to convert that java code into a shoes.widget in pure ruby. Looking at the java, the DataSeries class is actually the bigger problem because I know it's sordid secrets - it has no way to represent 'no data' or 'bad data' for 'this observation' and it assumes all DataSeries have the same date/time starting point and ending point and number of observations. In order to be part of Shoes, the drawing code has to be in lib/shoes/ and while there are reasons to have DataSeries be a Gem, there are good (Loose Shoes symlink) reasons to put it into lib/shoes/DataSeries/

I've uploaded some Stock Market data files (csv format) to here - the contents are obvious. There is a long ago, mostly useless reason they start with '^' but it's what I have.

I've created branch 'graph' which is where this experiment should live.

[ccoupe]

Pure Shoes/ruby - test of concept. It works. Of course it's missing all kinds of things.

Good enough to try sub classing Shoes.widget.

[passenger94]

i say YES !!! :high_heel: :high_heel: :high_heel: :high_brightness:

[ccoupe]

So it's now a Shoes::Widget and it's time to add the axes and labels. OOPS! Shoes doesn't have methods for drawing text at x,y. Time to move c/obj-c? Maybe Shoes should have an Art Shape (like oval, star, etc ) named Art_String ??? so art_string {x:, y:, font:, fontsize:,color:, length: , text:}. I don't know how hard that is to do but it has some appeal and might be useful for other folks for other applications. Probably easier to do than creating a new c/obj-c graph widget so I like that. It might be useful text on top a image (with transforms already available like rotate and skew). Opinions?

[ccoupe]

Considering that I'm just converting my java code, it would be conceptually easier to create a widget, similar to svg which I understand a lot better than Shape manipulation code (#266). It would have better performance for large data sets but not nearly as flexible for future enhancements. It's possible to do both a native widget and a Shoes::Widget version.

I think I'll call the native widget 'plot' unless someone has a better name. API is pretty simplistic so far: pl = plot width, height. {options} and pl.add {options} | dataseries and pl.redraw in case someone is appending real time data. The option names: and defaults to be determined later but :data => [array of floatables]`, :name => "string", :minv => float, :maxv => float are the minimum.

[ccoupe]

API alert - some options belong to plot or chart (title and caption, width and height) and some belong to the data (like name, display_name, values and many more); The plot/chart behaves differently depending on how many arrays it's supposed to display. So, it's a two step creation process in Shoes plot width, height, {options) and plot.add {many_hash_args}

[ccoupe]

[Updated 2016-08-19]

[ccoupe]

Getting better. So many things to do under the covers.

The potential api keeps expanding with all the options the users might like ( turning of the grid lines for example)

[ccoupe]

The plot widget is almost good_enough for people to try it and find out what doesn't work for them. It does work on all platforms. I might think about including it in 3.3.2. Even if its a little half baked. Documenting it is going to be a challenge.

and the script to create that:

# good-graph.rb
Shoes.app width: 620, height: 610 do
  @values1 = [24, 22, 10, 13, 20, 8, 22]
  @x_axis1 = ['a','b','c','d','e','f', 'g']
  @values2 = [200, 150, 75, 125, 75, 50, 125]
  stack do
    para "Plot Widget Demo"
    flow do 
      button "quit" do Shoes.quit end
    end
    widget_width = 600
    widget_height = 400
    @grf = plot widget_width, widget_height, title:"My Graph", caption: 
      "Look at that! Booyah!!" , font: "Helvetica", auto_grid: true
    @grf.add num_obs: @values1.size, values: @values1, xobs: @x_axis1,
       name: "foobar", minv: 6, maxv: 26 , long_name: "foobar Yy"
    button "add #2" do
      @grf.add num_obs: @values2.size, values: @values2,
        name: "bartab", minv: @values2.min, maxv: @values2.max
    end
  end
end

[ccoupe]

updated

[passenger94]

synched and expert-graph.rb works fine on my system :+1: also found graph.rb and getest.rb, didn't yet got what it does exactly. Is there a cvs test file somewhere ?

[ccoupe]

Just committed the latest. Two test programs graph.rb and grtest.rb at top dir. 2nd one depend on a big csv data file - their are gc/cairo-destroy problem with very large collections of data.

[passenger94]

Woa ! I'm pretty sure plot.c is doing coffee if asked politely, !!! will find the appropriate function tomorrow, bye :-)

[ccoupe]

Make coffee? Why not? It's only a matter of time until someone asks for bar charts and pie charts and and .... Actually, the #266 text shape would allow doing everything in ruby and a lot easier to code.

I really want to use pangocairo layout instead of the cairo toy interface but there are so many other things that need to be done first.

[ccoupe]

Can we have more than one plot?

It's just kind of widgety.

[passenger94]

"Bang for the buck" That's what i'm talking about ! :-D

[ccoupe]

There's nearly unlimited things that could be added. Visually oriented people might want so set the background color: plot {background: "cornsilk")

How hard would it be to draw a Column chart?

That needs some tuning.

[passenger94]

[ccoupe]

Nearing completion of the bar chart (I hope).

There is a problem - drawing x axis labels can't be set to dark gray. [fixed 2016-Sep-18]

Also a wiki update

[ccoupe]

So a scatter chart is like a line chart without the connecting lines. How hard could that be?

Actually, a little easier than I thought and a little harder because it exposed some glossed over drawing issues.

[ccoupe]

Turns out that is not what a scatter chart is according to my ancient papyrus Excel manual.

[ccoupe]

An almost good scatter plot. I wish I knew what I was doing.

also update the wiki

[ccoupe]

On the shiny side of life:

On the dull side of life, there are some odd problems with color choice for the wedges. The legend is correct. (same code) and % labels are not drawn, the latter is not something I care to think about tonight.

[dredknight]

Dang this looks awesome... I already know how I am going to use it :D. Great job!!!

[IanTrudel]

Are we going to have some radar chart? Also perhaps some way to easily extend the feature with new graphs?

[ccoupe]

There is no user extensible charting for this feature (#264). It's all done in C and cairo. If I get pie charts working I'll probably pause there. There are many things missing in the existing charts and I suspect errors too. And there are testing problems. One could spend a lifetime recreating charting functionality in Shoes when better standalone programs exist. OpenOffice/LibreOffice are free, better and easier to work with. Large data sets or complex analytics seem to be done in R+gnuplot or Matlib or other programs that Shoes shouldn't attempt to be (IMO).

If #266 was implemented then you have everything needed to implement your chart in Ruby in a Shoes user Widget (same painful problems as C though).

[ccoupe]

One interesting idea for extensibility: Consider rcairo (it's a gem so we can include it in Shoes in several ways). Perhaps a new widget? Call it raw_widget, for example

Shoes.app do
  raw_widget 600, 400, do 
     require 'rcairo'
     #ruby_cairo drawing code here
  end
end

That Shoes widget just calls the user's block at draw time and Shoes manages the rectangle placement in the overall layout. If you can figure out cairo & rcairo, you could draw damn near anything you want. Cairo is not friendly for beginners but it can also be easier than managing shoes shapes (#266). Just a thought that I wish I'd thought of earlier.

[IanTrudel]

It might be possible to write an overriding drawing method, or some kind of call back, that is attaching to your existing C code. Normally the input data is always the same. The difference is the representation on screen. The callback would be on Ruby level.

[ccoupe]

@BackOrder, sadly that would require a Shoes api (new methods) into the chart drawing code for the user to call and so far there is little in common in all the chart types that could be exposed to Shoes/Ruby and woe be those who call it incorrectly or out of order since it runs deep inside gtk/cocoa event draw code you may not get ruby or toolkit exceptions (I know this for a fact);

There is some abandoned python code that I'm slowly converting to C for the pie chart that might convince you that a charting framework can be done in pure Ruby/Shoes if some one wanted to do that (and #266 or the rcairo thing), It has something called a ringchart.py which might be close to your radar chart. Or not.

[ccoupe]

Nearing completion. Labels around the pie.

Raw values are the default labels. % is an option

[ccoupe]

Optional box around the legend - see update wiki page for an example of how that looks.

My search for radar chart producing code to crib from turns up very little. In the pycha there is a radial.py chart that might be close (or not) but its only used in his 'chavier' app and that has a startup bug in my python (2.7) and my python skills are low. Help? (it uses pygtk and pycairo so that my prevent some Windows or OSX help - don't know).

I think I get the idea: Using the download example data shown above, there would be 8 lines drawn from the center point (of equal angle - the length of the line would be scaled by it's % (above). Draw a litlte dot there and perhaps the value label. - Then draw lines connecting each of those dots. Is that correct? A helpful someone could even do the drawing the line and math in Shoes/Ruby and then I'd really have good example.

[ccoupe]

LibreOffice has a Net chart type that behaves almost like I describe above - the length of the radial axis lines scale to the largest value in the data set - I mistakenly said % of sum. Here's a Net using download data above [15, 5, 4, 4, 3, 3, 1, 1]

Not useful for me, for this data but the important point is whether this is what @BackOrder is thinking? Bar Chart in a circle ;-)

[IanTrudel]

Something along this line... Radar charts has many alternative names. Let me show you a real case scenario that I wrote before... Statistics Feature for Humintell Products. It should give you a better idea.

[ccoupe]

Thanks, @BackOrder . They are similar charts (LibreOffice Net and what you used) I've created a gr6.rb test case based on the Advertising sample you provided -- assuming I can convert that radial.py from pycha. I can also see why some folks consider radar as mis-leading eye candy. Not my fight. Probably left brain or right-brain related.

[IanTrudel]

Charts are charts. One could argue that the people are those whom are misleading rather than the charts.

The way radar chart is used in the document aforementioned is rather to visually see someone's progress, or area of improvements, at key point in time (pre-training, during training, post-training).

[ccoupe]

One could argue that the people are those whom are misleading rather than the charts.

Too true. I've seen bad, good and ineffective charts. I've created some of each myself. Another reason I don't want to get too complex in Shoes plots.

[ccoupe]

I think this is a reasonable target for Shoes -

Not nearly as pretty and LibreOffice goes counter clockwise and the msft package goes clockwise. Minor point if you think about it.

[ccoupe]

I managed to get the pycha app, chavier working. Too bad it's abandoned, sigh - its useful with bugs of course. I was able to entering the advertising sample (Test/plot/gr6.rb value) and it produced this for it's radial chart type:

That's damn close to what I was looking for which means I can probably use the radial.py python code and convert it to C/cairo, In case you want fix your copy of chavier/app.py, around app.py:96, make it look like

        legend=dict(
            opacity=float,
            borderColor=str,
            borderWidth=int,
            hide=bool,
            legendFont=str,
            legendFontSize=int,
            position=dict(
...

[IanTrudel]

@grf.add num_obs: @values1.size, values: @values1, xobs: @xobs,
      name: "Year 1", minv: 0, maxv: 200, color: "dodgerblue",
      nubs: "dot", strokewidth: 1

Why do we have to send the size when an array is sent to values:? Normally the number of items would be calculated internally.

[ccoupe]

Good Question. I could not find a documented C level api that returns the length of an array. It also complicates things at the C level. I'd be happy to be pointed to the C call for it but I couldn't find it.

[IanTrudel]

Just a wild guess... rb_ary_size()

[ccoupe]

Doc - https://github.com/ruby/ruby/blob/6a69ab937c7e6584d311cd444615cc1cd5b5499b/doc/extension.rdoc . No doubt there is a way otherwise ruby couldn't work.

[ccoupe]

macro RARRAY_LEN(value) seems to work. It's existence is hinted at if you know about RSTRING_LEN and read the doc very carefully. Excellent. num_obs: is now ignored. Thanks.

[IanTrudel]

Very good job.

I'd also like to recommend changing xobs to labels, which is more comprehensible for anyone whom ever used charts. minv and maxv could be just min and max.

What is nubs ?

[ccoupe]

Reasonable suggestions and easy to do.

What is nubs ?

Nubs are those little boxs or dots or diamonds drawn in line and scatter charts. I don't know what they should be called. I think I documented them on the wiki article. Since I haven't written the shoes manual yet, now is good time to be changing these things.

[IanTrudel]

Google calls them Points, isn't it?

Alternatives: LineStyle, Style, LineType, Type, PointShape, Shape, etc.

Point, Style or Shape could do it. Short and simple.

How does LibreOffice call them?