make graphics quality a qualifier of SET WINDOW, instead of a mode.

[karlmsmith]

Reported by @karlmsmith on 10 Dec 2012 19:38 UTC From Andrew:

In the documentation, when introducing SET GRAPHICS HIGH, be sure
to explain that it has no effect on existing windows; one has to open
a new window, or cancel and reopen an existing window for it to take
effect.  Actually, it would be better to have this happen
automatically -- otherwise, a script that requires GRAPHICS HIGH won't
know whether or not it has to cancel & reopen the window.

Definitely need to add more explanation. Hesitant about automatically switching open windows, since that will destroy whatever previously was in the window. One may want an draft or animation window as well as high-resolution window open at the same time.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/2006

[karlmsmith]

Comment by steven.c.hankin on 10 Dec 2012 21:47 UTC I wonder if the way to think of this is that it is an in-built dilemma to using a "modal" solution. What would be the trade-offs to eliminating the modal "SET GRAPHICS" altogether and instead adding a qualifier onto the SET WINDOW command

   yes? SET WINDOW/quality=high  [window number]
   yes? SET WINDOW/new/quality=high

These would seem to give an unambiguous control over the graphics environment and straight forwardly support all of the permutations that a user may want. Changing the quality mode of the current output window might have to clear the window in the process, but if the code has no reasonable way to automatically re-draw the window at higher quality this is unavoidable and not a big deal. If the user doesn't want the current window cleared, they can include a /NEW qualifier.

[karlmsmith]

Comment by @AndrewWittenberg on 11 Dec 2012 23:12 UTC I second Steve's suggestion -- having this as a qualifier to SET WINDOW would be more flexible, and conceptually clearer, than the MODE approach. I'm changing the summary and component to reflect this.

[karlmsmith]

Comment by @karlmsmith on 28 Dec 2012 17:56 UTC Will implement this change. However, "/quality=" only sounds reasonable for "high" (pyqt creates, saves, and displays the image) and "draft" (cairo creates and saves the image; pyqt displays the cairo image). Not sure if it sounds very good for "unmapped" (cairo creates and saves the image; no displayed image).

It is trying to specify one decision ("what graphics engine do you want to use to render the graphics"), so I am hesitant to split it into two command options. Maybe "/graphics=" instead, and maybe using "none" instead of "unmapped"? If two options, then the "/quality=" and "/unmapped" (or another name?) would be mutually exclusive. The default will be the high-quality graphics unless the -gif or -unmapped command-line options are given; then the default will be the unmapped graphics.

I will also add "/noaa" (no antialiasing); the default will be to use antialiasing.

I will also add "/engine=" for user-specified custom graphics engine to deal with the graphics for the window. This would be a power user/programmer option that I expect not really to be used at this time.

I guess the "set graphics" command would then just be removed completely. Any scripts with this commands will generate an appropriate error message.

[karlmsmith]

Modified by @karlmsmith on 28 Dec 2012 18:10 UTC

[karlmsmith]

Comment by @AndrewWittenberg on 28 Dec 2012 19:28 UTC I'm a little unclear on the "unmapped" qualifier. We want to be able to run scripts in either interactive or batch mode -- with or without a display attached -- without any changes to the script itself. Would all 4 of those combinations currently work in PyFerret, in tandem with the "-batch" command-line option? Would all graphics output types would be available for each of those combinations?

I'd suggest having PyFerret detect whether there's a valid display; if not, then just skip the PyQt rendering and proceed as normal.

[karlmsmith]

Comment by @karlmsmith on 28 Dec 2012 23:38 UTC With unmapped graphics (ie, using the Cairo graphics library), either from a /graphics=none or /unmapped qualifier, or -gif or -unmapped command-line option, nothing is displayed and a display device is not required. Everything in done in memory. Even when there is a valid display, the user may not want anything displayed, probably for speed, but possibly also to avoid temporarily displayed graphics when running some script.

Everything should work exactly the same for unmapped graphics, including "frame /file=(filename.ext)" commands if saving raster images (PNG files).

For vector images (PDF, PS files), everything should work the same if the saved filename and format is known prior to creating the "window" (or the first drawing), either from using the -batch (filename.ext) command-line option or from a "set mode batch:(filename.ext)" where the extension is recognized (.ps, .pdf) This is because Cairo writes the image directly to the file in the appropriate format. A "frame /file=(filename.ext)" will be silently ignored (since it is writing to file).

I could set it up so if opening a displayed window fails, it would revert to then unmapped option. The down side is that it might be confusing if one really wants displayed graphics but can't get it anything displayed. A warning would definitely be required if this was implemented and occurred.

[karlmsmith]

Comment by @AndrewWittenberg on 29 Dec 2012 01:02 UTC Just tried generating multiple PostScript files in a single PyFerret 1.0.0 session. It works ok when a display is available: -batch and -unmapped produce bitwise-identical PostScript files (apart from the %%CreationDate).

However, I was surprised that PyFerret popped up a display window. And when I unsetenv the DISPLAY variable, PyFerret gives me the following errors:

PyQtPipedViewer: cannot connect to X server 
grdelWindowDpi: error when calling the Python binding's windowDpi method: [Errno 4] Interrupted system call
STOP FGD_GETDPI: Uninitialized window DPI

There's also a minor bug in how the argument of "-script" is passed into Ferret -- it ought to be inserted inside of double-quotes, otherwise the "go" command will complain due to its misinterpreting directory slashes as qualifiers:

$FER_DIR/bin/pyferret -unmapped -script /home/atw/mytest/ferret/pyferret/ps_multi.jnl

    NOAA/PMEL TMAP
    FERRET v6.83 (PyFerret 1.0.0)
    Linux 2.6.32-279.5.2.el6.x86_64 - 10/19/12
    28-Dec-12 19:58     

 **ERROR: unknown command qualifier: home
go /home/atw/mytest/ferret/pyferret/ps_multi.jnl
Command file, command group, or REPEAT execution aborted

[karlmsmith]

Comment by @karlmsmith on 2 Jan 2013 23:08 UTC The reason for window in -unmapped mode was because the script run contains the command "set graphics/antialias high" which resets the graphics engine.

All the -unmapped argument does is set the default graphics engine to Cairo (undisplayed, in-memory graphics). But this default can be changed using the "set graphics" command, which in your script sets it to the PyQt engine (displayed, high-quality graphics). So both saved image files were actually produced by the same engine (PyQt). And the script with the Cairo engine won't do the same thing. (See below.)

When the change is made to be a qualifier of "set window", then the start-up default (unmapped if "-unmapped" used; otherwise high-quality) cannot be changed. I assume that if unmapped is the default, then the "/quality=" qualifier should be ignored so unmapped is always used.

Making another ticket about quoting the "-script" script filename for the record, but will immediately close it as I made the quick change required.

Andrew, The script you wrote will not work the same in unmapped mode as Cairo writes PostScript and PDF directly to a file that must already be named and frame/file commands are ignored. Since you did not name a PostScipt output file prior to opening the window, your script could only output PNG files, so your save as PostScript would fail in unmapped mode. So:

set graphics high
use coads_climatology
set win/size=.5/asp=.7 1

shade/l=1 sst
frame/file="sst_1.ps"

shade/l=2 sst
frame/file="sst_2.ps"

shade/l=3 sst
frame/file="sst_3.ps"

is fine for high-quality, or for unmapped saving as PNG (after removing the "set graphics high" and changing .ps to .png). But for unmapped saving as PostScript it would need to either look like:

set graphics unmapped
use coads_climatology
set win /size=0.5 /asp=0.7 1

! changing the "metafile" name closes the old image/file and opens a new file
set mode meta sst_1.ps
shade /l=1 sst
! frame /file=... would be silently ignored as the engine is writing to file

set mode meta sst_2.ps
shade /l=2 sst

set mode meta sst_3.ps
shade /l=3 sst

or instead, call pyferret three times with a batch script:

pyferret -batch sst_1.ps -script sst_shade 1
pyferret -batch sst_2.ps -script sst_shade 2
pyferret -batch sst_3.ps -script sst_shade 3

where sst_shade.jnl is

query /ignore $1"<usage: GO sst_shade l_val"
use coads_climatology
set win /size=0.5 /asp=0.7 1
shade /l=$1 sst

This last scenario - a script that creates a single image - is what I thought you meant when asking if using -batch (which implies -unmapped and saves to the indicated filename) will give you same results as high-quality.

[karlmsmith]

Comment by steven.c.hankin on 3 Jan 2013 23:19 UTC By way of disclaimer lest I say something foolish I confess that I don't understand all of the issues. My overall reaction is to urge simplicity as a guiding principle here; if the user can achieve a goal in a single self-evident way, then reconsider whether secondary approaches might be eliminated if they muddy the waters. Specifically, if "-unmapped" is available as a Unix command line option, is there a compelling need for it as a Ferret command qualifier as well?

The graphics output file syntax that pyFerret has inherited from Ferret of old, is already full of arbitrary, historically-accreted complexities and contradictions. It sounds as if it might be worth considering whether to reduce the number of distinct places in the syntax at which the output filename may get specified.

[karlmsmith]

Comment by @karlmsmith on 4 Jan 2013 00:44 UTC My original scenario for the /unmapped window was if you were working in draft quality window and then wanted to create a high-quality image of what you just made in the draft quality window. But more likely one would quit, tweak the ferret.jnl script, and rerun with -batch. Or now with high-quality as the default, one would just save what they have.

So yes, really don't need /unmapped. I will accept "/quality=unmapped" just in case there is a need, and not worry that it sounds awkward since it will be rarely used, if ever. And I will eliminate /engine; again, allow /quality=(enginename) for my use.

The -gif, -batch, and -unmapped were different things under Ferret, but they all now correspond to "/quality=unmapped" in PyFerret, with -batch also giving a filename (default of ferret.png) for the (final) image. So I think of -gif and -unmapped as supported for backwards compatibility (LAS uses -gif). Probably should try to focus the documentation on just using -batch; I think that is the primary use case - "(re-)run this script except write it to myimage.pdf instead of displaying it".

[karlmsmith]

Comment by @AndrewWittenberg on 15 Jan 2013 00:00 UTC There are a number of remaining issues with the current design of PyFerret's graphics output. In this comment I'll enumerate the issues, and in the next comment I'll make a proposal for how to address them.

First, a modified Ferret script to generate multiple output files in one Ferret session:

\can mode verify
!  ps_multi.jnl
! (at GFDL: /home/atw/mytest/ferret/pyferret/ps_multi.jnl)

def sym out_type $1"ps"
def sym graphics_mode $2"unmapped"

def sym w_num 1

use coads_climatology

def sym open_win can window (\$w_num)\; set graphics/antialias (\$graphics_mode)\; set window/size=.5/aspect=.7 (\$w_num)

def sym set_ps set mode meta \"(\$out_prefix).(\$out_type)\"
def sym flush_ps set mode meta null.(\$out_type)

def sym set_png " "
def sym flush_png frame/file=\"(\$out_prefix).(\$out_type)\"

def sym out_prefix "sst_1"
($open_win); ($set_($out_type)); shade/l=1 sst; ($flush_($out_type))

def sym out_prefix "sst_2"
($open_win); ($set_($out_type)); shade/l=2 sst; ($flush_($out_type))

def sym out_prefix "sst_3"
($open_win); ($set_($out_type)); shade/l=3 sst; ($flush_($out_type))

set mode/last verify

We can call the above Ferret script using the following shell script:

#!/bin/tcsh -f
# test_ps_multi.csh
# (at GFDL: /home/atw/mytest/ferret/pyferret/test_ps_multi.csh)

setenv ATW_FERRET_CLASS pyferret
source /home/atw/.atw_environment
set pyferret = $FER_DIR/bin/pyferret

set idir = /home/atw/mytest/ferret/pyferret
set script = $idir/ps_multi.jnl

# draft mode
set out_dir = $idir/draft/png; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -script \"$script\" png draft

set out_dir = $idir/draft/ps; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -script \"$script\" ps draft

# high mode
set out_dir = $idir/high/png; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -script \"$script\" png high

set out_dir = $idir/high/ps; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -script \"$script\" ps high

# batch mode
set out_dir = $idir/unmapped/png; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -batch -script \"$script\" png unmapped

set out_dir = $idir/unmapped/ps; rm -rf $out_dir; mkdir -p $out_dir; cd $out_dir
$pyferret -batch -script \"$script\" ps unmapped

This creates a tree of output files that look like:

.
|-- draft
|   |-- png
|   |   `-- PET0.ESMF_LogFile
|   `-- ps
|       `-- PET0.ESMF_LogFile
|-- high
|   |-- png
|   |   |-- PET0.ESMF_LogFile
|   |   |-- sst_1.png
|   |   |-- sst_2.png
|   |   `-- sst_3.png
|   `-- ps
|       |-- PET0.ESMF_LogFile
|       |-- sst_1.ps
|       |-- sst_2.ps
|       `-- sst_3.ps
`-- unmapped
    |-- png
    |   |-- ferret.png
    |   |-- PET0.ESMF_LogFile
    |   |-- sst_1.png
    |   |-- sst_2.png
    |   `-- sst_3.png
    `-- ps
        |-- PET0.ESMF_LogFile
        |-- sst_1.ps
        |-- sst_2.ps
        `-- sst_3.ps

The issues, in order of priority:

1) The high/{ps,png} images spuriously omit the color key, time & dataset labels, and Ferret logo.

2) We should retain the ability to enforce, from the Ferret command line, a displayless (graphics=unmapped) mode, so that this doesn't have to be detected by (or passed through to) all the user scripts that actually generate the images.

3) There should be a way to flush the graphics output (e.g. with CANCEL MODE METAFILE), without switching to a "null" file.

4) Extra file created at unmapped/png/ferret.png.

5) Unnecessary creation of PET0.ESMF_LogFile.

6) No graphics output created in draft/{png,ps}.

7) In GhostView (gv), the high/ps images are rendered much larger than the unmapped/ps images, and there are lots of PostScript differences between unmapped and high. The high (Qt-4.8.2) ps files have

%!PS-Adobe-1.0
%%BoundingBox: 0 0 612 792
%%Pages: (atend)
%%Orientation: Landscape
%%DocumentFonts: (atend)

and the unmapped (Cairo-1.12.2) ps files have

%!PS-Adobe-3.0
%%BoundingBox: 5 19 390 565
%%Pages: 1
%%DocumentMedia: 142x203mm 404 577 0 () ()

It would be best to make those PostScript files as similar as possible, at least within the header.

8) The high/png images are about 20% smaller in "screen size" than the unmapped/png files.

9) I've never liked the paradigm of specifying a single graphics output file from the command line with -batch. In principle it is limiting (though in practice, just confusing) when we actually want to create multiple figures within a single Ferret session.

10) From a user perspective, it's confusing to have both a FRAME command and a METAFILE mode, especially when there's no actual metafile.

In the next comment I'll outline a possible interface solution.

[karlmsmith]

Comment by @AndrewWittenberg on 15 Jan 2013 00:04 UTC To resolve the above issues with PyFerret graphics generation, I have a proposal. How about if we eliminate the following in PyFerret:

SET MODE METAFILE
CANCEL MODE METAFILE
-batch [filename]
-unmapped
-gif

and add a qualifier METAFILE to SET WINDOW:

SET WINDOW/SIZE=.5/ASPECT=.7/METAFILE=ps 1
SET WINDOW/SIZE=.5/ASPECT=.7/METAFILE=pdf 1

METAFILE would cue Ferret to save all of that window's vector information as needed to create a PS or PDF file, in a unique temporary file (like /tmp/ferret.metafile_window1.ps.$$, or perhaps even in memory at /dev/shm) which the user would never see. The user could then call FRAME to copy that file (for the current window) to its final destination:

FRAME/file="myfile.ps"

If the user wanted to run on a machine with no display, they could start up Ferret (either interactively or with -script) using "-nodisplay", which would essentially do what "-batch" does now, but with a more accurate name, and without the confusing filename argument.

[karlmsmith]

Comment by @karlmsmith on 18 Jan 2013 18:35 UTC Pondering some on whether it should be /metafile or /metaformat or /format - or something else, but I do like the idea of saying what the save format will be for unmapped / nodisplay windows without being committed to a filename. Then, like the other modes, the graphics file is saved with the FRAME /FILE=... command.

For displayed graphics this qualifier would have no meaning - high quality can save in any format any time; draft quality can only save in raster formats (PNG) any time. So this could be the indicator for unmapped graphics: SET WINDOW [ /QUALITY=... | /METAFORM=... ] Wondering if this would make sense to users.

Not sure we could eliminate the old commands/options from Ferret (for backwards compatibility) but certainly can deprecate them. I know LAS uses -gif with Ferret, but that can easily be changed when making the changes to use PyFerret. Maybe it is time for some house-cleaning?

The modifications being made will enforce the unmapped mode when the -unmapped / -nodisplay command line option is given; probably output a note that /QUALITY=... was ignored (when given) when run with this option.

[karlmsmith]

Comment by steven.c.hankin on 18 Jan 2013 18:58 UTC Replying to [comment:13 karl.smith]:

Pondering some on whether it should be /metafile or /metaformat or /format - or something else, but I do like the idea of saying what the save format will be for unmapped / nodisplay windows without being committed to a filename. Then, like the other modes, the graphics file is saved with the FRAME /FILE=... command.

For displayed graphics this qualifier would have no meaning - high quality can save in any format any time; draft quality can only save in raster formats (PNG) any time. So this could be the indicator for unmapped graphics: SET WINDOW [ /QUALITY=... | /METAFORM=... ] Wondering if this would make sense to users.

As a reader of this text I am confused. (Of course I am easily confused!) The confusion is as likely to be about the text of this trac ticket as about your proposed changes. Spelling out concrete examples of sessions that use Ferret in various ways would help -- batch versus interactive; high versus low quality; image versus PDF output.

Not sure we could eliminate the old commands/options from Ferret (for backwards compatibility) but certainly can deprecate them. I know LAS uses -gif with Ferret, but that can easily be changed when making the changes to use PyFerret. Maybe it is time for some house-cleaning?

Quite likely this is a good time to do house-cleaning. Making the shift from Ferret to pyFerret under LAS is as good a time for these sorts of mild disruptions.

The modifications being made will enforce the unmapped mode when the -unmapped / -nodisplay command line option is given; probably output a note that /QUALITY=... was ignored (when given) when run with this option.

"enforce the unmapped mode". Sorry for being dense ...

[karlmsmith]

Comment by @karlmsmith on 18 Jan 2013 19:47 UTC Some example scenarios:

% pyferret

yes? use coads_climatology

yes? SET WINDOW /NEW /QUALITY=HIGH 
(window is displayed)
yes? shade sst[T=15-aug]
yes? FRAME /FILE=augsst.pdf
(augsst.pdf is created)
yes? FRAME /FILE=augsst.png
(augsst.png is created)

yes? SET WINDOW /NEW /QUALITY=DRAFT
(window is displayed)
yes? shade sst[T=15-aug]
yes? FRAME /FILE=augsst.pdf
**ERROR: DRAFT QUALITY windows cannot save in PDF format
yes? FRAME /FILE=augsst.png
(augsst.png is created)

yes? SET WINDOW /NEW /METAFORM=pdf
(no window is displayed)
yes? shade sst[T=15-aug]
yes? FRAME /FILE=augsst.pdf
(augsst.pdf is created from temp file created by Cairo)
yes? FRAME /FILE=augsst.png
**ERROR: META windows can only save in the format originally specified

yes? SET WINDOW /NEW 
(default is a displayed high-quality window - everything same as for /quality=high)
 ...

yes? quit

% pyferret -nodisplay

yes? use coads_climatology

yes? SET WINDOW /NEW /METAFORM=pdf
(no window is displayed - everything same as /metaform=pdf above
 ...

yes? SET WINDOW /NEW /QUALITY=high
**WARNING: /QUALITY qualifier is ignored when pyferret started with -nodisplay
(no window is displayed - everything same as /metaform=??? - what format???)

yes? SET WINDOW /NEW
(again, what default format???)

So this has shown up an problem to be worked out. If -batch , or SET MODE META then the answer is the format is that indicated by . But if -batch and SET MODE META not given, then default to what? PDF? Could make this an error instead of a warning, but want to be able to run scripts that have "SET WINDOW /QUALITY=" in them without change. Maybe -nodisplay (maybe another name) needs an argument like /metaform, maybe defaulting to PDF if not given?

[karlmsmith]

Comment by steven.c.hankin on 18 Jan 2013 20:50 UTC Very helpful. Thanks.

One big change in the above examples is that when yes? SET WINDOW /NEW /METAFORM=pdf the new window is invisible. Does this mean that the user will see little or no indication of the subsequent graphics commands that are running? This seems like a problem. If things are unexpectedly slow or something hangs, the user can only wonder what's going on .... How compelling is the need to simulate batch behavior in the midst of an interactive session? Wouldn't it be better/simpler/more elegant just to ensure that Ferret responds consistently whether it is in batch or in interactive mode -- hard copy outputs created by the very same commands.

In the examples above the only argument shown or discussed for /META and SET MODE META is PDF. Is there some other possibility? If not, then can these syntaxes be simplified?

The goal when running in -batch mode (or whatever the non-interactive mode is called), I'd think, is that we want to be able to tell Ferret to create PDFs and/or images, right? And to be able to make sense of the same commands that we give during an interactive Ferret session.

Ignoring backwards compatibility for the moment (we can deprecate syntaxes that we plan to eliminate in the future), suppose we made these changes: o eliminate (deprecate) "-batch " and "SET MODE META" o add "SET MODE FRAME filename" and "pyFerret -framefilename " as 2 ways to set the default FRAME output filename o allow the FRAME command, when given without a filename, to use the default FRAME output filename

Would this give us the level of flexibility and consistency needed to cover all cases?

My suggestion would be to take another pass through the scenarios using the best, simplest, most consistent and flexible syntax you can think of. Give no thought to backwards compatibility initially. Then, after you have something you like, figure out how to interpret the backwards-compatible legacy commands.

[karlmsmith]

Comment by @karlmsmith on 18 Jan 2013 20:58 UTC Regarding other meta formats: in addition of PDF, there is also PostScript (PS) and Scalable Vector Graphics (SVG). Patrick Brockmann is a big fan of the latter, which is basically an XML listing of the drawing commands and so are easily edited.

[karlmsmith]

Comment by steven.c.hankin on 18 Jan 2013 22:24 UTC I accept the argument you are making here. The added flexibility of directly outputting SVG or PS graphics from Ferret is small, but so is the price that gets paid by having an optional argument to /META.

One consideration that hasn't been factored into the syntax yet: how to create multi-page PDF outputs. FRAME/APPEND??

[karlmsmith]

Comment by @AndrewWittenberg on 19 Jan 2013 02:18 UTC We do need to allow PS as an output type, since that is all that LaTeX accepts (and more than half of us use LaTeX to write our papers). There are also lots of free tools to manipulate/rotate/mosaic/append PS files; not necessarily so for PDF files.

Thinking outside the box here -- for consistency, I'd suggest treating the interactive case as the special case. I.e., first create the graphics entirely in-memory, so that we're always ready to dump graphics to a file. Then only if a display is available, copy those graphics from memory to the screen, whenever the display needs to be updated. For simplicity, I'd suggest having only one "quality", i.e. eliminating the QUALITY setting.

Then renaming SET WINDOW/METAFORM to SET WINDOW/FRAME, here's a suggested interface:

DISPLAYED MODE

% pyferret -frame_default=png
(and we could have a command-line "default" of -frame_default=png)

yes? use coads_climatology
yes? SET WINDOW 1    ! defaults to FRAME=PNG
yes? shade/l=1 sst
(window is displayed)
yes? FRAME/FILE=sst.png
(sst.png is created)
yes? FRAME/FILE=sst.ps
**ERROR: Cannot create PS file; need to SET WINDOW/FRAME=PS first

yes? SET WINDOW/NEW/FRAME=PS 1
yes? shade/l=1 sst
yes? FRAME/FILE=sst.ps
(sst.ps is created)
yes? FRAME/FILE=sst.png
(sst.png is created)

yes? SET WINDOW/NEW/FRAME=PDF 1
yes? shade/l=1 sst
yes? FRAME/FILE=sst.pdf
(sst.pdf is created)
yes? FRAME/FILE=sst.ps
**ERROR: Cannot create PS file; need to SET WINDOW/FRAME=PS first
yes? quit

NON-DISPLAYED MODE (generally used with "-script")

% pyferret -nodisplay -frame_default=png

yes? use coads_climatology
yes? SET WINDOW 1    ! defaults to FRAME=PNG
yes? shade/l=1 sst
(no window is displayed)
yes? FRAME/FILE=sst.png
(sst.png is created)
yes? FRAME/FILE=sst.ps
**ERROR: Cannot create PS file; need to SET WINDOW/FRAME=PS first

yes? SET WINDOW/NEW/FRAME=PS 1
yes? shade/l=1 sst
yes? FRAME/FILE=sst.ps
(sst.ps is created)
yes? FRAME/FILE=sst.png
(sst.png is created)

yes? SET WINDOW/NEW/FRAME=PDF 1
yes? shade/l=1 sst
yes? FRAME/FILE=sst.pdf
(sst.pdf is created)
yes? FRAME/FILE=sst.ps
**ERROR: Cannot create PS file; need to SET WINDOW/FRAME=PS first
yes? quit

So the non-displayed case would be exactly the same as the displayed case, and produce the same files.

Having a single invisible window in interactive mode isn't that important to me, at least not yet; if display speed or dimensions were an issue, I'd probably always go into batch mode after writing my script. So I don't think we need SET WINDOW/NODISPLAY, at least not until we identify a need for it.

At the command line and/or with SET WINDOW, I much prefer the idea of specifying the frame type (extension) rather than the frame file, since the most common use case (at least for us) is producing multiple files of the same type within a single Ferret session.

% pyferret -nodisplay -frame_default=ps

yes? use coads_climatology
yes? SET WINDOW 1    ! defaults to FRAME=PS
yes? shade/l=1 sst
(no window is displayed)
yes? FRAME/FILE=sst1.ps
(sst1.ps is created)
yes? shade/l=2 sst
[would we need to SET WINDOW again here???]
yes? FRAME/FILE=sst2.ps
(sst2.ps is created)
yes? FRAME/FILE=sst2.png
**ERROR: Cannot create PNG file; need to SET WINDOW/FRAME=PNG first
yes? quit

[karlmsmith]

Comment by @karlmsmith on 22 Jan 2013 19:30 UTC ''If'' I used Cairo library 1.10 or later (latest 1.12), I could create a "recording surface" that would be like the high-quality mode in that it saves all the drawing commands in memory, which then can output to whatever format whenever desired. This could then be used for saving undisplayed images or for the create-in-memory-then-display scenario. But RHEL5 is at Cairo 1.2 and RHEL6 is at Cairo 1.8, so this would mean statically linking in the latest Cairo library. We do this with HDF5 and NetCDF4, and with XGKS in Ferret, so not completely unreasonable, but this would be another library to keep up with. Not sure what issues might arise or what the memory requirements would be for keeping a complex image in memory. (Why did RHEL6 not go with 1.10?; and 1.12.10, just released, includes "a heap of bug fixes" - not sure when the bugs arose),

''If'' I stick with the OS version of Cairo, then my only in-memory option is raster (PNG); other formats write directly to file, which have to be closed out for display. Thus the draft mode (which uses exactly the create-in-memory-then-display scenario) can only display and save in raster formats. So we need to keep the high-quality mode (let Qt do the drawing).

''Assuming I stick with the OS version of Cairo:''

I do like the idea of the user specifying the format to use for a window ahead of time (with some default if not specified). I would be inclined to enforce that the FRAME command must be in this format, even for high-quality where I could relax this restriction. Making this restriction would ensure that rerunning the script with "pyferret -nodisplay -script ..." (ie, using Cairo) would always work. There would still be the options in the high-quality GUI menu bar to save in other formats if desired.

So undisplayed windows would always use Cairo; displayed vector graphics (PS, PDF, SVG) would use high-quality (PyQt); raster graphics (PNG) would use draft-quality (Cairo-Qt). FRAME /FILE=... must match the window format specified earlier. I would be inclined to say the default be PDF or PS (thus making high-quality the default).

To make undisplayed and displayed match behavior, once a FRAME /FILE=... command is entered, that would be the end of drawing on a PDF or PS window. (Cairo has to close out the file; to reopen this file and append appropriately would require some nasty hacking.) One could continue to draw on a PNG window, but we may want to disallow it to ensure consistent behavior between formats.

To generate multi-page PDF or PS, it would have to be a option with SET WINDOW, such as SET WINDOW /NEWPAGE, which would issue a new-page command in the PS or PDF file and keep it open. This is for undisplayed windows. For displayed windows, my viewer(s) would need to be told what to do with this command. To keep the viewer simple, I would be inclined to just treat it as a clear-window (thus previous pages are lost). Other options are possible (tiled windows, etc.), but I just don't want to waste time creating yet another PDF viewer.

[karlmsmith]

Comment by steven.c.hankin on 22 Jan 2013 19:52 UTC Hi Karl,

Here's how I am (wrongly?) understanding the trade-off you have laid out about which version of Cairo to use: o The next version of Cairo (1.10 or later) offers the "right" solution. To use this version requires (only) a change in the link statement. o If we stick to the current version of Cairo (pre-1.10) we will have to commit to a command syntax and a set of behaviors that is noticeably inferior to what we would choose if we had Cairo 1.10

If this summary is correct, then it sounds like we should advance to Cairo 1.10. The inconvenience of messing around with the Ferret link statements seems much less than the inconvenience of programing in graphics behaviors that are not what we want ... and then either living with the inferior behavior forever or updating code, command syntax and documentation later.

[karlmsmith]

Comment by @karlmsmith on 22 Jan 2013 20:58 UTC ''If I link in the latest Cairo library'':

Added step in preparing to build PyFerret: download Cairo source, build static library, link in static library. Need to see what other requirements might get pulled in with this (i.e., what other libraries, if any, need to be provided/updated), although appears it should work with whatever is provided with the OS (famous last words).

Can use newer features (such as the recording surface) and should produce better output files (for example, PDFs are not compressed using the RHEL 5 version of Cairo).

Could do away with pre-specifying output format when creating a window. All drawing for an undisplayed window will be made using the "recording surface" in Cairo, which can then be saved in any format when a FRAME command called. Not sure how speed and memory usage will be affected by this change.

Can "save and continue" a drawing (FRAME command does not end drawing to a window).

Draft quality (needed for animation) and high-quality would remain unchanged. Only undisplayed mode affected.

Could do away with quality specification: always high-quality (PyQt does it all) unless in an /animate loop, which would create a new draft-quality window (Cairo-PyQt hybrid), or if -undisplay (Cairo does it all) on the command line. Original reason for draft quality, however, was to provide a faster displaying window, so maybe there still is some use for this option.

Multi-page PDF or PS would still be as mentioned earlier - require something like SET WINDOW /NEWPAGE; probably only treated properly in undisplayed windows.

---

If I can get through the first step (building the library) without too many complications, then this probably is the way to go.

[karlmsmith]

Comment by @AndrewWittenberg on 22 Jan 2013 23:09 UTC I like the direction this conversation is taking. As long as the interface is simple, identical for displayed and nondisplayed, and we can easily generate multiple images within a single Ferret session (displayed or not), then I'm happy.

The new-Cairo route offers many advantages, and seems worth the effort to statically link, at least until the OS's catch up. The compressed PDFs, save-and-continue ability, and the ability to do multiple FRAMEs of different image types without pre-specifying the format, are all big pluses in my opinion, especially if the interface is simpler.

I agree that when generating multi-page PDFs, Ferret need only display the most recent output image to the screen.

Hopefully the new-Cairo is compatible with the older versions of Python that are in the current OS's.

We'll need to test the speed and memory usage -- is there a way to do that outside of Ferret, before doing a full implementation?

[karlmsmith]

Comment by @karlmsmith on 30 Jan 2013 01:17 UTC Working under Ubuntu 12.04 (which uses Cairo 1.10 by default), I am able to get the recording surface to work as expected, with correct looking output in PNG, PDF, and SVG formats. PS output appears to be getting trimmed for some reason; not sure if the problem is my code or theirs. If theirs, I should be able to work around it, or see if it is fixed in 1.12.

Cairo requires a couple of libraries (pixman, for one, which they provide; also libpng and libg2). Building on RHEL is complicated by the fact that there are system versions of these libraries which are too old for Cairo 1.12. So will need to build the new version of these required libraries, and then get the flags correct for telling the Cairo 1.12 build to use these new libraries, not the old system-wide ones.

To check memory consumption, made the plot using "pyferret -unmapped" with the script (pauses allowing memory usage assessment):

use etopo5
load rose
pause
shade /X=75E:75W /Y=85S:85N /lev=C /pal=red_blue_centered rose
pause
frame /file=rose.png
pause
frame /file=rose.pdf
pause

Prior to the plot, python2.7 (pyferret) was using 303 M total memory (virtual under top). After the plot, it increased to 1074 M, indicating this large, detailed plot took 771 M to store the recorded plot. Saving to a PNG briefly increased the size some (to 1081 M from what I saw) and proceeded quickly. Saving to a PDF increase the size to 1826 M at which point it was swapping on my virtual machine, and thus took a lot of elapsed time. The PNG produced was 291 K; the PDF 2.7 M. (So really need to repeat with something smaller.)

CPU times to just create and save the PNG plot (omit the load rose, omit the save to PDF): using recording surface 7-8 s, using draft mode ~7 s, using high-quality mode 45-55 s (most of that spent drawing to screen; was NOT swapping; the save seemed to take just a few seconds).

From this I think I can deduce a few things: (1) the recorded plot will take an expected (reasonable, I guess) chunk of memory; (2) using the recording surface will take a little more CPU time, but not unreasonable; (3) saving to PDF actually does not write directly to file (as suggested by the documentation) but instead appears to write to memory that is then output to the file. This last point means a PDF requires another big chunk of memory for complex plots. (I actually had noticed this in PyFerret prior to making the changes to use the recording surface.)

I think it is still reasonable to go ahead with this approach. Code changes were made so that it will still work as before if compiled against older Cairo without a recording surface. If we decide to not use a recording surface, it is a single line of code that needs to be commented out. Or skipped, if we decide to make it an option in the future.

[karlmsmith]

Comment by @AndrewWittenberg on 30 Jan 2013 02:04 UTC Thanks Karl! This sounds promising. Some other tests to try for the 3 graphics modes, which hopefully won't yield anything unexpected (but you never know). Timing? Memory leaks?

1) Try what you did before with "rose", but use FILL or CONTOUR instead of SHADE.

2) Tons of polygons.

can mode verif; plot/vl=0:100/hl=0:1000/vs 0,0
rep/j=1:100 go polymark poly/ov/nolab/pal=black/lev=(0)/line=2 i[i=1:1000] j+0*i[i=1:1000] 0 star .1

3) Writing lots of frames (e.g. for animations).

yes? use coads_climatology
yes? rep/l=1:1000 (shade sst; frame/file=sst_`l,zw=4`.png)

4) (Re)open window, do a simple shade plot, dump to file; repeat 1000x.

[karlmsmith]

Comment by steven.c.hankin on 30 Jan 2013 16:38 UTC Yikes. Making a PDF of a complicated plot peaked at 1.826 Gig of memory usage! And correspondingly swap-slowed. That's a problem, isn't it? I'm afraid that it means you will need to offer a mode (somehow) in which the writing of PDFs reverts to direct-to-disk. Discussion?

[karlmsmith]

Comment by @karlmsmith on 30 Jan 2013 19:34 UTC Yep, a lot of memory for a complicated PDF. But the added 800 M for creating the PDF even occurs when writing "directly to file". It appears Cairo is processing all the contents in memory before actually writing to file. There just won't be the 700 M of memory for the recorded plot.

Setting aside the PDF issue for the moment, results for some of Andrew's tests with just PNG saves. More memory was given to the Ubuntu virtual machine to reduce swapping.

(1a) etopo5 ROSE contour (contour instead of shade in the above, PNG save only): unmapped 2.7s draft 3.2s high 4.9s (1b) etopo5 ROSE fill (fill instead of shade in the above, PNG save only) unmapped OS kills it after taking up 5.5 G memory while drawing draft 30 s high 250 s with the PNG save taking about 7-10 s (appears to use about 850 M in the viewer, which contains a Qt recording of the drawing) (2) tons-of-polygons - added a frame /file=tonspoly.png at the end unmapped 9.8 s draft 25 s high viewer dies after about 12 min - the viewer holds a recording of the drawing and resulted in no-memory-available in Qt (at about 2G) (3a) lots of frames unmapped 174 s draft 84 s but does not produce plots - graphics not getting "flushed" to the viewer between plots high 573 s in all these the memory requirements are fairly constant and about the same (3b) lots of frames with "go fland" inserted between the shade and frame comamnds: unmapped 358 s draft 515 s (produces plots, but display flashes blank screens between images - similar to high quality behavior, but is draft mode) high 1071 s in all these the memory requirements steadily increase by about 400 M (apparently a memory leak cause by adding "go fland")

Some quirky things going on here. Still trying to make sense of it all. Some from differences in the way Cairo recordings (unmapped) and Qt recordings (high-quality mode) are stored (1b versus 2). But definitely some bugs to be worked out.

[karlmsmith]

Comment by @karlmsmith on 4 Feb 2013 19:00 UTC Realized on Friday that in Cairo one can use a PS or SVG "stream" surface to record a drawing. The intention of this surface is for user-customized write-to-file functions. But the documentation mentions a NULL write function can be specified and the surface used just as a source for another surface; in other words, a recording surface. This ability has been in Cairo since version 1.0.

So I switched to using the SVG stream surface instead of the recording surface. (1) Using Cairo 1.8.8 that comes with RHEL6, this method produces correct PDF, PS, PNG, and SVG output.
(2) Using Cairo 1.2 that comes with RHEL5 this produces embedded raster files instead of true vector files, and crashes on SVG output, so this version in unacceptable.
(3) Using Cairo 1.10.2 that comes with Ubuntu, or Cairo 1.12.12 (latest), the PS output is incomplete (right side is a ragged edge) for landscape, which has a translate/rotate transformation for output.

Testing on RHEL6/Cairo 1.8, memory requirements look similar to using the recording surface for the ROSE shade test. Still need to run the other tests.

But thinking of bundling Cairo 1.8 with PyFerret since it appears 1.8 can be built on RHEL5 using RHEL5-provided libraries. (Checking that now.)

[karlmsmith]

Comment by @karlmsmith on 7 Feb 2013 01:13 UTC Checked-in changes that eliminate the SET GRAPHICS command and instead adds /QUALITY=[ high | draft ], /AAlias, /NoAAlias, and /ENGINE=... qualifiers to SET WINDOW (the /ENGINE primarily for my own use). Also added the -nodisplay command-line option.

Uses Cairo 1.8.8 statically linked in and the SVG stream surface to record the drawing when -nodisplay is given. (Building Cairo 1.8.8 under RHEL5 is simple.) PDF, PS, PNG, and SVG output all looks good. Memory requirement for creating the SHADE ROSE plot are similar to above, and saving to PNG causes a temporary small bump-up in memory use. However, saving the plot as PDF or PS causes no increase in memory use. (Testing under RHEL5 and RHEL6.) Apparently it actually is saving directly to file from this recorded surface. Strangely, saving to SVG does cause the jump in memory like previously. But I am not going to worry about that format.

The -batch option is still available to write directly to file if memory becomes an issue. When -batch is given, the program uses the old behavior (writing PS or PDF directly to the specified file).

Still need to perform all the tests and report the results.

[karlmsmith]

Comment by steven.c.hankin on 7 Feb 2013 15:43 UTC Have you been able to follow the discussions of margins and line thicknesses? If so, those might be items to include in your testing.

[karlmsmith]

Comment by @karlmsmith on 7 Feb 2013 17:06 UTC Line thickness is ticket #2000; plot size/margins is ticket #2007. I doubt there are changes from previous behavior, but will keep an eye out for that.

One thing I did notice with the SHADE ROSE PNG plot is that it is significantly slower than when using the Cario 1.10 recording surface. Need pay attention to where the time is spent, and see if there is any tricks I have missing for speed this up. Also need to see if the time difference is that noticeable for PNG plots of LAS size and complexity.

If necessary, LAS could write directly to a PNG (-batch plotABC123.png) which would be the fastest and require the least memory. Then if a print-quality version is requested, recreate the plot directly as a PDF (-batch plotABC123.pdf).

[karlmsmith]

Comment by steven.c.hankin on 7 Feb 2013 17:34 UTC Testing now ... o The qualifiers AALIAS and NOAALIAS deviate from Ferret's normal command syntax, which is full words that may be truncated. For example MESSAGE/CONTINUE can become me/co instead of msg/ctnu. Either /ANTIALIAS or /ALIAS would be consistent (I prefer /ALIAS, myself) o the command SET WIND/QUAL=high creates an odd initial window, with a white rectangle partially filling a gray background. Not a big deal, but a little off looking. o (These may belong under a different trac ticket) o The initial sizing of the window isn't right. The underlying image is larger than the window, so scroll bars are needed. o The menu item called SCALE has an explanation of "Scale factor for the image". Wouldn't a more conventional term be "ZOOM"? The term "scale" seems ambiguous because there are multiple places in the pipeline where scale factors may be applied. o Wouldn't "VIEW" be the conventional name for this menu? And FILE for the SAVE AS ... (Maybe I am prejudiced by Microsoft applications ...) o The menu items under the "Image" menu do not highlight when the cursor is on them o After SET WIND/QUAL=high I issued a FRAME/FILE=myfile.pdf and got a beautiful PDF. But then I issued a FRAME/FILE=myfile.ps and got a size=0 output file o To test the previous bullet I suspended Ferret with ctrl-Z and restarted after with fg. Then I reissued the GO basemap script and it hung. The output window seems to be gone (??). Something amiss here ... Could not kill the session with ctrl-C. I had to use ctrl-z and kill %1 o re-starting from scratch ... I issued SET WINDOW/QUAL=NEW (my error). I got an error message, but I also got a new window. (The error was not trapped.) o If the first command is SET WIND I get a different-appearing window than if the first command is SET WIND/QUAL=HIGH. Interestingly SET WIND/QUAL=DRAFT looks the same as SET WIND/QUAL=HIGH. It is the default (no /QUAL included) that looks special. o I just compared speeds of GO basemap between /QUAL=high and /QUAL=draft. The HIGH seemed faster. Is there something special about this test case?

[karlmsmith]

Comment by @karlmsmith on 8 Feb 2013 00:09 UTC Issues with window frame versus underlying image size can be monitor resolution dependent. But automatic resizing of the frame to fit the image (if possible) - tickets #1939 and #2002 - should make things look better.

/AALIAS vs /ANTIALIAS vs /ALIAS /NOAALIAS vs /NOANTIALIAS vs /NOALIAS was just trying to make a shorter name. Not sure if something else (such as /SMOOTH and /NOSMOOTH or /ROUGH) might be more (or less) meaningful to users.

Not sure about Scale versus Zoom. Usually see "Zoom in" and "Zoom out", but not sure about a "Zoom factor". Maybe Size (as in SET WINDOW /SIZE=) would be better.

I focused so much on the no-display mode that I did not see all the save-from-displayed problems, which I think is the cause of much of the strange behavior. Hopefully just a bug that crept in from some code change.

Speed of high-quality versus draft is very dependent on image size, drawing complexity, and connection speed. High-quality sends drawing commands to the viewer which then creates the image to display. Draft quality creates the image in memory (Cairo) and sends the image to display.
-- For "go basemap" with the default size, (on my monitor) high quality mode sends 2 "sets" of 1024 drawing commands to render; draft mode sends 283 "blocks" of 8192 bytes of pixel data to display. -- More complex images will increase the number of drawing commands sent and rendered, but not affect the amount of pixel data sent and displayed. (Creating that pixel data will take a little longer.) -- Smaller images will not change the number of drawing commands sent, but will decrease the amount of pixel data sent. -- Slower connections obviously favors less data sent. Note, though, the display of an X window passes a lot of data and thus can be the main speed factor. The high-quality drawing knows what parts of the scene have and have not changed between updates, so it can reduce this chatter when updating a scene. Draft mode only knows to updates the whole scene. (Using a remote desktop keeps all or much of this X chatter on the remote machine, so this may not be such an issue for you.)

So not a clear-cut answer about speed. But a small complex plot will generally be faster in draft (vs high-quality); a large simple plot will generally be faster in high-quality (vs draft). No-display will (should) be fastest.

[karlmsmith]

Comment by steven.c.hankin on 8 Feb 2013 01:12 UTC For the words, words, words discussion see separate email with a couple of screen shot images. There seldom seem to be "right" answers for these questions. (Well, that is, until suddenly there are ... if you know what I mean.)

For the high/draft speed question, maybe a little informal timing is in order. The key point is that the added complexity of DRAFT mode needs to be justified by a significant performance advantage. (And there really is a significant added complexity to the user and the documentation, as well as to your code.) All the more so if DRAFT is actually slower for simple plots. To throw out a target, I'd say we'd want to show (say) at least a factor of 2 performance savings in DRAFT mode for some realistic images.

[karlmsmith]

Comment by @karlmsmith on 9 Feb 2013 02:02 UTC Fixed issue with displayed viewers. The problem was Cairo statically linked in PyFerret. The guts of PyFerret is a actually a shared-object library, so there were two definitions of the Cairo functions in shared-object libraries (libpyferret.so and the system's libcairo.so). Under RHEL6, Qt uses the libcairo.so library, and so this drives it crazy.

Thankfully, under RHEL5, Qt does not use libcairo.so. So I can still statically link in Cairo 1.8.8 under RHEL5, thus getting good -nodisplay outputs and PyQt viewers work fine. Under RHEL6 I just use the libcairo.so that comes with the system, which is version 1.8.8. So again, -nodisplay outputs look good and the PyQt viewers work fine.

For Ubuntu 12.04 (not officially supported), it will also have to use the libcairo.so that comes with the system, which is version 1.10.2. Previous tests showed a ragged right edge of landscape postscript drawings with -nodisplay using Cairo 1.10.2. But the PyQt viewers should be back to working fine. But still needs to be tested to verify.

Also fixed another bug introduced with changes to SET WINDOW that cause it to fail when there are no windows displayed.

Updated the tmap pyferret on dunkel, stout, and flat.

[karlmsmith]

Comment by @karlmsmith on 13 Mar 2013 18:18 UTC Change /AALIAS and /NOAALIAS qualifiers to /ANTIALIAS and /NOANTIALIAS.

Closing out this ticket, since the current implementation (using an Cairo SVG stream surface to record) appears to be working. (RHEL5 needs Cairo-1.8.8 statically linked in; other OS's can use the system Cairo library.)

The window size and resizing issue is another ticket.