WCS incompatible in graphics when 32 and 64 bits are mixed

[olebole]

Taken from iraf.net#1468030:

When 32 and 64 bit executables are mixed, WCS coordinate conversion in interactive grapics is wrong.

@iraf writes there:

A graphics WCS structure was changed in the 64-bit port and this is (...) due to an incompatibility between the 32-bit IGI and the 64-bit CL binary which does the graphics.

The problem is that the bit width of the task executable has to match the bit width of the CL executable. For example, STSDAS is only available for 32 bit, so stsdas.graphics.stplot.igi needs a 32 bit CL interpreter. With a 32 bit CL interpreter, the 64 bit tasks (f.e. images.imcoords.ccmap) however will then have the same problem.

[olebole]

Just as a reference for the workaround in PyRAF, https://aeon.stsci.edu/ssb/trac/pyraf/ticket/173 :

PyRAF does some semi-intelligent guessing (irafgwcs.py) to figure out what the LEN_WCS value being used (compiled into) the IRAF task is - for the current IRAF task. It even has some logic to switch to 64-bit mode when necessary.

This works with IRAF 2.14 graphics tasks, as well as STScI tasks built with 2.14. These are 32-bit.

This works with IRAF 2.15 graphics tasks, as well as STScI tasks built with 2.15 (assumed case, since at current none have been built that way). Both 32-bit and 64-bit.

This does NOT work with old STScI graphics tasks (e.g igi, sgraph, histogram), which were built with IRAF 2.14, but are packaged with IRAF 2.15.*. This was a case which was assumed would not occur. For various (good) reasons, it is occuring.

PyRAF should be given the smarts to switch back and forth between all the various LEN_WCS possibilities that come from old 32-bit tasks, new 32-bit tasks, and new 64-bit tasks (all different values).

The first time this was addressed was in ticket #156.

Solved in [1616].

[cmccully]

@olebole We recently came across this issue in our legacy code that relies on iraf and pyraf. We are getting the same error that is mentioned here: https://iraf.net/forum/viewtopic.php?showtopic=1469330

This appears to be after ticket 173. Also, the stsci ticket links appear to be dead.

Any ideas on what that 64 bit size should be? Or why the daophot_x.e is not picking up the 64 bit size assuming that is correct? I don't mind making a hotfix in our local version of pyraf if need be due to stsci dropping support.

We unfortunately can't switch to the 32 bit binaries because our host system is 64 bits and we have surpassed the inode numbers that can be represented by 32 bits. That was miserable to track down.

Any help is appreciated.

[olebole]

@cmccully Unfortunately, I have no idea. It may be helpful to look around in the spacetelescope/pyraf repository; this seems to have kept the commit history and look what the changeset 1616 (mentioned above) actually did. But I didn't look deeply into PyRAF, so I can't help here (although I don't mind to host a PyRAF repository in the @iraf-community organization to collect fixes like this one, see spacetelescope/pyraf#60). @perrygreenfield, just as a question: are the old Trac issues from https://aeon.stsci.edu/ssb/trac/pyraf/ somewhere available, or could you send me a copy so that I could put them onto the iraf-community web page for reference? I think they may be helpful in future.

[cmccully]

@cdsontag I think you are the one who fixed the pyraf graphics issues based on the commit history. Any ideas? Or maybe @pllim?

[pllim]

AFAIK, we never shipped PyRAF with 64-bit libraries because IRAF only worked in 32-bit. See https://astroconda.readthedocs.io/en/latest/faq.html#why-is-iraf-32-bit-instead-of-64-bit

[perrygreenfield]

Not quite right as I recall. STSDAS and TABLES were never converted to 64-bit IRAF for various reasons, and so for us, though I don't think that prevented PyRAF from working with 64-bit IRAF executables, but it's been a long time...

I'll look into whether the trac issues are available.

[pllim]

Trac issues are trapped in a purgatory that is SQL DB tarball on internal disk somewhere. I'll see if I remember what @jhunkeler thought me on how to access it.

[cmccully]

So pyraf does work with a majority of 64 bit iraf. It is specifically having some issues using 64 bit graphics for some (but not all?) tasks as reported here: https://iraf.net/forum/viewtopic.php?showtopic=1469330. I don't even know where to begin to debug this. If someone could point me in the right direction, I might be able to look at it.

[pllim]

Perhaps it works with 64-bit, but it was never tested as far as STScI PyRAF delivery was concerned because we were locked to 32-bit for one reason or another. I'll post if I can dig up the old tickets but don't hold your breath.

[pllim]

UPDATE: Turns out I don't have that info at hand. I extracted stsci_python for another thingy and it doesn't have pyraf Trac. Waiting to hear back from Joe H. I think he's the only one who knows where everything is.

[cdsontag]

My beloved PyRAF FAQ would have answers to a lot of this, but it seems to no longer be on our website. Yes, PyRAF should work with 64-bit, though I think we (ST) decided some of our binary IRAF-ish packages would never go to 64-bit. So without those binaries, yes, the whole deal could safely run 64-bit.

[pllim]

@cdsontag , they moved PyRAF FAQ to ServiceNow but that link is broken too. Investigating.

[cmccully]

@cdsontag any ideas on that iraf ticket I posed above? Is it possible the daophot package binaries never got updated to have the correct wcs structure for iraf 2.15. @olebole maybe you have an idea?

[cdsontag]

@cmccully - to be clear, are you trying to run 64-bit PyRAF/Python with some 32-bit IRAF task binaries? Even if we got that working in some situations, I'd say that's probably not a stable configuration and not to be relied upon. Or are you trying to rebuild a task as 64-bit?

[olebole]

@cmccully iraf-community only distributes sources to be self-compiled. So, if you took it from us (including f.e. Debian or Ubuntu binaries), you got everything correctly. So, can you confirm that you indeed compiled it yourself?

I have however no idea whether daophot may have a local version of the wcs structure which is not updated (not so uncommon in the IRAF sources). It may be worth to look into its sources for that.

[pllim]

And just in case you can't fix this in a reasonable time frame, why not use Python? 😬

[cmccully]

@pllim That is absolutely my first choice. This is meant to be a stop gap for some crazy legacy code until I can rewrite the whole thing.

@cdsontag I am trying to run 64 bit iraf with 64 bit pyraf, but I am getting an error 851 as described in the link above.

@olebole as you say, I am compiling the source myself. You can find the Dockerfile here: https://github.com/LCOGT/lcogtsnpipe/blob/feature/docker-compose-db/Dockerfile

[cdsontag]

@cmccully - ok, if all of your binary tasks are 64-bit, then (as far as PyRAF knows) you should be good. This can be easily tested by trying out other 64-bit graphics tasks and making sure the key/cursor commands work as expected.

With everything 64-bit and you still see ERROR (851, "Cannot restore graphics world coordinate systems"), then there is obviously a bug, but I can't say whether the fix is to the binary or to PyRAF. However, at this point I can suggest that you may now be in the realm of worth-rewriting-my-task-in-python. Sorry to not be more helpful.

Also, if everything works as you need from the CL and you need to use the binary, you can possibly always just wrap your task caller in some Python which shells out a CL command.

[pllim]

FWIW, the FAQ is back up but now it is a PDF attachment to this knowledge base article, in particular this attachment.

[cmccully]

Ok. Thanks everyone for your help. I'll see if we can just rewrite the part we need for a stop gap. What a mess...