"Not a legal pipe record b''" from simple copy() commands

[jehturner]

I have encountered this in tests that copy FITS files (using 8ea61699 under Python 3.7), but the following reproduces it without requiring any additional data:

>>> from pyraf import iraf
>>> iraf.copy('dev$pix.imh', './')

PANIC in `/rtfproc/anaconda3_2019.10/envs/pyraf3test/iraf/bin.linux/x_system.e': Out of memory
Killing IRAF task `copy'
Traceback (most recent call last):
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 880, in _run
    irafexecute.IrafExecute(self, iraf.getVarDict(), **redirKW)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 371, in IrafExecute
    raise exc
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 348, in IrafExecute
    irafprocess.run(task, pstdin=stdin, pstdout=stdout, pstderr=stderr)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 512, in run
    self.slave()
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 636, in slave
    self.msg = self.readString()
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 578, in readString
    return Iraf2AscString(self.read())
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 607, in read
    str(header[0:2]))
pyraf.irafexecute.IrafProcessError: Not a legal IRAF pipe record: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/rtfproc/rtftests/gemini_iraf/gmosspec2/iraf/test/test3.py", line 8, in <module>
    iraf.copy('dev$pix.imh', './')
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 836, in __call__
    return self.run(*args, **kw)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 375, in run
    self._run(redirKW, specialKW)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 883, in _run
    str(value))
stsci.tools.irafglobals.IrafError: Error running IRAF task copy
Not a legal IRAF pipe record: b''
>>>

[olebole]

The relevant error message is the first line

PANIC in `/rtfproc/anaconda3_2019.10/envs/pyraf3test/iraf/bin.linux/x_system.e': Out of memory

I guess that you are using an old 32 bit i386 version of IRAF, right? Then you are hit by this problem: iraf-v216#61. This is solved in the first community release of IRAF; you should consider to switch here as well :grin: (probably external packages would require a recompilation, since they are statically linked). Since this was resolved already some years ago, I removed the workaround in PyRAF in 481f20d4ef2319b356a424be6b6b5166a561d6b5, which was probably too early. I will revert this one.

[jehturner]

Thanks; I'm glad I didn't have to (re-)diagnose that one.

Yes, Gemini decided maybe 10 years ago not to switch to 64-bit IRAF, largely because STScI decided not to port STSDAS & TABLES, which we depend on quite heavily (there was an NOAO port of just TABLES in IRAF, but it wasn't working for us as-is). We really needed to keep our limited effort focused on Python development, rather than constantly getting sucked back into IRAF, which had slowed us down a lot. This issue has come up again recently, of course, thanks to Apple, but we still think that porting Gemini IRAF to 64 bits, plus all our dependencies, plus re-testing and verifying everything scientifically (with rather a brittle regression testing framework) is likely more than we can manage, so we currently plan to limp along using qemu for those that have to work on M1. We've been making very good progress on our newer DRAGONS package lately, but it's going to take a number of years of focusing fully on it to finish supporting all our instruments. After that it may, ironically, be easier to converge with community-iraf for more general use.

Thanks for reverting the removal of the workaround, which I expect we'll need for some years to come. It seems a good idea not to assume that pyraf will be used specifically with community-iraf, so that we can at least all adopt the same pyraf code base?

When I get back to this, I'll confirm that these errors are gone with the latest version.

[olebole]

I'd be quite curious about other problems before I release a new PyRAF problems, so I will wait.

BTW, the bug fix (iraf-v216#62) is mainly for 32-bit Linux, as the 64-bit variants are not affected by iraf-v216#61, and the community IRAF works on 32 bit as well.

[jehturner]

:+1: I'll try not to take months to send more feedback, especially since I suspect we might be able to rattle through everything pretty quickly with your insight. Have you encountered errors such as "Error in parameter 'apscat2.niterate' for task apscatter\n'Key apscat2.niterate not found'" (and variants thereof, involving other tasks) before? I haven't looked into it yet, so it might be something straightforward, such as parameter names being abbreviated.

Yes, it could be more feasible to move to 32-bit community IRAF, though we'd still have to go through and make sure all our fixes are in your code base (not a bad thing in itself) and re-test everything and then it wouldn't actually solve our main problem, so it's unlikely to happen soon. I recall you picking this up just as my effort on core IRAF for Ureka/Astroconda dried up, which was bad timing :slightly_frowning_face:.

[olebole]

I didn't see such errors before. If you have it reproducible, I could look into it.

[olebole]

I am merging the fixes so far (#98) into main so that you can take main for another check.

[jehturner]

Although this problem was resolved on Linux, it turns out that it still happens with AstroConda IRAF on macosx. It can be reproduced with the original example above when there's no login.cl, but when login.cl exists, pyraf now fails to start at all (when running type; I think the manifestation might be slightly different due to character encoding fixes?).