Open londumas opened 5 years ago
@londumas can you try without the .gz
? I've encountered problems with large files when gzipping in the past.
@ngbusca, indeed changing this line out = fitsio.FITS('test.fits.gz','rw',clobber=True)
by this line out = fitsio.FITS('test.fits','rw',clobber=True)
fixes everything. Strange!
I think this problem is occurring in cfitsio
, the library that is wrapped by this python package. It might be that this error should be reported to the cfitsio
maintainers. However, in that case you would need to write the example in C.
@esheldon, thank you for the answer. I have no idea on how to write an example in C for this bug. Is there anyway you or someone else could do it?
Was the cfitsio compiled with large file support?
https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/f_user/node13.html
I think this is now the default
https://github.com/esheldon/fitsio/blob/master/cfitsio3430/configure#L4523
but maybe this is broken for gzip
certainly I routinely read and write files larger than 2.1G
Where do we stand on this issue? Does it work for the latest version (1.1.3), for which we have updated cfitsio version to 3.49?
@esheldon, I still have no ways to try on the latest version, since I can not pip install it (issue https://github.com/esheldon/fitsio/issues/298#issuecomment-726088903). But I can say, that as 1.0.5, it is still there
Have you tried using conda to install?
@esheldon, no I have not. If you give me the set of commands, I be happy to try.
If you have an anaconda install it is easy
conda install fitsio
If conda-forge isn't in your channels, do this first
conda config --add channels conda-forge
That will add conda forge to your ~/.condarc.
Note, I usually make sure that conda-forge is first in the list so it gets packages from there first, e.g. make sure that file looks like this
channels:
- conda-forge
- defaults
but that might be a matter of taste
Just tested on fitsio=1.1.3, with conda, and still an issue.
I have the same issue with fitsio
1.1.8 that I installed from conda-forge on Cent OS 7 and macOS (ARM).
I think this is an issue with cfitsio
Is there a flag for building that we didn't set that allows it to run on larger arrays?
fitsio works fine on large arrays, the issue seems to be specifically for .gz files
Running on
fitsio.__version__ = 0.9.11
and onsp.__version__ = 1.2.0
, with the minimalist example bellow. This code shows that fitsio can't write large arrays but does not raise any error. In the case ofmain(coef=2)
, where we save a large array, the output fits file has 6 HDUs instead of 11, and has a corrupted last HDU:OSError: FITSIO status = 107: tried to move past end of file
.Do you have an idea of why fitsio can't write the file? Is it too large? Do you have an idea why fitsio is not raising any writing error?
returns