msys2 cannot find cfitsio version

[simonrw]

Build under msys is broken:

error: failed to run custom build command for `fitsio-sys v0.5.0`

Caused by:
  process didn't exit successfully: `D:\rust\live_stacking\target\debug\build\fitsio-sys-58e5685ff357160f\build-script-build` (exit code: 101)
  --- stdout
  cargo:rerun-if-env-changed=CFITSIO >= 3.37_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=CFITSIO >= 3.37_STATIC
  cargo:rerun-if-env-changed=CFITSIO >= 3.37_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-pc-windows-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_pc_windows_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at 'Unhandled error: `"pkg-config" "--libs" "--cflags" "cfitsio >= 3.37"` did not exit successfully: exit code: 1
  error: could not find system library 'cfitsio >= 3.37' required by the 'fitsio-sys' crate

  --- stderr
  Package dependency requirement 'cfitsio >= 3.37' could not be satisfied.
  Package 'cfitsio' has version '.', required version is '>= 3.37'
  ', E:\Development\vscode\tools\rust\cargo\registry\src\github.com-1ecc6299db9ec823\fitsio-sys-0.5.0\build.rs:32:19       
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Info about my cfitsio package:

$ pacman -Q --info mingw-w64-x86_64-cfitsio
Name            : mingw-w64-x86_64-cfitsio
Version         : 1~3.48-1
Description     : A library of C and Fortran subroutines for reading and writing data files in FITS
                  (Flexible Image Transport System) data format (mingw-w64)
Architecture    : any
URL             : https://heasarc.gsfc.nasa.gov/fitsio/
Licenses        : custom
Groups          : None
Provides        : None
Depends On      : mingw-w64-x86_64-gcc-libs  mingw-w64-x86_64-zlib
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 3.93 MiB
Packager        : CI (msys2-autobuild/16f20bf2/896880892)
Build Date      : Tue Jun 1 22:33:14 2021
Install Date    : Wed Dec 21 01:02:24 2022
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

cc @art-den, originally reported in #194

[simonrw]

@art-den: if you run cargo update (which should pick up fitsio-sys 0.5.1) and re-build, is your build fixed?

[art-den]

New error now

error[E0308]: mismatched types
    --> src\main.rs:28:13
     |
23   |         sys::ffomem(
     |         ----------- arguments to this function are incorrect     
...
28   |             &mut ptr_size as *mut u64,
     |             ^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u32`, found `u64`
     |
     = note: expected raw pointer `*mut u32`
                found raw pointer `*mut u64`
note: function defined here
    --> E:\Development\vscode\tools\rust\cargo\registry\src\github.com-1ecc6299db9ec823\fitsio-sys-0.5.1/src/bindings_64.rs:4435:12
     |
4435 |     pub fn ffomem(
     |            ^^^^^^
          ^^^^^^

[simonrw]

This is in your code right? Windows uses.32-bit pointers even on 64-bit systems, or at least cfitsio does. Can you change the as *mut u64 to as *mut u32 and see if that fixes it?

[art-den]

ffomem signature is

int ffomem
      (fitsfile **fptr, const char *filename, int mode, void **memptr,
       size_t *memsize, size_t deltasize,
       void *(*mem_realloc)(void *p, size_t newsize), int *status)

size_t is 64 bit on 64 machine. So sys::ffomem must take pointer to 64 bit unsigned value in memsize (or ptr_size) argument

[simonrw]

I agree a size_t is the largest integer type for the machine representing the address space, but the function takes a pointer to size_t which for some reason is 32-bit based on the generated bindings.

I'll try to understand this further when I next get time but I can't guarantee when that is right now, sorry

Edit: my misunderstanding, I'm wrong about this, but my next comment is right

[simonrw]

Right: I found out why this is happening. It's not the size of a pointer, it's the size of a size_t. The bindings for cfitsio (in fitsio-sys) define size_t as ::std::os::raw::c_ulong (for some reason...). On Linux this is an 8 bytes, but on Windows it's 4 bytes. Consider this code:

use fitsio_sys::size_t;
fn main() {
    dbg!(std::mem::size_of::<libc::c_int>());
    dbg!(std::mem::size_of::<libc::size_t>());
    dbg!(std::mem::size_of::<libc::c_long>());
    dbg!(std::mem::size_of::<*mut libc::size_t>());
    dbg!(std::mem::size_of::<usize>());
    dbg!(std::mem::size_of::<size_t>());
    dbg!(std::mem::size_of::<*mut size_t>());
}

On windows this prints

[fitsio\examples\sizes.rs:3] std::mem::size_of::<libc::c_int>() = 4
[fitsio\examples\sizes.rs:4] std::mem::size_of::<libc::size_t>() = 8
[fitsio\examples\sizes.rs:5] std::mem::size_of::<libc::c_long>() = 4
[fitsio\examples\sizes.rs:6] std::mem::size_of::<*mut libc::size_t>() = 8
[fitsio\examples\sizes.rs:7] std::mem::size_of::<usize>() = 8
[fitsio\examples\sizes.rs:8] std::mem::size_of::<size_t>() = 4
[fitsio\examples\sizes.rs:9] std::mem::size_of::<*mut size_t>() = 8

On linux this prints

[fitsio/examples/sizes.rs:3] std::mem::size_of::<libc::c_int>() = 4
[fitsio/examples/sizes.rs:4] std::mem::size_of::<libc::size_t>() = 8
[fitsio/examples/sizes.rs:5] std::mem::size_of::<libc::c_long>() = 8
[fitsio/examples/sizes.rs:6] std::mem::size_of::<*mut libc::size_t>() = 8
[fitsio/examples/sizes.rs:7] std::mem::size_of::<usize>() = 8
[fitsio/examples/sizes.rs:8] std::mem::size_of::<size_t>() = 8
[fitsio/examples/sizes.rs:9] std::mem::size_of::<*mut size_t>() = 8

The differences being

$ diff /tmp/windows.txt /tmp/linux.txt
3c3
< std::mem::size_of::<libc::c_long>() = 4
---
> std::mem::size_of::<libc::c_long>() = 8
6c6
< std::mem::size_of::<size_t>() = 4
---
> std::mem::size_of::<size_t>() = 8

This explains why you are experiencing a difference on windows. You can probably cast the num_bytes variable in the example to fitsio::sys::size_t and cast the pointer to *mut _ and this should fix the example, except I'm now getting link errors saying "undefined reference to 'stderr'" which is new.

[art-den]

Yes, if to change your example to

    let (bytes, mut ptr_size) = {
        let filename = "../testdata/full_example.fits";
        let mut f = std::fs::File::open(filename).unwrap();
        let mut bytes = Vec::new();
        let num_bytes = f.read_to_end(&mut bytes).unwrap();

        (bytes, num_bytes as sys::size_t) // <---here. Not good if FITS file is larger than 4 GB
    };

    let mut ptr = bytes.as_ptr();

    let mut fptr = std::ptr::null_mut();
    let mut status = 0;

    let c_filename = std::ffi::CString::new("full_example.fits").unwrap();
    unsafe {
        sys::ffomem(
            &mut fptr as *mut *mut _,
            c_filename.as_ptr(),
            sys::READONLY as _,
            &mut ptr as *const _ as *mut *mut libc::c_void,
            &mut ptr_size as *mut sys::size_t, // <---here
            0,
            None,
            &mut status,
        );
    }

it compiles but I see link error too

[simonrw]

$ cargo run -p fitsio --example reading_from_memory
   Compiling fitsio-sys v0.5.1 (C:\Users\Simon Walker\Desktop\rUSt-fitsio\fitsio-sys)
warning: No version specifier available for pkg-config on windows, so the version of cfitsio used when compiling this program is unspecified
   Compiling fitsio v0.21.0 (C:\Users\Simon Walker\Desktop\rUSt-fitsio\fitsio)
error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1
  |
  = note: "x86_64-w64-mingw32-gcc" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-m64" "-Wl,--high-entropy-va" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.1arhofn3an9tayr0.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.1chs3pazbhtthhhd.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.1kk0nl9sjhzzp6iw.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.1oa4elhzh2hm4jq8.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.1ttpbq4dj1av1760.rcgu.o" 
"C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.251mjeefmc9vy3h5.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.37dch5urv9qtr7xh.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.3im2lhe31rf39yif.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.3rkgpdp0k9dqwmp4.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.49f93y3bl070nxtu.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.4cwefbr618zvdd90.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.4df583sa019fvxme.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.4negaxce5rn0pd63.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.50q1u0xam46oty04.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.6xtaqq3jhxrerng.rcgu.o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.4jt2olmo4u6rk379.rcgu.o" "-L" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\deps" "-L" "C:\\Users\\Simon Walker\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\winapi-x86_64-pc-windows-gnu-0.4.0\\lib" "-L" "C:/tools/msys64/mingw64/lib" "-L" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-Wl,-Bstatic" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\deps\\libfitsio-ea98ba45a951eaef.rlib" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\deps\\liblibc-5bf754604d4a138b.rlib" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\deps\\libfitsio_sys-a83aa729b73491fa.rlib" "-Wl,--start-group" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd-0ef6a1f4ede94b56.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libpanic_unwind-7a832a7c7800efa8.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libobject-6eb666a7c5a4af6a.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libmemchr-1af0bf8a350915e8.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libaddr2line-d5a17343e1b65ac7.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libgimli-4ec9e70d4a72f3e3.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd_detect-0d849b46e87570a1.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\librustc_demangle-2d5a548828b0c630.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libhashbrown-0295737ee98d507a.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\librustc_std_workspace_alloc-a983b7030a28478d.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libunwind-a8ffafefa8b105ed.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcfg_if-19a066a82c294742.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblibc-9b54bf68af38be86.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\liballoc-c313ef63115ce558.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\librustc_std_workspace_core-d7812c91b2f09baf.rlib" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcore-2fd0cc2a5fa24e6b.rlib" "-Wl,--end-group" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcompiler_builtins-499045c7fe445a30.rlib" "-Wl,-Bdynamic" "-lcfitsio" "-lkernel32" "-lws2_32" "-lbcrypt" "-ladvapi32" "-luserenv" "-lkernel32" "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-luser32" "-lkernel32" "-Wl,--nxcompat" "-L" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-o" "C:\\Users\\Simon Walker\\Desktop\\rUSt-fitsio\\target\\debug\\examples\\reading_from_memory-f34cbb1e12c44281.exe" "-Wl,--gc-sections" "-no-pie" "-nodefaultlibs" "C:\\Users\\Simon Walker\\.rustup\\toolchains\\stable-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsend.o"
  = note: C:/tools/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/11.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\Users\Simon Walker\Desktop\rUSt-fitsio\target\debug\examples\reading_from_memory-f34cbb1e12c44281.4negaxce5rn0pd63.rcgu.o:4negaxce5rn0pd63:(.rdata$.refptr.stderr[.refptr.stderr]+0x0): undefined reference to `stderr'
          collect2.exe: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `fitsio` due to previous error

[simonrw]

In the short term I can get around the link error by calling check_status rather using the cfitsio error reporting function. See my branch. ~Unfortunately the example still does not run, but at least it compiles!~ IT RUNS!

To be clear, I'm using this code:

// `fitsio` does not currently support opening files from memory, `cfitsio` _does_. This means we
// can use `Fitsfile::from_raw` to load a `FitsFile` from a file that was opened via
// `fits_open_memfile` in `cfitsio`.

use fitsio::errors::check_status;
use fitsio::{sys, FileOpenMode, FitsFile};
use std::io::Read;

fn main() {
    // read the bytes into memory and return a pointer and length to the file

    let (bytes, mut ptr_size) = {
        let filename = "./testdata/full_example.fits";
        let mut f = std::fs::File::open(filename).unwrap();
        let mut bytes = Vec::new();
        let num_bytes = f.read_to_end(&mut bytes).unwrap();

        (bytes, num_bytes as u32)
    };

    let mut ptr = bytes.as_ptr();

    // now we have a pointer to the data, let's open this in `fitsio_sys`
    let mut fptr = std::ptr::null_mut();
    let mut status = 0;

    let c_filename = std::ffi::CString::new("full_example.fits").unwrap();
    unsafe {
        sys::ffomem(
            &mut fptr as *mut *mut _,
            c_filename.as_ptr(),
            sys::READONLY as _,
            &mut ptr as *const _ as *mut *mut libc::c_void,
            &mut ptr_size as *mut u32,
            0,
            None,
            &mut status,
        );
    }

    check_status(status).unwrap();

    let mut f = unsafe { FitsFile::from_raw(fptr, FileOpenMode::READONLY) }.unwrap();
    f.pretty_print().expect("pretty printing fits file");
}

[simonrw]

Er... it's inconsistent. I can run the code multiple times and get different answers :facepalm: That's going to be really hard to debug on windows (for me anyway)!

[simonrw]

Another option for you could be to use the bindgen feature of fitsio - add this to your Cargo.toml:

fitsio = { version = "*", features = ["bindgen"], default-features = false }

This requires the additional dependency clang (on msys2: mingw64/mingw-w64-x86_64-clang). With this the example runs just fine, since it compiles the rust bindings at compile time, so it should always match your system.

[simonrw]

Right. I've made some updates to the crate. I've mostly updated the bindings, which makes more sense for msys2. I can run the tests for fitsio including the example above (moved to fitsio/tests/test_reading_from_memory.rs. I sidestep the stderr issue by using the fitsio function check_status. I've also added a test stage for msys2 that runs the basic tests (no additional features).

For the time being you have to use the main branch to pick up these changes:

# in Cargo.toml
fitsio = { git = "https://github.com/simonrw/rust-fitsio" }