search

JuliaAstro / FITSIO.jl

Flexible Image Transport System (FITS) file support for Julia
http://juliaastro.org/FITSIO.jl/
MIT License
55 stars 29 forks source link

error in loading a fits table as a DataFrame #182

Open Illuminista opened 2 years ago

Illuminista commented 2 years ago

I am a rookie but it seems that something is to working as expected

from the manual: 

julia> f[2]
File: file.fits
HDU: 2
Type: Table
Rows: 20
Columns: Name  Size  Type    TFORM  
         .... ....
Table HDUs implement the [Tables.jl] interface, so you can load them into other table types, like [DataFrames].
julia> df = DataFrame(f[2])

my case:

julia> f[2]
File:XXX.fits
HDU: 2
Type: Table
Rows: 24
Columns: Name               Size  Type     TFORM  
                    col1                         Float64  D   
.... ....

julia> df=DataFrame(f[2])
ArgumentError: no default `Tables.columns` implementation for type: TableHDU

Stacktrace:
 [1] columns
   @ ~/.julia/packages/Tables/M26tI/src/fallbacks.jl:264 [inlined]
 [2] DataFrame(x::TableHDU; copycols::Nothing)
   @ DataFrames ~/.julia/packages/DataFrames/BM4OQ/src/other/tables.jl:58
 [3] DataFrame(x::TableHDU)
   @ DataFrames ~/.julia/packages/DataFrames/BM4OQ/src/other/tables.jl:49
 [4] top-level scope
   @ In[17]:1
 [5] eval
   @ ./boot.jl:373 [inlined]
 [6] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
   @ Base ./loading.jl:1196
mileslucas commented 2 years ago

Hi, thanks for the report. Can you please list your version of FITSIO.jl in this environment?

julia>] st FITSIO
mileslucas commented 2 years ago

I just gave this a try and did not run into any issues-

(jl_K0Xnrq) pkg> st
Status `/private/var/folders/q6/j_0w4_mx13jg_7mw9__902f40000gn/T/jl_K0Xnrq/Project.toml`
  [a93c6f00] DataFrames v1.3.2
  [525bcba6] FITSIO v0.16.12

julia> using DataFrames, FITSIO

julia> hdus = FITS("https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits")
File: https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits
Mode: "r" (read-only)
HDUs: Num  Name                   Type
      1                           Image
      2    u5780205r_cvt.c0h.tab  ASCIITable

julia> df = DataFrame(hdus[2]);

julia> first(df, 3)
3×49 DataFrame
 Row │ CRVAL1   CRVAL2   CRPIX1   CRPIX2   CD1_1        CD1_2        CD2_1        CD2_2        DATAMI ⋯
     │ Float64  Float64  Float64  Float64  Float64      Float64      Float64      Float64      Float6 ⋯
─────┼─────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ 182.631  39.3963    420.0    424.5  -1.06704e-6  -1.25958e-5  -1.26016e-5   1.06655e-6  -73.19 ⋯
   2 │ 182.626  39.4121    423.5    414.0  -2.75605e-5   2.08221e-6   2.08021e-6   2.75871e-5  -19.95
   3 │ 182.652  39.4109    436.0    424.5   2.23913e-6   2.75426e-5   2.75684e-5  -2.23704e-6  -58.53
                                                                                     41 columns omitted
Illuminista commented 2 years ago

On 25 Feb 2022, at 21:56, Miles Lucas @.***> wrote:

Hi, thanks for the report. Can you please list your version of FITSIO.jl in this environment?

julia>] st FITSIO

Hello this is it

@.***) pkg> st FITSIO Status ~/.julia/environments/v1.7/Project.toml [525bcba6] FITSIO v0.15.0

I see from a following mail that there is a newer version of FITSIO, v0.16.0

I’ll try to update and recheck

Thanks

R.

— Reply to this email directly, view it on GitHub https://github.com/JuliaAstro/FITSIO.jl/issues/182#issuecomment-1051260202, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXV5EP6EMHB7WRK2HGDAIW3U47UGBANCNFSM5N5T7LAA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.

Illuminista commented 2 years ago

Hello, after updating everything unfortunately FITSIO is still at v0.15.0…

So I get the same error as before. Can you have the general registry to point to the latest version?

Moreover, I get an error even trying to replicate your example (?!). I acclude the output below.

Cheers

R.

using DataFrames, FITSIO ; hdus = FITS("https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits")

While processing file https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits: could not open the named file

Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] fits_assert_ok @ ~/.julia/packages/FITSIO/4lCUb/src/libcfitsio.jl:206 [inlined] [3] fits_open_file(filename::String, mode::Int64) @ FITSIO.Libcfitsio ~/.julia/packages/FITSIO/4lCUb/src/libcfitsio.jl:281 [4] FITS(filename::String, mode::String) (repeats 2 times) @ FITSIO ~/.julia/packages/FITSIO/4lCUb/src/FITSIO.jl:111 [5] top-level scope @ In[18]:1 [6] eval @ ./boot.jl:373 [inlined] [7] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String) @ Base ./loading.jl:1196

On 26 Feb 2022, at 01:17, Miles Lucas @.***> wrote:

I just gave this a try and did not run into any issues-

(jl_K0Xnrq) pkg> st Status /private/var/folders/q6/j_0w4_mx13jg_7mw9__902f40000gn/T/jl_K0Xnrq/Project.toml [a93c6f00] DataFrames v1.3.2 [525bcba6] FITSIO v0.16.12

julia> using DataFrames, FITSIO

julia> hdus = FITS("https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits") File: https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits Mode: "r" (read-only) HDUs: Num Name Type 1 Image 2 u5780205r_cvt.c0h.tab ASCIITable

julia> df = DataFrame(hdus[2]);

julia> first(df, 3) 3×49 DataFrame Row │ CRVAL1 CRVAL2 CRPIX1 CRPIX2 CD1_1 CD1_2 CD2_1 CD2_2 DATAMI ⋯ │ Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float6 ⋯ ─────┼───────────────────────────────────────────────────────────────────────────────────────────────── 1 │ 182.631 39.3963 420.0 424.5 -1.06704e-6 -1.25958e-5 -1.26016e-5 1.06655e-6 -73.19 ⋯ 2 │ 182.626 39.4121 423.5 414.0 -2.75605e-5 2.08221e-6 2.08021e-6 2.75871e-5 -19.95 3 │ 182.652 39.4109 436.0 424.5 2.23913e-6 2.75426e-5 2.75684e-5 -2.23704e-6 -58.53 41 columns omitted — Reply to this email directly, view it on GitHub https://github.com/JuliaAstro/FITSIO.jl/issues/182#issuecomment-1051390435, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXV5EP5QHWK5JNSGHUOHHGLU5AL3LANCNFSM5N5T7LAA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.

mileslucas commented 2 years ago

Okay, let's start with the version issue.

Can you try updating your whole environment? This should upgrade your registry.

From the REPL-mode:

julia>] up

or

julia> using Pkg
julia> Pkg.update()

Please check if this upgrades your FITSIO version. If it does not, but you see the "updating registry" message, that means some other dependency is manually restricting your FITSIO.jl version. If you do not see a message that the registry is updating, please manually try

julia>] registry up
Illuminista commented 2 years ago

OK, once removed the culprit DustExtinction package, I managed to update FITSIO and now it works, see below. Manny thanks!

R.

On 1 Mar 2022, at 05:38, Miles Lucas @.***> wrote:

Okay, let's start with the version issue.

Can you try updating your whole environment? This should upgrade your registry.

From the REPL-mode:

julia>] up or

julia> using Pkg julia> Pkg.update() @.***) pkg> up Updating registry at ~/.julia/registries/General.toml Updating git-repo https://github.com/heliosdrm/GRutils.jl Updating git-repo https://github.com/emmt/EasyFITS.jl Installed MKL_jll ───── v2022.0.0+0 Installed FFTW ──────── v1.4.6 Installed Optim ─────── v1.6.2 Installed StatsModels ─ v0.6.29 Updating ~/.julia/environments/v1.7/Project.toml [7a1cc6ca] ↑ FFTW v1.4.5 ⇒ v1.4.6 [429524aa] ↑ Optim v1.6.1 ⇒ v1.6.2 Updating ~/.julia/environments/v1.7/Manifest.toml [7a1cc6ca] ↑ FFTW v1.4.5 ⇒ v1.4.6 [429524aa] ↑ Optim v1.6.1 ⇒ v1.6.2 [3eaba693] ↑ StatsModels v0.6.28 ⇒ v0.6.29 [856f044c] ↑ MKL_jll v2021.1.1+2 ⇒ v2022.0.0+0 Precompiling project... 31 dependencies successfully precompiled in 77 seconds (472 already precompiled)

Please check if this upgrades your FITSIO version.

This seems not to be the case…

So I did

@.**) pkg> add FITSIO#master Cloning git-repo https://github.com/JuliaAstro/FITSIO.jl.git Updating git-repo https://github.com/JuliaAstro/FITSIO.jl.git Resolving package versions... ERROR: Unsatisfiable requirements detected for package FITSIO [525bcba6]: FITSIO [525bcba6] log: ├─possible versions are: 0.16.12 or uninstalled ├─FITSIO [525bcba6] is fixed to version 0.16.12 └─found to have no compatible versions left with DustExtinction [fb44c06c] └─DustExtinction [fb44c06c] log: ├─possible versions are: 0.3.0-0.10.1 or uninstalled └─restricted to versions by an explicit requirement, leaving only versions 0.3.0-0.10.1

@.***) pkg> rm DustExtinction Updating ~/.julia/environments/v1.7/Project.toml [fb44c06c] - DustExtinction v0.10.1 Updating ~/.julia/environments/v1.7/Manifest.toml [124859b0] - DataDeps v0.7.7 [fb44c06c] - DustExtinction v0.10.1

@.***) pkg> update Updating registry at ~/.julia/registries/General.toml Updating git-repo https://github.com/heliosdrm/GRutils.jl Updating git-repo https://github.com/emmt/EasyFITS.jl Installed Dierckx ───── v0.5.2 Installed Dierckx_jll ─ v0.1.0+0 Downloaded artifact: Dierckx Updating ~/.julia/environments/v1.7/Project.toml [525bcba6] ↑ FITSIO v0.15.0 ⇒ v0.16.12 Updating ~/.julia/environments/v1.7/Manifest.toml [39dd38d3] ↑ Dierckx v0.4.2 ⇒ v0.5.2 [525bcba6] ↑ FITSIO v0.15.0 ⇒ v0.16.12 [cd4c43a9] + Dierckx_jll v0.1.0+0 Precompiling project... 5 dependencies successfully precompiled in 15 seconds (497 already precompiled)

@.***) pkg> status FITSIO Status ~/.julia/environments/v1.7/Project.toml [525bcba6] FITSIO v0.16.12

Pkg; Pkg.status("FITSIO") ; Pkg.update("FITSIO") Status ~/.julia/environments/v1.7/Project.toml [525bcba6] FITSIO v0.16.12 Updating registry at ~/.julia/registries/General.toml No Changes to ~/.julia/environments/v1.7/Project.toml No Changes to ~/.julia/environments/v1.7/Manifest.toml

using DataFrames, FITSIO ; hdus = FITS("https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits")

File: https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits https://fits.gsfc.nasa.gov/samples/WFPC2u5780205r_c0fx.fits Mode: "r" (read-only) HDUs: Num Name Type
1 Image
2 u5780205r_cvt.c0h.tab ASCIITable

If it does not, but you see the "updating registry" message, that means some other dependency is manually restricting your FITSIO.jl version. If you do not see a message that the registry is updating, please manually try

julia>] registry up — Reply to this email directly, view it on GitHub https://github.com/JuliaAstro/FITSIO.jl/issues/182#issuecomment-1055009098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXV5EPZBV3TA7HEVMXHJQ5DU5RESVANCNFSM5N5T7LAA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.

mileslucas commented 2 years ago

Okay, I think something is wrong with your registry, because the latest version of DustExtinction.jl (v0.11) has compatibility for FITSIO.jl v0.16.

The easiest fix is to remove the registry and let Julia fix it itself, can you give this a shot?

$ cd ~/.julia/registries
$ rm -r General*

and then from Julia (in the default, v1.7 environment)

julia>] up
giordano commented 2 years ago

Or from within julia:

]registry rm General
registry add General
Illuminista commented 2 years ago

Hi, there is something strange indeed.

According to the suggestion I removed the registry, added, it, updated, then added DustExtinction. that I had removed previously.

However in added DustExtinction FITSIO was downgraded to 0.15 again and DustExtinction shows v0.10, not v0.11 that you quote.

Logs are below.

I think I’ll erase julia from my mac laptop and reinstall from scratch, hoping the problems will not appear again.

Thanks

R.

On 2 Mar 2022, at 01:07, Miles Lucas @.***> wrote:

Okay, I think something is wrong with your registry, because the latest version of DustExtinction.jl (v0.11) has compatibility for FITSIO.jl v0.16.

The easiest fix is to remove the registry and let Julia fix it itself, can you give this a shot?

$ cd ~/.julia/registries $ rm -r General* and then from Julia

julia>] up — Reply to this email directly, view it on GitHub https://github.com/JuliaAstro/FITSIO.jl/issues/182#issuecomment-1055993379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXV5EP6BVYUYZCB6FAYOHNTU52WNHANCNFSM5N5T7LAA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.

@.***) pkg> status FITSIO Status ~/.julia/environments/v1.7/Project.toml [525bcba6] FITSIO v0.16.12

@.***) pkg> registry rm General Removing registry General from ~/.julia/registries/General.toml

@.***) pkg> registry add General

@.***) pkg> up Updating registry at ~/.julia/registries/General.toml Updating git-repo https://github.com/heliosdrm/GRutils.jl Updating git-repo https://github.com/emmt/EasyFITS.jl Installed BitTwiddlingConvenienceFunctions ─ v0.1.3 Installed ExponentialUtilities ───────────── v1.12.1 Installed Plots ──────────────────────────── v1.26.0 Installed Krylov ─────────────────────────── v0.7.13 Updating ~/.julia/environments/v1.7/Project.toml [91a5bcdd] ↑ Plots v1.25.11 ⇒ v1.26.0 Updating ~/.julia/environments/v1.7/Manifest.toml [62783981] ↑ BitTwiddlingConvenienceFunctions v0.1.2 ⇒ v0.1.3 [d4d017d3] ↑ ExponentialUtilities v1.11.0 ⇒ v1.12.1 [ba0b0d4f] ↑ Krylov v0.7.12 ⇒ v0.7.13 [91a5bcdd] ↑ Plots v1.25.11 ⇒ v1.26.0 Precompiling project... 28 dependencies successfully precompiled in 170 seconds (474 already precompiled)

@.***) pkg> add DustExtinction Resolving package versions... Updating ~/.julia/environments/v1.7/Project.toml [fb44c06c] + DustExtinction v0.10.1 [525bcba6] ↓ FITSIO v0.16.12 ⇒ v0.15.0 Updating ~/.julia/environments/v1.7/Manifest.toml [124859b0] + DataDeps v0.7.7 [39dd38d3] ↓ Dierckx v0.5.2 ⇒ v0.4.2 [fb44c06c] + DustExtinction v0.10.1 [525bcba6] ↓ FITSIO v0.16.12 ⇒ v0.15.0 [cd4c43a9] - Dierckx_jll v0.1.0+0 Precompiling project... 5 dependencies successfully precompiled in 15 seconds (498 already precompiled)

@.***) pkg> status FITSIO Status ~/.julia/environments/v1.7/Project.toml [525bcba6] FITSIO v0.15.0

@.***) pkg> status DustExtinction Status ~/.julia/environments/v1.7/Project.toml [fb44c06c] DustExtinction v0.10.1