search

Alexander-Barth / NCDatasets.jl

Load and create NetCDF files in Julia
MIT License
149 stars 32 forks source link

Int64 attribute is converted to Int32 #258

Open sjdaines opened 5 months ago

sjdaines commented 5 months ago

Describe the bug

A Julia Int64 (or Vector{Int64}) attribute is written to netcdf as Int32

(tested with NetCDF dataset attributes, not tried NetCDF variable attributes)

Edit: see comment below, looks like this is an intentional conversion to deal with a netcdf limitation?

To Reproduce

import NCDatasets
ds = NCDatasets.NCDataset("/tmp/test.nc","c")
v_i64 = 42
@assert typeof(v_i64) === Int64
ds.attrib["testInt64"] = 42
typeof(ds.attrib["testInt64"])

Int32

Expected behavior

Int64 attributes should remain as Int64

Environment

versioninfo()
Julia Version 1.10.3
Commit 0b4590a5507 (2024-04-30 10:59 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × AMD Ryzen 5 7600 6-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS =

In case of an error, please paste the full error message and stack trace.

sjdaines commented 5 months ago

Looking at the code, https://github.com/Alexander-Barth/NCDatasets.jl/blob/354de308b31c27eac66f1de0ca37d0bad15147a7/src/netcdf_c.jl#L566C1-L577C1

this is intentional, so is this a netcdf limitation? (int64 is netcdf4 only, not available with netcdf3 https://docs.unidata.ucar.edu/nug/current/md_types.html)

(if so, perhaps should be added to Corner cases doc?)

Alexander-Barth commented 5 months ago

Yes, in NetCDF 3 and NetCDF 3 (classic format), Int64 are not supported and in the NetCDF CF convention prior to 1.9 they are not allowed.

https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#_data_types https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#_data_types

Since CF 1.9 they are part of the standard. I agree that this should be documented, but I am also open to reconsider the current approach. Since Int64 is the default integer on many platforms, the potential of breaking code is unfortunately quite high.

Does your code, need to store integer larger than typemax(Int32) or it is rather the unexpected type change which is the issue?