CDMS issues a warning message when working with complex numbers

chaosphere2112 commented 7 years ago

So, this probably doesn't come up too frequently, but there appears to be a slight type conversion issue with regards to fill_value when fill_value is complex.

>> import numpy, cdms2
>> arr = numpy.ma.MaskedArray(range(4), fill_value=1j, dtype="complex128")
>> cdms2.tvariable.TransientVariable(arr)
/Users/fries2/anaconda2/envs/cdms_dev/lib/python2.7/site-packages/cdms2/tvariable.py:282: ComplexWarning: Casting complex values to real discards the imaginary part
  fill_value = numpy.array(fill_value).astype(dtype)
variable_2
masked_array(data = [ 0.+0.j  1.+0.j  2.+0.j  3.+0.j],
             mask = False,
       fill_value = 1j)

doutriaux1 commented 7 years ago

@chaosphere2112 what's the issue here? I see the fill_value being 1j

dnadeau4 commented 7 years ago

@doutriaux1 Do we really want CDMS to handle complex number? I don't know any model/satellite that has complex number values.

chaosphere2112 commented 7 years ago

@doutriaux1 @dnadeau4 The issue is that there's a warning message:

/Users/fries2/anaconda2/envs/cdms_dev/lib/python2.7/site-packages/cdms2/tvariable.py:282: ComplexWarning: Casting complex values to real discards the imaginary part
  fill_value = numpy.array(fill_value).astype(dtype)

It's not a "real" problem (or a "complex" one :drum:) but it is a thing that happens, and we do try and squash warnings when we can.

The only reason I can see that we should support complex numbers is because NumPy supports complex numbers, and we're trying to be as transparent a wrapper around NumPy as possible. I don't really care either way, but this message popped up in my testing, so I thought I'd file the issue.

oliviermarti commented 7 years ago

Do we really want CDMS to handle complex number? I don't know any model/satellite that has complex number values.

Charles,

I'm currently processing model data with Fourier transform (np.fft) and wavelets (pyw). Both methods give complex as outputs, and I whish I could write the results in NetCDF Files.

So yes, I would really appreciate that cmds handles complex data ...

Olivier

doutriaux1 commented 7 years ago

@dnadeau4 look like we now have a real need for this.

dnadeau4 commented 7 years ago

let see if this can be done.

doutriaux1 commented 7 years ago

thx

dnadeau4 commented 7 years ago

@oliviermarti Can you provide me with a netcdf file that contains complex numbers?

oliviermarti commented 7 years ago

Le 4 juil. 2017 à 00:55, Denis Nadeau notifications@github.com a écrit :

@oliviermarti https://github.com/oliviermarti Can you provide me with a netcdf file that contains complex numbers?

Denis,

from https://www.unidata.ucar.edu/software/netcdf/docs/data_type.html https://www.unidata.ucar.edu/software/netcdf/docs/data_type.html, the NetCDF data type are :

This close the case for cmds. I guess I will escalate the case to Unidata. But with very little hope ...

Sincere thanks for caring.

Olivier

-- Dr. Olivier Marti - mailto:olivier.marti@lsce.ipsl.fr Laboratoire des Sciences du Climat et de l'Environnement Institut Pierre Simon Laplace Laboratoire CEA-CNRS-UVSQ - UMR 8212 Tel : +33 1 69 08 77 27 - Mob : +33 6 45 36 43 74 Fax : +33 1 69 08 30 73 http://www.lsce.ipsl.fr/Pisp/olivier.marti

dnadeau4 commented 7 years ago

@oliviermarti Can you cut/paste the Unidata issue link that you opened so that we can follow it from #73. I would be interested to be part of the discussion.

oliviermarti commented 7 years ago

I'm still trying to escalate the problem : complex numbers are not event available as a HDF5 datatype :-((

Olivier

doutriaux1 commented 7 years ago

@oliviermarti these might help you push it up. @dnadeau4 can we try to implement this in cdms? https://stackoverflow.com/questions/24937785/best-way-to-save-an-array-of-complex-numbers-with-hdf5-and-c Of course we would need to read it back correctly. It might be faster than waiting for netcdf4/hdf5 to implement it.

stefraynaud commented 7 years ago

@oliviermarti why don't you save the real and imaginary part separately? You can probably subclass cdms2 classes to intercept complex arrays before writing them with extensions like _real and _imag, and read them based on the same id pattern.

oliviermarti commented 7 years ago

Stéphane,

This is indeed not a blocking problem, and I won't wait for a proper solution from HDF5, then NetCDF, then cmds, etc .. ! I can save my data in several ways. I choose to write modulus and angle, which are the quantity that I will plot later.

But a solution with an acknowledge data standard would be nice.

A solution specific to cmds will not be recognized in NetCDF-Fortran or NetCDF-C, or by plotting softwares.

Olivier

CDAT / cdms

CDMS issues a warning message when working with complex numbers #73