Initial Distribution Integer IDs Fail Silently

[electronsandstuff]

The following code using integer IDs saves an initial distribution file with an empty ID array.

easygdf.save_initial_distribution(distname, ... ID=np.arange(x.shape[0])+1)

It works correctly when called like the following which will pass an array of floats.

easygdf.save_initial_distribution(distname, ... ID=np.arange(x.shape[0])+1.0)

The expected behavior is that easygdf either saves the integers or converts to floats if GPT requires IDs in the distribution file to be floats.

[electronsandstuff]

Update: The issue is not that easygdf doesn't save the IDs. It does save them, but GPT does not read them in. gdf2a reports them as an invalid datatype.

Here is the output of gdf2a.

         x          y        GBx        GBy        GBz          t         ID          z 
 0.000e+00  0.000e+00  0.000e+00  0.000e+00  1.000e+02  0.000e+00 Invalid type 0.000e+00 
-3.000e-06  0.000e+00  0.000e+00  0.000e+00  1.000e+02  0.000e+00 Invalid type 0.000e+00 
-2.000e-06  0.000e+00  0.000e+00  0.000e+00  1.000e+02  0.000e+00 Invalid type 0.000e+00

Here is the output from gpt.

Warning: Environment variable OMP_WAIT_POLICY=PASSIVE not set.
gpt: /tmp/gptrunner_31965129272c4cd98dab5258f4b714f0.in(7): Error: Invalid length of "ID": Contains 0 points where 37 are required

[electronsandstuff]

By inspecting the output file and dropping breakpoints I was able to confirm that easygdf does correctly detect the datatype of the numpy array as an int64 and then outputs it as that to the file. I also was able to confirm from GPT documentation and headers that the ID easygdf is using for the int64 datatype is correct.

From what I can tell, easygdf is doing what's expected of it. However, there's still the issue of it creating files that gpt does not read in. This will confuse users and I think I should do something about it. I can see a couple of options.

• Automatically cast the bad datatypes to something GPT can load
• Issue a warning to the user that this datatype will create a file GPT has issues with and do nothing
• Do nothing, but add it to documentation
• Don't support saving files with the bad datatypes and error with invalid type (same as complex numbers for instance)

I have to leave for the moment, but will look at it further in a second.

TODO:

• [x] Make decision on how to address incompatible datatypes
• [x] Test GPT's support for all other datatypes

[electronsandstuff]

OK, I ran a test by using gdf2a to open a file with every dtype in both array and single form. Here are the results:

Here are the results for single types.

(double)  0.000e+00
(float)   0.000e+00
(s8)              0
(s16)             0
(s32)             0
Type=80  Type=80
(u8)               0
(u16)              0
(u32)              0
Type=70  Type=70
(ascii)  deadbeef
(nul)       No data
(Undef)  No data

Here are the results for array types.

 (double)    (float)  (s8)  (s16)  (s32)       Type=80  (u8)  (u16)  (u32)       Type=70       (Undef) 
0.000e+00  0.000e+00     0      0      0  Invalid type     0      0      0  Invalid type  Invalid type
1.000e+00  1.000e+00     1      1      1  Invalid type     1      1      1  Invalid type  Invalid type

Interestingly, neither 64 bit integer types appear to be supported. They just get shown as an unrecognized type in single form and invalid in array form. If I run strings on the gdf2a binary I don't even see the values "(u64)" or "(s64)" in there. I think it really doesn't exist and that the constants in the header files were just never implemented.

The undefined datatype also does not produce a valid array.

[electronsandstuff]

Since gpt is really the only program using GDF files I think I will consider it as defining the standard and so if 64 bit integers are not a valid datatype then I don't think my program should emit them. The question is then what to do about 64 bit ints being passed to easygdf.

• The consistent way to handle this would be by erroring out on an undefined datatype. I already do this for others
• It's not possible for a user to pass a 64 bit integer single value to easygdf. Python ints are already handled as 32 values. I don't have to worry about this case
• I want easygdf to handle as much for the user as possible, this means casting 64 bit numpy ints down to 32 bit
• I'm not OK with casting integers silently and potentially causing the user to lose information. I think I should add a warning for this
• This also raises the issue of potentially casting single valued python ints down. Apparently the python interpreter chooses the largest int available on the system and so I need to determine if I'm 32 or 64 bit at the time
• Perhaps I should only warn users when I actually clip values. I think this is even more convenient for users and realistically there's almost no use case in GPT for having huge integer values that would cause the clipping. This would mean that everyone except fringe users gets to benefit from having easygdf just do the conversion for them. Those few users would then have to deal with issues anyways since the file format simply does not support 64 bit ints. In this case I would be comfortable with the code erroring out instead of warning.