demeshened GADGET does not initialize "float_type"

[claytonstrawn]

Bug report

Bug summary

Function "_yield_coordinates" which sets "self._float_type" is never called in demeshened version of yt on binary gadget file, so it has a "no-attribute" error.

Code for reproduction

Claytons-MacBook-Pro:~ claytonstrawn$ iyt

In [1]: ds = yt.load("/Users/claytonstrawn/Desktop/astroresearch/code/sampledatasets/snaps/snap_M1196_4x_320") #some gadget file in binary

In [2]: SlicePlot(ds,1,"density")

Copy-paste of console:

In [1]: ds = yt.load("/Users/claytonstrawn/Desktop/astroresearch/code/sampledatasets/snaps/snap_M1196_4x_320")
yt : [INFO     ] 2017-11-19 15:57:49,247 Calculating time from 7.000e-01 to be 2.899e+17 seconds
yt : [INFO     ] 2017-11-19 15:57:49,247 Assuming length units are in kpc/h (comoving)
yt : [INFO     ] 2017-11-19 15:57:49,280 Parameters: current_time              = 2.89916464569e+17 s
yt : [INFO     ] 2017-11-19 15:57:49,280 Parameters: domain_dimensions         = [1 1 1]
yt : [INFO     ] 2017-11-19 15:57:49,281 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2017-11-19 15:57:49,281 Parameters: domain_right_edge         = [ 72000.  72000.  72000.]
yt : [INFO     ] 2017-11-19 15:57:49,282 Parameters: cosmological_simulation   = 1
yt : [INFO     ] 2017-11-19 15:57:49,282 Parameters: current_redshift          = 0.428571438349
yt : [INFO     ] 2017-11-19 15:57:49,282 Parameters: omega_lambda              = 0.74
yt : [INFO     ] 2017-11-19 15:57:49,282 Parameters: omega_matter              = 0.26
yt : [INFO     ] 2017-11-19 15:57:49,282 Parameters: hubble_constant           = 0.72

In [2]: SlicePlot(ds,1,"density")
yt : [INFO     ] 2017-11-19 15:57:50,341 Allocating for 2.079e+06 particles
Loading particle index: 100%|████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 860.77it/s]
yt : [INFO     ] 2017-11-19 15:57:50,353 Allocating for 2.079e+06 particles
Loading particle index: 100%|███████████████████████████████████████████████████████| 4/4 [00:00<00:00, 1147.79it/s]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/claytonstrawn/yt/scripts/iyt in <module>()
----> 1 SlicePlot(ds,1,"density")

/Users/claytonstrawn/yt/yt/visualization/plot_window.pyc in SlicePlot(ds, normal, fields, axis, *args, **kwargs)
   1961             del kwargs['north_vector']
   1962 
-> 1963         return AxisAlignedSlicePlot(ds, normal, fields, *args, **kwargs)
   1964 
   1965 def plot_2d(ds, fields, center='c', width=None, axes_unit=None,

/Users/claytonstrawn/yt/yt/visualization/plot_window.pyc in __init__(self, ds, axis, fields, center, width, axes_unit, origin, right_handed, fontsize, field_parameters, window_size, aspect, data_source)
   1274             slc = ds.slice(axis, center[axis], field_parameters=field_parameters,
   1275                            center=center, data_source=data_source)
-> 1276             slc.get_data(fields)
   1277         validate_mesh_fields(slc, fields)
   1278         PWViewerMPL.__init__(self, slc, bounds, origin=origin,

/Users/claytonstrawn/yt/yt/data_objects/data_containers.pyc in get_data(self, fields)
   1364 
   1365         read_particles, gen_particles = self.index._read_particle_fields(
-> 1366                                         particles, self, self._current_chunk)
   1367         for f, v in read_particles.items():
   1368             self.field_data[f] = self.ds.arr(v, input_units = finfos[f].units)

/Users/claytonstrawn/yt/yt/geometry/geometry_handler.pyc in _read_particle_fields(self, fields, dobj, chunk)
    225             chunks,
    226             selector,
--> 227             fields_to_read)
    228         return fields_to_return, fields_to_generate
    229 

/Users/claytonstrawn/yt/yt/utilities/io_handler.pyc in _read_particle_selection(self, chunks, selector, fields)
    197         # psize maps the names of particle types to the number of
    198         # particles of each type
--> 199         self._count_particles_chunks(psize, chunks, ptf, selector)
    200 
    201         # Now we allocate

/Users/claytonstrawn/yt/yt/frontends/sph/io.pyc in _count_particles_chunks(self, psize, chunks, ptf, selector)
     18 
     19     def _count_particles_chunks(self, psize, chunks, ptf, selector):
---> 20         for ptype, (x, y, z), hsml in self._read_particle_coords(chunks, ptf):
     21             psize[ptype] += selector.count_points(x, y, z, hsml)
     22         return dict(psize)

/Users/claytonstrawn/yt/yt/frontends/gadget/io.py in _read_particle_coords(self, chunks, ptf)
    293                 f.seek(poff[ptype, "Coordinates"], os.SEEK_SET)
    294                 pos = self._read_field_from_file(
--> 295                     f, tp[ptype], "Coordinates")
    296                 f.seek(poff[ptype, "SmoothingLength"], os.SEEK_SET)
    297                 hsml = self._read_field_from_file(

/Users/claytonstrawn/yt/yt/frontends/gadget/io.py in _read_field_from_file(self, f, count, name)
    342             dt = self._endian + self.ds._id_dtype
    343         else:
--> 344             dt = self._endian + self._float_type
    345         if name in self._vector_fields:
    346             count *= self._vector_fields[name]

AttributeError: 'IOHandlerGadgetBinary' object has no attribute '_float_type'

This works on the stable version, but it's done so differently between meshed and demeshed that it's hard for me to understand where this function should be called.

Version Information installed the dev version, after demeshening as according to https://nbviewer.jupyter.org/url/trident-project.org/notebooks/trident_demesh_install.ipynb

[ngoldbaum]

Sounds like I messed up a merge with upstream. Any chance you can share a dataset that triggers this?

[ngoldbaum]

Alternatively you could see if this triggers with one of the sample datasets at yt-project.org/data.

[ngoldbaum]

It looks like I'm not able to trigger this with any of the Gadget data I have available to me. @claytonstrawn any chance you can share a Gadget output that triggers this? You can use the yt curldrop to share files:

https://docs.hub.yt/services.html#curldrop

[claytonstrawn]

I uploaded a file called "snap_M1196_4x_320" to the curldrop which triggers this.

I tried running this code on the sample file "Gadget3-snap-format2" and it seemed to work at first, but after I played with the source code a little to see when it runs "_yield_coordinates" it had the same problem. Is "_yield_coordinates" called inside a try-except clause or something (so it was just ignored instead of throwing an error like I expected)? What's weird is that after that, I undid my changes and reinstalled yt with pip, and it did not go back to working correctly, so the loading might have saved some auxiliary file without that variable and isn't being overwritten when I load it again.

[claytonstrawn]

Edit: It works again after I delete the .ewah file, it looks like it is possible for that file to be created missing this "self._float_type" variable, and if it is, then future loads which use this .ewah file will have this error.

[ngoldbaum]

Can you share the link to the curldropped file? It should have printed the link when you finished uploading.

[ngoldbaum]

The file itself doesn't set the _float_type attribute, but it's possible that there's a bug in how we restore the bitmap index that causes that attribute to not be set.

FWIW I'm unable to reproduce with Gadget3-snap-format2 both with and without a .ewah file existing on-disk.

[claytonstrawn]

Oh I see, I didn't realize what that was. http://use.yt/upload/e7aade88

Actually it looks like even if I create the .ewah file correctly, I still get this error as soon as I try to load it a second time, so maybe it is not reading the values back out of it correctly.

[ngoldbaum]

OK, great, I'm able to reproduce with that file.

[ngoldbaum]

I've just pushed a fix. Thanks for the report! Please feel free to reopen the issue if you're still having trouble with this issue. If you are having unrelated issues, please open a new ticket.