search

prjemian / punx

Python Utilities for NeXus HDF5 files
https://prjemian.github.io/punx
5 stars 7 forks source link

structure: IOError: Unable to read attribute #68

Closed prjemian closed 7 years ago

prjemian commented 7 years ago

part of structure report:

  I:NX_FLOAT32[119] = [58957.637, 52409.66, 47508.793, '...', 21545.27]
    @IGORWaveType = 2
    @IGORWaveNote = IOError: Unable to read attribute (No appropriate function for conversion path)
    @signal = 1
    @uncertainities = Idev
    @units = 1/cm

HDFview says this is: Type = String, length = 5674 with Array Size = Scalar and Value =

DataFileName=52_0003.hdf;DataFileType=Nexus;;DataFilePath=Prg:Users:ilavsky:Desktop:TestNexusData:;ImageCounter=259
make=Dectris
model=Pilatus
maxSizeX=487
maxSizeY=195
USAXSmode=12_09_oils3.dat
    NEXUS_StringDataStartHere;AD_template_ID=$Id: nexus_template_pilatus.xml 9446 2016-08-05 16:46:02Z jemian $;definition=NXsas;end_time=Fri Dec 09 19:38:48 2016;program_name=NeXus areaDetector;run_cycle=2016-03;start_time=Fri Dec 09 19:39:01 2016;title=pinSAXS for 52;EPICS_PV_metadata:APS_run_cycle=2016-03;EPICS_PV_metadata:EmptyFileName=Unknown.hdf5;EPICS_PV_metadata:EndTime=Fri Dec 09 19:38:48 2016;EPICS_PV_metadata:GUPNumber=Internal time;EPICS_PV_metadata:SampleTitle=pinSAXS for 52;EPICS_PV_metadata:ScanMacro=12_09_oils3.dat;EPICS_PV_metadata:StartTime=Fri Dec 09 19:39:01 2016;EPICS_PV_metadata:UserName=Ross;EPICS_PV_metadata:monoE_EGU=keV;EPICS_PV_metadata:wavelength_EGU=Angstroms;control:mode=timer;data:description=Pilatus;data:local_name=Pilatus 100K;data:make0=Dectris;data:model=Pilatus;instrument:name=USAXS;instrument:aperture:description=USAXSslit;instrument:collimator:absorbing_material=Tungsten;instrument:collimator:geometry:shape:shape=nxbox;instrument:collimator:geometry:shape:size=see xsize and ysize;instrument:source:facility_beamline=9ID;instrument:source:facility_name=APS;instrument:source:facility_sector=XSD/9ID;instrument:source:facility_station=C;instrument:source:name=Advanced Photon Source Undulator A, sector 9ID-C;instrument:source:probe=x-ray;instrument:source:type=Synchrotron X-ray Source;link_rules:link=LEFT BLANK;sample:name=pinSAXS for 52;user1:name=Ross;user1:proposal_number=Internal time;        NEXUS_StringDataEndHere;        NEXUS_VariablesDataStartHere;EPICS_PV_metadata:GuardslitHap=1;EPICS_PV_metadata:GuardslitHpos=1.7764e-14;EPICS_PV_metadata:GuardslitVap=0.232;EPICS_PV_metadata:GuardslitVpos=2.2204e-16;EPICS_PV_metadata:I000_cts=182.5;EPICS_PV_metadata:I00_V=-0.065308;EPICS_PV_metadata:I00_cts=1;EPICS_PV_metadata:I00_gain=1e+09;EPICS_PV_metadata:I0_V=4.2587;EPICS_PV_metadata:I0_cts=1.3368e+07;EPICS_PV_metadata:I0_cts_gated=1.3368e+07;EPICS_PV_metadata:I0_gain=1e+08;EPICS_PV_metadata:Linkam_ci94_temp=30;EPICS_PV_metadata:Linkam_ci94_temp2=30;EPICS_PV_metadata:PIN_Y=12.2;EPICS_PV_metadata:PIN_Z=61;EPICS_PV_metadata:Pin_TrI0=1.3431e+06;EPICS_PV_metadata:Pin_TrI0gain=1e+08;EPICS_PV_metadata:Pin_TrPD=9.2648e+05;EPICS_PV_metadata:Pin_TrPDgain=1e+06;EPICS_PV_metadata:PresetTime=30;EPICS_PV_metadata:SDD=538.79;EPICS_PV_metadata:SRcurrent=101.79;EPICS_PV_metadata:USAXS_Q=0;EPICS_PV_metadata:USAXSslitHap=0.80016;EPICS_PV_metadata:USAXSslitHpos=8e-06;EPICS_PV_metadata:USAXSslitVap=0.20014;EPICS_PV_metadata:USAXSslitVpos=-3e-05;EPICS_PV_metadata:WAXS_X=0;EPICS_PV_metadata:ccdProtection=0;EPICS_PV_metadata:filterAl=0;EPICS_PV_metadata:filterGlass=0;EPICS_PV_metadata:filterTi=0;EPICS_PV_metadata:filterTrans=1;EPICS_PV_metadata:idE_ds=24.217;EPICS_PV_metadata:idE_us=24.198;EPICS_PV_metadata:is_2D_USAXS_scan=0;EPICS_PV_metadata:m2rp=1.68;EPICS_PV_metadata:monoE=24;EPICS_PV_metadata:mr_enc=7.8677;EPICS_PV_metadata:msrp=5.005;EPICS_PV_metadata:mx=24;EPICS_PV_metadata:my=1.7053e-13;EPICS_PV_metadata:pin_ccd_center_x=17.971;EPICS_PV_metadata:pin_ccd_center_x_pixel=104.48;EPICS_PV_metadata:pin_ccd_center_y=-0.21672;EPICS_PV_metadata:pin_ccd_center_y_pixel=-1.26;EPICS_PV_metadata:pin_ccd_pixel_size_x=0.172;EPICS_PV_metadata:pin_ccd_pixel_size_y=0.172;EPICS_PV_metadata:pin_ccd_tilt_x=0.398;EPICS_PV_metadata:pin_ccd_tilt_y=-6.866;EPICS_PV_metadata:sa=-8.679;EPICS_PV_metadata:scaler_freq=1e+07;EPICS_PV_metadata:sthick=1;EPICS_PV_metadata:sx=31.65;EPICS_PV_metadata:sy=84.5;EPICS_PV_metadata:wavelength=0.5166;EPICS_PV_metadata:wavelength_spread=8e-05;areaDetector_reduced_250:Q0=0.0074473;areaDetector_reduced_250:R0=1.9999e-05;areaDetector_reduced_250:dR=1.9551e-06;areaDetector_reduced_full:Q0=0.0048534;areaDetector_reduced_full:R0=1.9898e-05;areaDetector_reduced_full:x0=0.215;control:integral=1.3368e+07;control:preset=30;instrument:aperture:hcenter=8e-06;instrument:aperture:hsize=0.80016;instrument:aperture:vcenter=-3e-05;instrument:aperture:vsize=0.20014;instrument:collimator:geometry:shape:xcenter=8e-06;instrument:collimator:geometry:shape:xsize=0.80016;instrument:collimator:geometry:shape:ycenter=-3e-05;instrument:collimator:geometry:shape:ysize=0.20014;instrument:detector:beam_center_x=17.971;instrument:detector:beam_center_y=-0.21672;instrument:detector:distance=538.79;instrument:detector:x_pixel_size=0.172;instrument:detector:y_pixel_size=0.172;instrument:monochromator:energy=24;instrument:monochromator:wavelength=0.5166;instrument:monochromator:wavelength_spread=8e-05;sample:aequatorial_angle=-8.679;sample:thickness=1;       NEXUS_VariablesDataEndHere;;Processed on=Wed, Dec 28, 2016,5:35:18 PM;Units=Arbitrary;SampleThickness=1;SampleTransmission=0.90244;CorrectionFactor=1;SampleMeasurementTime=30;EmptyMeasurementTime=30;BackgroundMeasTime=30;SampleMonitorFnct=NI1_15IDDFindI0;SampleI0=0.13368;EmptyMonitorFnct=NI1_15IDDFindEfI0;EmptyI0=0.13853;Nika_SampleToDetectorDistacne=538.79;Nika_Wavelength=0.5166;Nika_XrayEnergy=24;Nika_PixelSizeX=0.172;Nika_PixelSizeY=0.172;Nika_HorizontalTilt=0.398;Nika_VerticalTilt=-6.866;Nika_BeamCenterX=104.48;Nika_BeamCenterY=-1.26;Nika_BeamSizeX=0.80016;Nika_BeamSizeY=0.20014;CalibrationFormula=1/I0/St*(1/T*(Sa2D)-I0/I0ef*(EF2D));SampleThickness=1;SampleTransmission=0.90244;SampleI0=0.13368;EmptyI0=0.13853;UseMonitorForEF=1;CurrentEmptyName=empty_0015.hdf;CurrentMaskFileName=9IDC default SAXS mask;QvectorMaxNumPnts=0;QvectorNumberPoints=120;AngularSector=270;AngularHalfWidth=30;PolarizationCorrection=None;UncertainityCalculationMethod=OldNikaMethod;
prjemian commented 7 years ago

This attribute is not reported by the validate code at this time.

prjemian commented 7 years ago

Data file:

52_0003_Nika.hdf.zip

prjemian commented 7 years ago

Here is code to reproduce that error:

import h5py
root = h5py.File('52_0003_Nika.hdf', 'r')
nxentry = root[root.attrs['default']]
nxdata = nxentry[nxentry.attrs['default']]
data = nxdata[nxdata.attrs['signal']]
wave_note = data.attrs.get('IGORWaveNote')

the last line triggers the error

changing it to this produces the same error:

wave_note = data.attrs['IGORWaveNote']
prjemian commented 7 years ago

part of h5dump that describes this attribute (and another nearby one):

  ATTRIBUTE "IGORWaveNote" {
     DATATYPE  H5T_STRING {
        STRSIZE 23352;
        STRPAD H5T_STR_NULLPAD;
        CSET H5T_CSET_UTF8;
        CTYPE H5T_C_S1;
     }
     DATASPACE  SCALAR
     DATA {
     (0): --lots of text--
     }
  }
  ATTRIBUTE "uncertainities" {
     DATATYPE  H5T_STRING {
        STRSIZE H5T_VARIABLE;
        STRPAD H5T_STR_NULLTERM;
        CSET H5T_CSET_UTF8;
        CTYPE H5T_C_S1;
     }
     DATASPACE  SCALAR
     DATA {
     (0): "Idev"
     }
  }
prjemian commented 7 years ago

The STRSIZE value looks wrong. Looking through the entire h5dump shows this is the problem:

     STRSIZE H5T_VARIABLE;
     STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
              STRSIZE 23352;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
              STRSIZE 23332;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
           STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE 23198;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
              STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
        STRSIZE H5T_VARIABLE;
prjemian commented 7 years ago

Both h5dump and HDFview can read this data so it must be valid HDF5. IgorPro writes it. The problem is that h5py cannot read it. Might be a limitation with the htype variable at this line:

File "...../lib/python2.7/site-packages/h5py/_hl/attrs.py", line 79, in __getitem__
prjemian commented 7 years ago

no fix for this problem in punx project except to wrap in try:except clause That is working within the structure module. The validate module has not hit that problem yet.