Re-processing of gPhoton DB (v2)

scfleming commented 7 years ago

Make sure the files are generated with Python 3.x

Re-examine the data types of existing gPhoton columns to make smaller if possible.
Think about new columns to add from RAW6, including eclipse numbers.
Fix eclipses that failed originally.
Fix null and non-null photon events being in wrong places (both directions).

bshiao commented 7 years ago

here's a list of the eclipse that didn't process, i.e. died in gPipe eclipse tilename

38150 MISGCSN2_22274_0339_css38150 38151 MISGCSN3_23778_0484 38152 MISGCSN3_11396_0115 38153 MISGCSN3_11396_0115 38154 MISGCSN3_21639_0484 38155 MISGCSN3_21639_0484 38156 MISGCSN3_21847_0337 38157 PS_ELAISN1_MOS16 38158 PS_ELAISN1_MOS15 38159 PS_ELAISN1_MOS14 38166 MISGCSN3_21903_0136 38167 MISGCSN3_22003_0137 38168 MISGCSN3_22003_0137 38169 MISGCSN3_21967_0137 38170 MISGCSN3_21967_0137 38182 MISGCSN3_22184_0137 38183 MISGCSN3_22184_0137 38184 MISGCSN3_22224_0137 38185 MISGCSN_21958_1658 38186 MISGCSN3_22224_0137 38187 MISGCSN3_22536_0138 38188 MISGCSN3_22536_0138 38189 MISGCSN3_22306_0137 38190 MISGCSN3_22306_0137 38191 MISGCSN3_22305_0137 38192 MISGCSN2_22096_0131_css38192 38193 MISGCSN3_22305_0137 38194 MISGCSN3_22097_0131 38195 MISGCSN3_22097_0131 38196 GI6_026004_NGC6789 38197 MISGCSN3_22292_0131 38198 MISGCSN3_11763_0017 38199 MISGCSN3_22054_0017 38204 MISGCSN2_22259_0137_css38204 38205 MISGCSN3_22299_0132 38206 MISGCSN3_22299_0132 38207 MISGCSN3_22343_0132 38208 MISGCSN1_22631_0138 38209 MISGCSN3_22343_0132 38210 MISGCSN3_22431_0132 38211 MISGCSN3_22431_0132 38212 MISGCSN3_23042_0133 38213 MISGCSN3_23042_0133 38214 MISGCSN2_22288_0127_css38214 38215 MISGCSN4_22429_0132_css38215 38216 MISGCSN4_22429_0132_css38216 38218 MISGCSN4_22283_0017_css38218 38219 MISGCSN2_22281_0018_css38219 38220 MISGCSN4_22323_0018_css38220 38221 MISGCSN2_11012_0014_css38221 38222 MISGCSN2_22985_0133_css38222 38223 MISGCSN2_22985_0133_css38223 38225 MISGCSN1_22375_0127 38226 MISGCSN2_22325_0018_css38226 38227 MISGCSN4_22699_0124_css38227 38228 MISGCSN1_23041_0133 38229 MISGCSN1_22369_0018 38233 GI6_079001_Kepler_UZ_Lyr 38234 GI6_079001_Kepler_UZ_Lyr 38235 GI6_079001_Kepler_UZ_Lyr 38236 GI6_079001_Kepler_UZ_Lyr 38237 GI6_079001_Kepler_UZ_Lyr 38238 GI6_079001_Kepler_UZ_Lyr 38239 GI6_079001_Kepler_UZ_Lyr 38240 GI6_079001_Kepler_UZ_Lyr 38241 GI6_079001_Kepler_UZ_Lyr 38242 GI6_079001_Kepler_UZ_Lyr 38243 GI6_079001_Kepler_UZ_Lyr 38244 GI6_079001_Kepler_UZ_Lyr 38245 GI6_079001_Kepler_UZ_Lyr 38246 GI6_079001_Kepler_UZ_Lyr 38247 GI6_079001_Kepler_UZ_Lyr 38248 GI6_079001_Kepler_UZ_Lyr 38250 GI6_079001_Kepler_UZ_Lyr_css3825 38252 GI6_079001_Kepler_UZ_Lyr 38253 GI6_079001_Kepler_UZ_Lyr 38254 GI6_079001_Kepler_UZ_Lyr 38255 GI6_079001_Kepler_UZ_Lyr 38256 GI6_079001_Kepler_UZ_Lyr 38257 GI6_079001_Kepler_UZ_Lyr 38258 GI6_079001_Kepler_UZ_Lyr 38259 GI6_079001_Kepler_UZ_Lyr 38261 GI6_079002_Kepler_BR_Cyg 38262 GI6_079002_Kepler_BR_Cyg 38263 GI6_079002_Kepler_BR_Cyg 38264 GI6_079002_Kepler_BR_Cyg 38265 GI6_079002_Kepler_BR_Cyg 38267 GI6_079002_Kepler_BR_Cyg 38916 PS_VVDS22H_MOS01 39893 GI5_028106_W1_18139_0274o 40326 GI6_060015_ESSENCE_D4 40330 GI6_060013_ESSENCE_D2 40427 PS_CDFS_MOS00 40442 PS_CDFS_MOS00 40497 PS_CDFS_MOS04 41840 GI6_061021_F_VOID_241 41866 DLS_F2_MOS02 42369 MISGCSN1_14141_0225 42428 MISGCSN1_24506_0317 42435 DLS_F5_MOS01 44600 SMC_42591_0399 44615 SMC_42592_0399 45369 AIS_419 45369 AIS_420 45431 MIS2DFSGP_38117_0396 45481 MISGCSS1_38363_0424 46388 MISGCSN1_03193_0069

(107 row(s) affected)

bshiao commented 7 years ago

current table schema description. initial thoughts are to change the datatypes of xa, ya, q to tinyint since the range is between 0 and 255, this has has to be confirmed though. Will shrink the row size by 9 bytes. If a unique id and eclipse are added it will increase the row by 12 bytes, 8 for a unique id and 4 for eclipse. So a overall increase of 3 bytes so far. CREATE TABLE [dbo].[FUVPhotonsPOZ]( [zoneID] [int] NOT NULL, [time] [bigint] NOT NULL, [cx] [float] NOT NULL, [cy] [float] NOT NULL, [cz] [float] NOT NULL, [x] [real] NOT NULL, [y] [real] NOT NULL, [xa] [int] NOT NULL, [ya] [int] NOT NULL, [q] [int] NOT NULL, [xi] [real] NOT NULL, [eta] [real] NOT NULL, [ra] [float] NOT NULL, [dec] [float] NOT NULL, [flag] [tinyint] NOT NULL )

confirmed the min/max valus for xa, ya, q select min(xa), max(xa), min(ya), max(ya), min(q), max(q) from FUVPhotonsPOZ

0 31 0 31 0 31

(1 row(s) affected)

cmillion / gPhoton

Re-processing of gPhoton DB (v2) #242