Failure to produce output files after successful run

hvbish commented 8 months ago

When I run DIRTY for hundreds of stars individually, something like ~1% of the runs usually fail. It appears that the radiative transfer completes successfully, but there’s an issue with saving the results to file. The basename.fits file is the only one generated, and it is empty (size of zero bytes). Here is the output generated by DIRTY:

total number of cells = 295704000
number of orig. cells = 295704000
number of subdiv. cells = 0
working on wavelength [micron] = 1565
tau = 0 albedo = 0.6 g = 0.7
current # = 500000 stel sl = 0.524427 scat sl = 0.284147 ave 1st tau = 1.55943
current # = 1000000 stel sl = 0.524427 scat sl = 0.284461 ave 1st tau = 1.56047
current # = 1500000 stel sl = 0.524427 scat sl = 0.284381 ave 1st tau = 1.56101
current # = 2000000 stel sl = 0.524427 scat sl = 0.284687 ave 1st tau = 1.5607
current # = 2500000 stel sl = 0.524427 scat sl = 0.284654 ave 1st tau = 1.56102
current # = 3000000 stel sl = 0.524427 scat sl = 0.284665 ave 1st tau = 1.5605
current # = 3500000 stel sl = 0.524427 scat sl = 0.284713 ave 1st tau = 1.5602
current # = 4000000 stel sl = 0.524427 scat sl = 0.284699 ave 1st tau = 1.56007
current # = 4500000 stel sl = 0.524427 scat sl = 0.284703 ave 1st tau = 1.56043
current # = 5000000 stel sl = 0.524427 scat sl = 0.284691 ave 1st tau = 1.56054
current # = 5500000 stel sl = 0.524427 scat sl = 0.284652 ave 1st tau = 1.56095
current # = 6000000 stel sl = 0.524427 scat sl = 0.284725 ave 1st tau = 1.56092
current # = 6500000 stel sl = 0.524427 scat sl = 0.284674 ave 1st tau = 1.5608
current # = 7000000 stel sl = 0.524427 scat sl = 0.284649 ave 1st tau = 1.56085
current # = 7500000 stel sl = 0.524427 scat sl = 0.284591 ave 1st tau = 1.56095
current # = 8000000 stel sl = 0.524427 scat sl = 0.284573 ave 1st tau = 1.56092
current # = 8500000 stel sl = 0.524427 scat sl = 0.284515 ave 1st tau = 1.56083
current # = 9000000 stel sl = 0.524427 scat sl = 0.284513 ave 1st tau = 1.56086
current # = 9500000 stel sl = 0.524427 scat sl = 0.284556 ave 1st tau = 1.56089
Output for observer = 1
# photons = 1e+07
Stellar weight = 5.24427e+06
Stellar S/L = 0.524427 +/- 0 (0%)
ratio_to_sb_image = 2.15119e-39
Stellar flux [ergs cm^-2 s^-1 A^-1] = 6.84848e-45
# scattered photons = 1.41501e+07
Scattered weight = 2.84461e+06
Scattered S/L = 0.284461 +/- 0.000161764 (0.0568669%)
Scattered flux [ergs cm^-2 s^-1 A^-1] = 3.71477e-45
fits_write_key : output results (geometry details) status = 402 : bad float to string conversion

Interestingly, if I rerun all the stars again it doesn’t always seem to be the same ones that fail.

hvbish commented 8 months ago

Here is the header from a file that was successfully written out:

SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    4 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
NAXIS3  =                  540 / length of data axis 3                          
NAXIS4  =                    1 / length of data axis 4                          
EXTEND  =                    T / FITS dataset may contain extensions            
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
COMMENT **---------------------------------**                                   
COMMENT Output of the DIRTY model                                               
COMMENT Karl D. Gordon & Karl A. Misselt                                        
COMMENT version v2.0prealpha (Oct 2009)                                         
COMMENT **---------------------------------**

karllark commented 8 months ago

This is the primary header. Can i see the headers of the extensions? At least that is what I am expecting. And I can see the comment needs to be update. :-)

hvbish commented 8 months ago

Ah, the header above was for the *_rad_field.fits file. Here is the header of the main output file from a successful run:

Filename: fullres_extcorr_1565A_1.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     100   ()      
  1  TOT_SBoI      1 ImageHDU         9   (740, 740)   float64   
  2  TOT_SBoI_unc    1 ImageHDU         9   (740, 740)   float64   
  3  SCAT_SBoI     1 ImageHDU         9   (740, 740)   float64   
  4  SCAT_SBoI_unc    1 ImageHDU         9   (740, 740)   float64   
  5  STEL_SBoI     1 ImageHDU         9   (740, 740)   float64   
  6  STEL_SBoI_unc    1 ImageHDU         9   (740, 740)   float64   
  7  N_STEL_PHOT    1 ImageHDU         9   (740, 740)   float64   
  8  N_SCAT_PHOT    1 ImageHDU         9   (740, 740)   float64

# HDU 0 in fullres_extcorr_1565A_1.fits:
SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                    8 / number of bits per data pixel                  
NAXIS   =                    0 / number of data axes                            
EXTEND  =                    T / FITS dataset may contain extensions            
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
COMMENT **---------------------------------**                                   
COMMENT Output of the DIRTY model                                               
COMMENT Karl D. Gordon & Karl A. Misselt                                        
COMMENT version v2.0-r84ish (Mar 2010)                                          
COMMENT **---------------------------------**                                   
COMMENT **---------------------------------**                                   
COMMENT **-Parameter File------------------**                                   
COMMENT fullres_extcorr_1565A_params_1.txt                                      
COMMENT [Geometry]                                                              
COMMENT ## Global Details                                                       
COMMENT distance=800.0                                                          
COMMENT n_obs_angles=1                                                          
COMMENT obs_theta=0.                                                            
COMMENT obs_phi=0.                                                              
COMMENT #obs_file=obs_pos.dat                                                   
COMMENT ## Source Properties                                                    
COMMENT source_type=stars                                                       
COMMENT n_stars=1                                                               
COMMENT star_pos_x=-296.81                                                      
COMMENT star_pos_y=105.9                                                        
COMMENT star_pos_z=-100.81                                                      
COMMENT star_file=star_pos.dat                                                  
COMMENT ## Dust Properties                                                      
COMMENT filling_factor=0.15                                                     
COMMENT density ratio=0.01                                                      
COMMENT type=file                                                               
COMMENT #radius=1.0                                                             
COMMENT # shell parameters                                                      
COMMENT #very_inner_radius=0                                                    
COMMENT #inner_radius=0.5                                                       
COMMENT #outer_radius=1.0                                                       
COMMENT #subdivide_radius=0                                                     
COMMENT #shell_density_poly=2                                                   
COMMENT # radial optical depth                                                  
COMMENT tau=3.0                                                                 
COMMENT ## Dust Distribution                                                    
COMMENT # filling factor & density control the clumpiness                       
COMMENT filling_factor=0.05                                                     
COMMENT density_ratio=1.0                                                       
COMMENT # this controls how to subdivide cells                                  
COMMENT max_tau_per_cell=50.0                                                   
COMMENT # spherical or cubical clumps [choices are sphere/cube]                 
COMMENT clump_type=cube                                                         
COMMENT # size of grid (all dimensions equal)                                   
COMMENT grid_size=740                                                           
COMMENT #                                                                       
COMMENT # Input files for arbitrary distribution                                
COMMENT type_file_pos=/home/x-hbish/sharks/fullres_extcorr_1565A/pos_pcval_long.
COMMENT fits                                                                    
COMMENT type_file_tau_pc=/home/x-hbish/sharks/fullres_extcorr_1565A/tau_per_pc_1
COMMENT 565_long.fits                                                           
COMMENT [Dust Grains]                                                           
COMMENT type=single_wavelength                                                  
COMMENT wavelength=0.1565                                                       
COMMENT albedo=0.6                                                              
COMMENT g=0.7                                                                   
COMMENT [Model Book Keeping]                                                    
COMMENT Path to Dust Properties=/home/x-hbish/sharks/fullres_extcorr_1565A/     
COMMENT Cross Section SubDir=OpticalProperties/CrossSections/                   
COMMENT Calorimetry SubDir=OpticalProperties/Calorimetry/                       
COMMENT Model SubDir=Models/                                                    
COMMENT Model Name=test                                                         
COMMENT [SED]                                                                   
COMMENT [Run]                                                                   
COMMENT verbose=2                                                               
COMMENT num_photons=1e7                                                         
COMMENT do_dust_emission=0                                                      
COMMENT energy_conserve_target=0.05                                             
COMMENT do_global_output=0                                                      
COMMENT do_image_output=1                                                       
COMMENT output_image_size=740                                                   
COMMENT output_filebase=fullres_extcorr_1565A_1                                 
COMMENT output_type=ratio                                                       
COMMENT output_model_grid=1                                                     
COMMENT do_emission_type=1                                                      
COMMENT do_emission_grain=1                                                     
COMMENT **---------------------------------**                                   
RADIUS  =                 370. / model radius [pc]                              
DIST    =                 800. / distance to model [pc]                         
DUST_A  =                  0.6 / dust grain albedo                              
DUST_G  =                  0.7 / dust grain g                                   
DUST_TAU=                   0. / radial dust optical depth at current wavelength
DUST_FF =                   0. / dust filling factor                            
DUST_DR =        -8.172734E-05 / dust density ratio (k2/k1)                     
DUST_K1 =         5.900319E-39 / k1 optical depth per pc                        
DUST_K2 =        -5.820968E-05 / k2 optical depth per pc                        
N_OBS   =                    1 / number of observer line-of-sights              
CUR_OBS =                    1 / current observer line-of-sight                 
LOSTHETA=                   0. / theta of line-of-sight [degrees]               
LOSPHI  =                   0. / phi of line-of-sight [degrees]                 
STEL_SL =    0.531017887410648 / stellar/emitted luminosity                     
STEL_SLE=                   0. / STEL_SL uncertainty                            
SCAT_SL =     0.20588789933381 / scattered/emitted luminosity                   
SCAT_SLE= 0.000163221140367054 / SCAT_SL uncertainty                            

# HDU 1 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'TOT_SBoI'           / Name of Extension                              
ITYPE   = 'Total I/L'          / Image Type                                     

# HDU 2 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'TOT_SBoI_unc'       / Name of Extension                              
ITYPE   = 'Total unc I/L'      / Image Type                                     

# HDU 3 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'SCAT_SBoI'          / Name of Extension                              
ITYPE   = 'Scattered I/L'      / Image Type                                     

# HDU 4 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'SCAT_SBoI_unc'      / Name of Extension                              
ITYPE   = 'Scattered unc I/L'  / Image Type                                     

# HDU 5 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'STEL_SBoI'          / Name of Extension                              
ITYPE   = 'Stellar I/L'        / Image Type                                     

# HDU 6 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'STEL_SBoI_unc'      / Name of Extension                              
ITYPE   = 'Stellar unc I/L'    / Image Type                                     

# HDU 7 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'N_STEL_PHOT'        / Name of Extension                              
ITYPE   = '# stellar photons'  / Image Type                                     

# HDU 8 in fullres_extcorr_1565A_1.fits:
XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -64 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                  740 / length of data axis 1                          
NAXIS2  =                  740 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
EXTNAME = 'N_SCAT_PHOT'        / Name of Extension                              
ITYPE   = '# scattered photons' / Image Type

Is this what you need?

karllark commented 8 months ago

Thank is helpful. Thanks!

karllark commented 8 months ago

Definitely unset variables could be the issue. Below definitely are numbers that make no sense, hence allocated but not set variables. Will set them in the open PR for the internal observer and see if that fixes your 1% problem.

DUST_DR = -8.172734E-05 / dust density ratio (k2/k1)
DUST_K1 = 5.900319E-39 / k1 optical depth per pc
DUST_K2 = -5.820968E-05 / k2 optical depth per pc

karllark commented 8 months ago

Should be fixed with above PR. If it isn't, please reopen.

karllark / DIRTY_dustrt

Failure to produce output files after successful run #28