Problem with verbose argument?

[Evidlo]

I noticed that in https://github.com/EdGrrr/pyLRT/blob/master/pyLRT/RadTran.py#L64-L70, calling run(verbose=True) sends stderr to subprocess.PIPE. Doesn't this have the effect of turning off error messages from radtran? This seems to be my experience in calling run().

>>> slrt.run(verbose=True)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 slrt.run(verbose=True)

File ~/resources/pyLRT/pyLRT/RadTran.py:87, in RadTran.run(self, verbose, print_input, print_output, regrid)
     85 if error_flag:
     86     error = ''.join(error)
---> 87     raise ValueError(error)
     89 if print_output:
     90     print('Output file:')

ValueError: UVSpec Error Message:
Error 2 while opening the representative wavelengths file.
Error 2 returned by function set_transmittance_wl_grid_reptran in (line 291, function 'setup_wlgrid', file 'ancillary.c')
Error 2 setting up transmittance wavelength grid in uvspec (uvspec.c)
Error 2 during execution of uvspec

>>> slrt.run(verbose=False)
 ... using CH4 volume mixing ratio profile from US standard atmosphere.
 ... using N2O volume mixing ratio profile from US standard atmosphere.
 ... using CO volume mixing ratio profile from US standard atmosphere.
 ... using N2 volume mixing ratio profile from US standard atmosphere.
 ... calling uvspec_check(), checking model input data
 ... calling setup_wlgrid(), generating transmittance wavelength grid
     reading bands from ../data/correlated_k/reptran/reptran_solar_medium.cdf.
**********************************************************************************
*Error: Data files for REPTRAN not found in directory data/correlated_k/reptran. *'
*       Please check whether you have downloaded the required REPTRAN data files *
*       from http://www.libradtran.org/doku.php?id=download and unzipped the data*
*       in the libRadtran folder.                                                *
**********************************************************************************
Error 2 while opening the representative wavelengths file.
Error 2 returned by function set_transmittance_wl_grid_reptran in (line 291, function 'setup_wlgrid', file 'ancillary.c')
Error 2 setting up transmittance wavelength grid in uvspec (uvspec.c)
Error 2 during execution of uvspec
/home/evan/resources/pyLRT/pyLRT/RadTran.py:101: UserWarning: genfromtxt: Empty input file: "<_io.StringIO object at 0x7fbf698e2430>"
  return np.genfromtxt(io.StringIO(process.stdout))
Out[5]: array([], dtype=float64)

[EdGrrr]

Hi @Evidlo, thanks for raising this.

Redirecting stderr to subprocess.PIPE was intended to help convert a uvspec error to a python exception - https://github.com/EdGrrr/pyLRT/blob/master/pyLRT/RadTran.py#L77-L87 checks stderr for any lines that indicate an error, then converts this to a python exception. This setting should not depend on the 'verbose' argument though, so I suggest removing the if statement at https://github.com/EdGrrr/pyLRT/blob/master/pyLRT/RadTran.py#L64, sending all output to stderr to be parsed.

This has the effect of silencing uvspec warnings though (which I didn't realise). To address this, I suggest an extra loop that looks for warnings in the uvspec output then prints them. This can be silenced with the 'quiet' option for RadTran.run(). This would be inserted at L77.

# Check uvspec output for errors/warnings                                                                                              
if not quiet:                                                                                                                          
    print_flag = False                                                                                                                 
    for line in io.StringIO(process.stderr):                                                                                           
        if line.startswith('*** Warning'):                                                                                                    
            # Some uvspec warnings start with three stars                                                                              
            # These have three stars for every line                                                                                    
            print(line.strip())                                                                                                        
        elif line.startswith('*****'):                                                                                                 
            # Many uvspec warnings are in a star box                                                                                   
            print(line.strip())                                                                                                        
            print_flag = not(print_flag)                                                                                               
        elif print_flag:                                                                                                               
            # Print line if we are within a star box                                                                                   
            print(line.strip()) 

I can see the naming of the 'verbose' option here can be confusing - it comes from the libradtran naming, so the 'verbose' argument shouldn't affect the amount of output printed to screen, but it controls whether we try to parse the uvspec verbose output (which has optical depths, concentrations etc.). I am not sure whether it is a good idea to change this though, as we might want to keep the link to the libradtran name?

[EdGrrr]

Commit https://github.com/EdGrrr/pyLRT/commit/0d0ae0f759278f14cd7d6f701e4c74576dc90bce should include these changes.

As a test, the first two of these calls should return a warning, the third should not.

from pyLRT import RadTran, get_lrt_folder                                                                                              

LIBRADTRAN_FOLDER = get_lrt_folder()                                                                                                   

slrt = RadTran(LIBRADTRAN_FOLDER)                                                                                                      
slrt.options['rte_solver'] = 'disort'                                                                                                  
slrt.options['source'] = 'solar'                                                                                                       
slrt.options['wavelength'] = '200 2600'                                                                                                
slrt.options['output_user'] = 'lambda eglo eup edn edir'                                                                               
slrt.options['zout'] = '0 5 TOA'                                                                                                       
slrt.options['albedo'] = '0'                                                                                                           
slrt.options['umu'] = '-1.0 1.0'                                                                                                       
slrt.options['quiet'] = ''                                                                                                             
slrt.options['sza'] = '0'                                                                                                              

##############                                                                                                                         
# Run the RT #                                                                                                                         
##############                                                                                                                         
print('Verbose run - prints warning')                                                                                                  
sdata, sverb = slrt.run(verbose=True)                                                                                                  
print('')                                                                                                                              
print('Non-verbose run - prints warning')                                                                                              
sdata = slrt.run(verbose=False)                                                                                                        
print('')                                                                                                                              
slrt.options['wavelength'] = '250 2600'                                                                                                
print('Verbose run - doesn\'t print warning')                                                                                          
sdata, sverb = slrt.run(verbose=True)     

[EdGrrr]

Which reminds me #4

[Evidlo]

Thanks for the fix!