twavpol only displaying wave polarization data for certain time ranges

[hsalinasGIT]

Hello Good People, I am trying to use twavpol to perform a wave-polarization analysis (wave angle and ellipticity with polarization > 0.7) on the SCM burst data for this interval ['2015-09-19/10:07:00', '2015-09-19/10:07:12']. Here is a sample code that follows this mms example (https://github.com/spedas/mms-examples/blob/master/advanced/Wave_polarization_using_SCM_data.ipynb):

    achDate = ['2015-09-19/10:07:00', '2015-09-19/10:07:12']
    SCMbrst_vars = scm(probe = 4, data_rate = 'brst',trange=achDate, time_clip=True)
#SCM burst sampling rate = 8192/inputs as done in example tutorial
    nopfft_input = 8192 # number of points for FFT
    steplength_input = nopfft_input/2 # number of points for shifting between 2 FFT
    bin_freq_input = 3 # number of bins for frequency averaging
    pyspedas.twavpol('mms4_scm_acb_gse_scb_brst_l2', nopfft=nopfft_input, steplength=steplength_input,bin_freq=bin_freq_input)
    #-Chaning units of wave-angle from radians to degrees
    time_wa, val_wa, val_freq = get_data('mms4_scm_acb_gse_scb_brst_l2_waveangle')
    val_wa = val_wa*180./np.pi
    store_data('mms4_scm_acb_gse_scb_brst_l2_waveangle', data={'x':time_wa, 'y':val_wa, 'v':val_freq})
    #'''
    #Plots results if wave-polarization is greater than 0.7 (<0.7 is considered noise)
    deg_pol_c = 0.7
    timedp, val, val_freq = get_data('mms4_scm_acb_gse_scb_brst_l2_degpol')
    index_deg_pol = val < deg_pol_c
    val[index_deg_pol] = np.nan
    store_data('mms4_scm_acb_gse_scb_brst_l2_degpol',data={'x':timedp, 'y':val, 'v':val_freq})

    timewa,val,val_freq = get_data('mms4_scm_acb_gse_scb_brst_l2_waveangle')
    val[index_deg_pol] = np.nan
    store_data('mms4_scm_acb_gse_scb_brst_l2_waveangle',data={'x':timewa, 'y':val, 'v':val_freq})

    timeEL, val, val_freq = get_data('mms4_scm_acb_gse_scb_brst_l2_elliptict')
    val[index_deg_pol] = np.nan
    store_data('mms4_scm_acb_gse_scb_brst_l2_elliptict',data={'x':timeEL, 'y':val, 'v':val_freq})
    #''';
    end = time.perf_counter()
    print('Run-time: %0.2f sec or %0.2f min'%(end-start, (end-start)/60))
    #Tplotting wavepol tplots variables
    freq_min = np.floor(8192/nopfft_input)
    freq_max = 4096.
     #-Changing ylim range
    options('mms4_scm_acb_gse_scb_brst_l2_degpol', 'yrange', [freq_min, freq_max])
    options('mms4_scm_acb_gse_scb_brst_l2_waveangle', 'yrange', [freq_min, freq_max])
    options('mms4_scm_acb_gse_scb_brst_l2_elliptict', 'yrange', [freq_min, freq_max])
    #-Turn back on spectrogram mode
    options('mms4_scm_acb_gse_scb_brst_l2_degpol', 'spec', True)
    options('mms4_scm_acb_gse_scb_brst_l2_waveangle', 'spec', True)
    options('mms4_scm_acb_gse_scb_brst_l2_elliptict', 'spec', True)
    #-Changing ztitle
    options('mms4_scm_acb_gse_scb_brst_l2_degpol', 'ztitle', 'Deg. Pol.')
    options('mms4_scm_acb_gse_scb_brst_l2_waveangle', 'ztitle', 'Wave Angle')
    options('mms4_scm_acb_gse_scb_brst_l2_elliptict', 'ztitle', 'Ellipticity')
    #-Turn on ylog mode
    options('mms4_scm_acb_gse_scb_brst_l2_degpol', 'ylog', 1)
    options('mms4_scm_acb_gse_scb_brst_l2_waveangle', 'ylog',1)
    options('mms4_scm_acb_gse_scb_brst_l2_elliptict', 'ylog',1)
    #-Changing color bar range
    options('mms4_scm_acb_gse_scb_brst_l2_degpol', 'zrange', [0.7, 1.])
    options('mms4_scm_acb_gse_scb_brst_l2_waveangle', 'zrange', [0., 90.])
    options('mms4_scm_acb_gse_scb_brst_l2_elliptict', 'zrange', [-1.0, 1.0])
    tplot(['mms4_scm_acb_gse_scb_brst_l2_degpol',
           'mms4_scm_acb_gse_scb_brst_l2_waveangle', 'mms4_scm_acb_gse_scb_brst_l2_elliptict'])

However, when applied to the full time interval and even small subsections I am met with the following error log and no polarization data plotted:

03-May-24 15:07:13: wavpol Warning: file sampling frequency changes from 8208.031311154598Hz to 8192.0Hz
03-May-24 15:07:13: n_batches: 1552
03-May-24 15:07:13: Total number of steps:1552
03-May-24 15:07:13: Fourier Transform is not possible. 
03-May-24 15:07:13: Ngood = 6
03-May-24 15:07:13: Required number of points for FFT = 8192
03-May-24 15:07:13: Fourier Transform is not possible. 
03-May-24 15:07:13: Ngood = 40
03-May-24 15:07:13: Required number of points for FFT = 8192
03-May-24 15:07:13: Fourier Transform is not possible. 
....
03-May-24 15:07:17: Ngood = 55
03-May-24 15:07:17: Required number of points for FFT = 8192
03-May-24 15:07:17: Fourier Transform is not possible. 
03-May-24 15:07:17: Ngood = 82
03-May-24 15:07:17: Required number of points for FFT = 8192
03-May-24 15:07:17: 
wavpol completed successfully

It appears that only for this specific time range of ['2015-09-19/10:07:18', '2015-09-19/10:07:30'], that twavpol works correctly(see attatched terminal log) and I have polarization data plotted:

03-May-24 15:26:26: wavpol: File sampling frequency=8192.0Hz
03-May-24 15:26:26: n_batches: 1
03-May-24 15:26:26: Total number of steps:24
03-May-24 15:26:26: Total number of possible FFT in the batch no 0 is:23.0
03-May-24 15:26:28: wavpol step: 0 
03-May-24 15:26:58: 
wavpol completed successfully

Could someone explain to me why twavpol is only able to create the polzarization data for that certain time subsection even when there's SCM burst data available for the entire time interval? Do I have incorrect inputs or is it something else?

[jameswilburlewis]

Thanks for letting us know; I'll take a look. I wonder if there are any NaNs in the SCM data before it gets passed to twavpol? Even if there aren't that many, it could still clobber a lot of FFT window intervals....I wonder if it would work any better with nopfft = 4096 or lower?

[hsalinasGIT]

Hey JIm, I went ahead and tried scrubbing the SCM data of NaNs and also lowering nopfft to 4096 before passing through twavpol, and neither of the suggestions changed anything. twavpol still only produced wave polarization data for the ['2015-09-19/10:07:18', '2015-09-19/10:07:30'] trange and none of the other time sections.

Thanks for letting us know; I'll take a look. I wonder if there are any NaNs in the SCM data before it gets passed to twavpol? Even if there aren't that many, it could still clobber a lot of FFT window intervals....I wonder if it would work any better with nopfft = 4096 or lower?