search

simpeg / aurora

software for processing natural source electromagnetic data
MIT License
14 stars 2 forks source link

If Nan present in run decimation propogates to all nans #328

Open kujaku11 opened 4 months ago

kujaku11 commented 4 months ago

I've got some runs that are sampled at 4096, but the channels don't all start and end at the exact same time, there are a few samples missing at the beginning and end of each channel time series, but not always the same ones. So there are a few Nans at the beginning and end of the RunTS channels. Looks like only Ey, Hx, Hy are the channels that have a few Nans.

Decimation level 0 seems to run fine, but on the subsequent decimations basically the entire time series of Ey, Hx, Hy become Nans. So there is a leakage somewhere of Nans between decimation levels. It seems to be with the decimation method prototype_decimation.

In PR #327 I added a prototype_decimation_4 which uses the scipy.signal accessor methods added to xarray from mth5.timeseries.scipy_filters. This accounts for nans and uses the more efficient and accurate resample_poly method. This seems to alleviate the nan issue and is slightly faster.

Example log output

24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392055,). Setting end_index to 2392055
24:05:29T14:30:12 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T01:09:41.999023437
24:05:29T14:30:14 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T01:09:41.998779296+00:00 does not match metadata end 2024-05-18T01:09:41.999023437+00:00 updating metatdata value to 2024-05-18T01:09:41.998779296+00:00
24:05:29T14:30:14 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392049,). Setting end_index to 2392049
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392048,). Setting end_index to 2392048
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392052,). Setting end_index to 2392052
24:05:29T14:30:15 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T07:09:41.998047000
24:05:29T14:30:17 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392055, data length = (2392054,). Setting end_index to 2392054
24:05:29T14:30:17 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T13:09:41.997559000
24:05:29T14:30:18 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T13:09:41.997559+00:00 does not match metadata end 2024-05-18T13:09:41.999023437+00:00 updating metatdata value to 2024-05-18T13:09:41.997559+00:00
24:05:29T14:30:18 | INFO | line:431 |aurora.transfer_function.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
24:05:29T14:30:18 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 4 Null values detected in xrds channel hy. Check if this is unexpected.
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 7 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 8 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 5 Null values detected in xrds channel hz. Check if this is unexpected.
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:22 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 1 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.006281s  (159.197990Hz)
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.004866s  (205.523721Hz)
24:05:29T14:30:26 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.003702s  (270.111088Hz)
24:05:29T14:30:27 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.002868s  (348.711915Hz)
24:05:29T14:30:28 | INFO | line:123 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:05:29T14:30:29 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hy. Check if this is unexpected.