search

ROBelgium / MSNoise

A Python Package for Monitoring Seismic Velocity Changes using Ambient Seismic Noise | http://www.msnoise.org
European Union Public License 1.1
178 stars 83 forks source link

ERROR WITH STACK #368

Open Thully1 opened 2 months ago

Thully1 commented 2 months ago

Hi everyone,

I’m currently processing data using MSNoise and have encountered an issue. I followed the documentation closely, and everything went smoothly until the cross-correlation (CC) computations, which are marked as done. However, no output folder was created. When I ran the command msnoise stack -r, it did not stack the data; it simply responded with “Let’s stack” and moved on to the next command line. Here’s some additional information:

(base) C:\Users\S215122739.MANDELA>msnoise info

General:

Database Information (from db.ini):

Configuration:

Configuration Values: | Normal color indicates the default value is used | Green indicates modified values M data_folder: D:\AMBIENT NOISE\Converted data\2023 M output_folder: C:\Users\S215122739.MANDELA\Desktop\CROSS_CORRELATIONS data_structure: SDS archive_format: '' network: channels: M startdate: 2023-08-01 M enddate: 2023-08-16 analysis_duration: 86400 cc_sampling_rate: 20.0 M resampling_method: Decimate preprocess_lowpass: 8.0 M preprocess_highpass: 60 preprocess_max_gap: 10.0 preprocess_taper_length: 20.0 remove_response: N response_format: dataless response_path: inventory response_prefilt: (0.005, 0.006, 30.0, 35.0) maxlag: 120. corr_duration: 1800. overlap: 0.0 windsorizing: 3 whitening: A whitening_type: B stack_method: linear pws_timegate: 10.0 pws_power: 2.0 M crondays: 2 components_to_compute: ZZ cc_type: CC components_to_compute_single_station: '' cc_type_single_station_AC: CC cc_type_single_station_SC: CC autocorr: N M keep_all: Y keep_days: Y M ref_begin: 2023-08-01 M ref_end: 2023-08-16 M mov_stack: 5,10 M export_format: BOTH sac_format: doublets dtt_lag: static dtt_v: 1.0 dtt_minlag: 5.0 dtt_width: 30.0 dtt_sides: both dtt_mincoh: 0.65 dtt_maxerr: 0.1 dtt_maxdt: 0.1 plugins: '' M hpc: Y stretching_max: 0.01 stretching_nsteps: 1000

Filters: ID: [low:high] [mwcs_low:mwcs_high] mwcs_wlen mwcs_step Used? 1: [8.000:60.000] [8.000:60.000] 10 5 Y

Stations: NET.STA Long. Lat. Alt. Coord. Used? CT.A1R 297533.9734 6264092.9170 - UTM Y CT.A9E 297528.7908 6264021.6120 - UTM Y CT.A9G 297558.4761 6264066.6230 - UTM Y CT.A9H 297591.5238 6264038.0430 - UTM Y CT.A9J 297499.1735 6264046.2570 - UTM Y CT.A9M 297723.8472 6264099.7150 - UTM Y CT.A9N 297595.5992 6263967.2420 - UTM Y CT.A9P 297562.6170 6263992.7410 - UTM Y CT.A9U 297687.5286 6264136.8550 - UTM Y CT.A9W 297612.8097 6264114.9300 - UTM Y CT.A9Y 297720.2856 6264024.7450 - UTM Y CT.AAD 297650.4023 6264078.4330 - UTM Y CT.AAE 297671.8126 6263979.3360 - UTM Y CT.B4V 297680.6710 6264047.3290 - UTM Y CT.B4W 297629.1973 6264009.8700 - UTM Y CT.B4X 297760.8919 6264064.7470 - UTM Y CT.B51 297627.8549 6263939.5710 - UTM Y CT.B5X 297649.9817 6264171.1950 - UTM Y CT.B5Y 297612.2163 6264203.6810 - UTM Y

Jobs: CC: D : 2426 STACK: T : 2426 MWCS: none defined DTT: none defined

(base) C:\Users\S215122739.MANDELA>msnoise stack Let's STACK! Has anyone encountered a similar issue or have suggestions on resolving it? Are there any additional logs or configurations I should check? Any help or insights would be greatly appreciated!

Thank you!

ThomasLecocq commented 2 months ago

Your preprocess lowpass/highpass seem wrong (reversed), I suspect the compute_cc fails somehow due to that. These params are the real lowpass and highpass filter frequencies (contrary that the low and high values of the filters, can be misleading)

Thully1 commented 2 months ago

Thank you so much for your prompt response. I updated the preprocess high-pass and low-pass filters, then ran msnoise stack -r. Here’s an example of the output:

2024-09-21 15:22:19.316607 msnoise [pid 37352][INFO]: There are STACK jobs for some days to recompute for CT.A9G:CT.B5X 2024-09-21 15:22:19.347855 msnoise [pid 37352][INFO]: There are STACK jobs for some days to recompute for CT.A9N:CT.B5X 2024-09-21 15:22:19.379106 msnoise [pid 37352][INFO]: There are STACK jobs for some days to recompute for CT.A9P:CT.B5X 2024-09-21 15:22:19.410357 msnoise [pid 37352][INFO]: There are STACK jobs for some days to recompute for CT.A1R:CT.B5X 2024-09-21 15:22:19.425983 msnoise [pid 37352][INFO]: You just finished REF stacking. Remember to reset the STACK jobs if you need to compute MOV stacks. Run 'msnoise reset STACK' when all processes have finished. 2024-09-21 15:22:19.425983 msnoise [pid 37352][INFO]: The current STACK jobs have been intentionally left 'I'n progress so they can be reset. After that, I executed msnoise reset STACK and then ran msnoise stack -m, which resulted in the following output:

2024-09-21 15:23:23.198095 msnoise [pid 12084][INFO]: There are STACK jobs for some days to recompute for CT.A9P:CT.B4V 2024-09-21 15:23:23.291849 msnoise [pid 12084][INFO]: There are STACK jobs for some days to recompute for CT.A9E:CT.B4X 2024-09-21 15:23:23.385601 msnoise [pid 12084][INFO]: There are STACK jobs for some days to recompute for CT.A9J:CT.B4V 2024-09-21 15:23:23.448131 msnoise [pid 12084][INFO]: There are STACK jobs for some days to recompute for CT.B4W:CT.B4X Should I reset and stacking again?

ThomasLecocq commented 2 months ago

You have to rerun the compute_cc step! Run it with the -v option to see if it works fine.

Thully1 commented 2 months ago

Thank you for your response once again. I ran the compute_cc step with -v and got the following error: 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: CT.B4W.Z Reading 1 Files 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: ERROR reading file D:/AMBIENT NOISE/Converted data/2023/CT/B4W/SPZ.D\CT.B4W..SPZ.D.2023.228 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: CT.B4X.Z Reading 1 Files 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: ERROR reading file D:/AMBIENT NOISE/Converted data/2023/CT/B4X/SPZ.D\CT.B4X..SPZ.D.2023.228 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: CT.B51.Z Reading 1 Files 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: ERROR reading file D:/AMBIENT NOISE/Converted data/2023/CT/B51/SPZ.D\CT.B51..SPZ.D.2023.228 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: Not enough data for this day ! 2024-09-21 15:44:51.143560 msnoise [pid 13164][DEBUG]: Marking job Done and continuing with next ! 2024-09-21 15:44:52.085534 msnoise [pid 13164][INFO]: Finished: Compute CC I checked to see if my data can be read by Obspy and I ran a script in python and it was able to read and plot the data. I'm not ure where I went wrong

ThomasLecocq commented 2 months ago

Is your disk connected? Using the same disk letter ?

Thully1 commented 2 months ago

Yes, my disk is connected, I followed the workflow. I have the same problem, it seems there is a confusion with the paths. The data is located on my computer at C:\Users\S215122739.MANDELA\Desktop\MSNOISE CODE\2023, while in the database, the path is D:/AMBIENT NOISE/Converted data/2023/CT/B51/SPZ.D. The error with compute_cc indicates it is trying to read the file from C:/Users/S215122739.MANDELA/Desktop/MSNOISE CODE/2023/CT/B4X/SPZ.D\CT.B4X.00.SPZ.D.2023.228. I attempted to update the folder path in the configurations table, but I'm still encountering the same error.

ThomasLecocq commented 2 months ago

It is indeed because the path to the data is not correct, it was probably somewhere else when you did the scan_archive step. You should redo that step. You can clean the data_availability table using an expert function

msnoise db execute "delete from data_availability"

Thully1 commented 2 months ago

I deleted the data from the database and ran the scan_archive command again, but I still encounter the same error. I also tried changing the slashes to forward slashes in the configuration. The scan archive correctly identifies the path for the folder:

2024-09-23 11:07:45.723475 msnoise [pid 22180][INFO]: C:\Users\S215122739.MANDELA\Desktop\MSNOISECODE\2023\CT\B5Y\SPN.D: Added 12 | Modified 0 | Unchanged 0 However, the issue arises when running compute_cc. The slashes seem to change when reading the files in the folders. For example:

(base) C:\Users\S215122739.MANDELA>msnoise -v compute_cc 2024-09-23 11:09:33.117969 msnoise [pid 8968][INFO]: Starting: Compute CC 2024-09-23 11:09:33.180471 msnoise [pid 8968][INFO]: Will compute [ZZ] for different stations 2024-09-23 11:09:33.180471 msnoise [pid 8968][INFO]: Will compute [] for single stations 2024-09-23 11:09:33.180471 msnoise [pid 8968][INFO]: Checking if there are jobs to do 2024-09-23 11:09:33.196106 msnoise [pid 8968][INFO]: Getting the next job 2024-09-23 11:09:33.367971 msnoise [pid 8968][INFO]: New CC Job: 2023-08-01 (156 pairs with 13 stations) 2024-09-23 11:09:33.399225 msnoise [pid 8968][DEBUG]: CT.A1R.Z Reading 1 Files 2024-09-23 11:09:33.399225 msnoise [pid 8968][DEBUG]: ERROR reading file C:/Users/S215122739.MANDELA/Desktop/MSNOISECODE/2023/CT/A1R/SPZ.D\CT.A1R.00.SPZ.D.2023.213 The file path includes both forward and backward slashes, which may be causing the reading error.

ThomasLecocq commented 2 months ago

And that file opens without problems with obspy when you test?

Thully1 commented 2 months ago

I ran the following code in Spyder:

from obspy import read

Specify the file path using raw string notation

file_path = r"C:\Users\S215122739.MANDELA\Desktop\MSNOISECODE\2023\CT\A1R\SPZ.D\CT.A1R.00.SPZ.D.2023.213"

# Attempt to read the file
st = read(file_path, dtype=float)
print("File read successfully.")
print(st)  # Print the contents of the stream

except Exception as e: print(f"Error reading file: {e}") The output was:

%runfile C:/Users/S215122739.MANDELA/.spyder-py3/autosave/untitled13.py --wdir File read successfully. 1 Trace(s) in Stream: CT.A1R..SPZ | 2023-08-01T23:59:59.002451Z - 2023-08-02T23:59:44.832451Z | 400.0 Hz, 34554333 samples

ThomasLecocq commented 1 day ago

I didn't see this information

The file path includes both forward and backward slashes, which may be causing the reading error.

checking how this can happen

ThomasLecocq commented 1 day ago

can you show me a screenshot of the data_availability web page of msnoise admin, the table?