search

d-chambers / Detex

A Python package for subspace detection and waveform similarity clustering
Other
32 stars 6 forks source link

Missing information in SQLite database #40

Open Amy137 opened 5 years ago

Amy137 commented 5 years ago

I ran the detex command and got all the detections for each station. When I went to use the results module, I discovered that the only table that made it to the SQLite database is ss_df, the one containing the results of the detections for the subspaces. After it gave the detections for the final station, it threw this up before stopping:

sta lta req of 8 failing on station IMU, dropping sta/lta requirement sta lta req of 8 failing on station IMU, dropping sta/lta requirement sta lta req of 8 failing on station TCRU, dropping sta/lta requirement UU.TCRU starting on 2018-11-20T15:23:00 is shorter than expected sta lta req of 8 failing on station TCRU, dropping sta/lta requirement UU.TCRU starting on 2019-05-01T01:00:00 is shorter than expected sta lta req of 8 failing on station TCRU, dropping sta/lta requirement Traceback (most recent call last): File "", line 1, in File "/home/arecord/Subspace/detex/detex/subspace.py", line 1873, in detex self.setSinglesThresholds() File "/home/arecord/Subspace/detex/detex/subspace.py", line 1089, in setSinglesThresholds useSubSpaces=False, **kwargs) File "/home/arecord/Subspace/detex/detex/subspace.py", line 1743, in getFAS issubspace=False) File "/home/arecord/Subspace/detex/detex/fas.py", line 59, in _initFAS ssArrayTD, ssArrayFD, staltalimit) File "/home/arecord/Subspace/detex/detex/fas.py", line 119, in _getDSVect detex.log(name, msg, level='error') File "/home/arecord/Subspace/detex/detex/init.py", line 138, in log raise e(msg) Exception: Could not get any data for ECUT

d-chambers commented 5 years ago

Hey @Amy137, This looks like its coming from the fas module (false alarm statistic). This usually happens when calling the svd method. It is trying to get an empirical distribution of the null space in order to determine a threshold but apparently cant get data from ECUT. Can you remove that station from the station key and see if it works? You may also check that the ECUT data are there and don't have a ton of gaps, weird spikes, etc.

Amy137 commented 5 years ago

Hi @d-chambers , I don't know why this didn't occur to me earlier. ECUT was not installed until August 2018 and so is "missing" the first 50 days of data. The other stations start on day 152 while this one start on day 200. Is there a way to account for this? ECUT is an important station for this area so I don't want to drop it completely.

d-chambers commented 5 years ago

I would recommend doing two runs; one for days 152-200 and one after ECUT comes online. Keep in mind the sensitivity of the detector will change (increase) when you add more stations so you have to be careful with your interpretation.

Amy137 commented 5 years ago

For doing two runs with different time windows, I should just change the station key to reflect the time window I'm running, correct? And can I start at the SVD step or should I go back to the clustering or creating the subspace step?

d-chambers commented 5 years ago

I should just change the station key to reflect the time window I'm running, correct?

Yes

I think you can start at the subspace step and share the cluster object between the runs, but I am not 100% sure.