memory error

[aileenfisher]

Could probably get this to run on a box with higher RAM; I have only like 8G and 8G SWAP on this box local. But (and I have very little knowledge of processing) is there a way to make less memory-heavy? It was like 90% memory and 3% of my 4 CPU and then crashed out.

(infrapy_env) bash-4.2$ pwd /data/studies/PyGS/git/infrapy/examples (infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --cpu-cnt 4 --spectral-option cwt --cluster-min-samples 500 --cluster-eps 5

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters: local_wvfrms: data/YJ.BRP1..EDF.SAC local_latlon: None local_detect_label: None cpu_cnt: 4

Algorithm parameters: spectral_option: cwt morlet_omega0: 12.0 freq_min: 1.0 freq_max: 20.0 signal_start: None signal_end: None window_len: 900.0 window_step: 450.0 p_value: 0.01 freq_tm_factor: 35.0 cluster_eps: 5.0 cluster_min_samples: 500 cpu_cnt: 4

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary: YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis... Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] Clustering into detections... Traceback (most recent call last): File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')()) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, *kwargs) File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_detection.py", line 917, in run_sd det_list = spectral.run_sd(stream[0], spectral_option, morlet_omega0, [freq_min, freq_max], 0.8, p_value, window_len, window_step, freq_tm_factor, cluster_eps, cluster_min_samples, pl) File "/data/studies/PyGS/git/infrapy/infrapy/detection/spectral.py", line 217, in run_sd clustering = DBSCAN(eps=clustering_eps, min_samples=clustering_min_samples).fit(spec_dets_logf) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/base.py", line 1151, in wrapper return fit_method(estimator, args, **kwargs) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/cluster/_dbscan.py", line 410, in fit dbscan_inner(core_samples, neighborhoods, labels) File "sklearn/cluster/_dbscan_inner.pyx", line 34, in sklearn.cluster._dbscan_inner.dbscan_inner MemoryError: std::bad_alloc (infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --spectral-option cwt --cluster-min-samples 500 --cluster-eps 5

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters: local_wvfrms: data/YJ.BRP1..EDF.SAC local_latlon: None local_detect_label: None

Algorithm parameters: spectral_option: cwt morlet_omega0: 12.0 freq_min: 1.0 freq_max: 20.0 signal_start: None signal_end: None window_len: 900.0 window_step: 450.0 p_value: 0.01 freq_tm_factor: 35.0 cluster_eps: 5.0 cluster_min_samples: 500

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary: YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis... Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] Clustering into detections... Traceback (most recent call last): File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')()) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, *kwargs) File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_detection.py", line 917, in run_sd det_list = spectral.run_sd(stream[0], spectral_option, morlet_omega0, [freq_min, freq_max], 0.8, p_value, window_len, window_step, freq_tm_factor, cluster_eps, cluster_min_samples, pl) File "/data/studies/PyGS/git/infrapy/infrapy/detection/spectral.py", line 217, in run_sd clustering = DBSCAN(eps=clustering_eps, min_samples=clustering_min_samples).fit(spec_dets_logf) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/base.py", line 1151, in wrapper return fit_method(estimator, args, **kwargs) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/sklearn/cluster/_dbscan.py", line 410, in fit dbscan_inner(core_samples, neighborhoods, labels) File "sklearn/cluster/_dbscan_inner.pyx", line 34, in sklearn.cluster._dbscan_inner.dbscan_inner MemoryError: std::bad_alloc (infrapy_env) bash-4.2$

[aileenfisher]

Also:

(infrapy_env) bash-4.2$ infrapy run_sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC' --cpu-cnt 4

#####################################

InfraPy

Spectral Detection Analyses

#####################################

Data parameters: local_wvfrms: data/YJ.BRP1..EDF.SAC local_latlon: None local_detect_label: None cpu_cnt: 4

Algorithm parameters: spectral_option: spectrogram morlet_omega0: 12.0 freq_min: 1.0 freq_max: 20.0 signal_start: None signal_end: None window_len: 900.0 window_step: 450.0 p_value: 0.01 freq_tm_factor: 35.0 cluster_eps: 10.0 cluster_min_samples: 40 cpu_cnt: 4

Loading local data from data/YJ.BRP1..EDF.SAC

Data summary: YJ.BRP1..EDF 2012-04-09T18:00:00.008300Z - 2012-04-09T18:19:59.998300Z

Running spectral detection (sd) analysis... Progress: [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] Clustering into detections... Identified 5 detections.

Writing detection results using label: data/YJ.BRP1_2012.04.09_18.00.00-18.19.59 (infrapy_env) bash-4.2$ infrapy plot sd --local-wvfrms 'data/YJ.BRP1..EDF.SAC'

#####################################

InfraPy

Spectral Detection (sd)

Visualization

#####################################

Data parameters: local_wvfrms: data/YJ.BRP1..EDF.SAC local_latlon: None local_detect_label: None

Loading local data from data/YJ.BRP1..EDF.SAC Plotting spectrogram with detection info...

Traceback (most recent call last): File "/data/studies/PyGS/miniconda3/envs/infrapy_env/bin/infrapy", line 33, in sys.exit(load_entry_point('infrapy', 'console_scripts', 'infrapy')()) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1130, in call return self.main(args, kwargs) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/data/studies/PyGS/miniconda3/envs/infrapy_env/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(args, **kwargs) File "/data/studies/PyGS/git/infrapy/infrapy/cli/cli_visualization.py", line 511, in sd det_vis.plot_sd(stream[0], det_list, [freq_min, freq_max], spec_option=spectral_option, morlet_omega0=morlet_omega0, output_path=figure_out, show_fig=show_figure) File "/data/studies/PyGS/git/infrapy/infrapy/detection/visualization.py", line 127, in plot_sd a[2].get_shared_y_axes().join(a[2], a[1]) AttributeError: 'GrouperView' object has no attribute 'join' (infrapy_env) bash-4.2$

[aileenfisher]

Works with --single-det-index added, but I'm pulling straight from the --help examples. I'm assuming there's some default stuff it wants but it's hard to track all that down. Would request either forcing inputs from the user or better error messaging, documentation and template config files with every option and explanations if there needs to be a certain combination of options.