Closed BGunnett8 closed 1 year ago
CameronBodine
commented
1 year ago Thanks for the detailed report! I assume you are trying to batch process multiple recordings? Does the mosaic process only fail on the last recording in the batch? If so, can you try running the main.py script on only that recording to see if the issue persists? Also, can you copy the output from the console?
BGunnett8
commented
1 year ago I am trying to batch process several different recordings and it worked just fine when I did it a few weeks ago. This the output I get for the first recording and it continues to do this. It's processing everything normally but seems to skip over the step of creating the mosaics. It does this for every recording (Check my comment below this one to see the correct copy and paste of the output).
+++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++ Working On ./BenData2\R00021.DAT Start Time: 2023-06-12 17:11
READING
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 38.5 | 6.4
Getting DAT Metadata... Done! Time (s): 0.1
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 38.5 | 6.4
Getting Header Structure... Header Length: 72 ds_highfreq : Done! ss_port : Done! ss_star : Done! ds_vhighfreq : Done! Time (s): 0.0
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
3.5 | 38.5 | 6.4
Getting SON file header metadata... [Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers. [Parallel(n_jobs=4)]: Done 1 tasks | elapsed: 6.2s [Parallel(n_jobs=4)]: Done 2 out of 4 | elapsed: 6.2s remaining: 6.2s [Parallel(n_jobs=4)]: Done 4 out of 4 | elapsed: 6.3s remaining: 0.0s [Parallel(n_jobs=4)]: Done 4 out of 4 | elapsed: 6.3s finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
7.8 | 42.4 | 7.1
Locating missing pings and adding NoData... [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 8 | elapsed: 15.2s remaining: 45.8s [Parallel(n_jobs=8)]: Done 3 out of 8 | elapsed: 15.3s remaining: 25.5s [Parallel(n_jobs=8)]: Done 4 out of 8 | elapsed: 15.3s remaining: 15.3s [Parallel(n_jobs=8)]: Done 5 out of 8 | elapsed: 15.3s remaining: 9.2s [Parallel(n_jobs=8)]: Done 6 out of 8 | elapsed: 15.4s remaining: 5.1s [Parallel(n_jobs=8)]: Done 8 out of 8 | elapsed: 15.7s remaining: 0.0s [Parallel(n_jobs=8)]: Done 8 out of 8 | elapsed: 15.7s finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 46.4 | 7.8
Summary of Ping Metadata:
B001 : ds_highfreq Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 0 | 29460 | 14730.0 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 194.1 | 289.7 | 252.144 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 200.0 | 200.0 | 200.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 67027781.0 | 33508641.331 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.32 | 1417.718 | True orig_record_num | 0.0 | 29196.0 | 14623.5 | True
B002 : ss_port Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 2 | 29461 | 14732.001 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.139 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1050.0 | 1050.0 | 1050.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 65683064.0 | 32841751.489 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1388.906 | True orig_record_num | 1.0 | 29197.0 | 14625.48 | True
B003 : ss_star Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 3 | 29463 | 14732.991 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.146 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1050.0 | 1050.0 | 1050.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 65683064.0 | 32841751.489 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1388.953 | True orig_record_num | 1.0 | 29197.0 | 14626.46 | True
B004 : ds_vhighfreq Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 1 | 29462 | 14731.008 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.139 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1100.0 | 1100.0 | 1100.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 67027781.0 | 33508641.331 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1417.77 | True orig_record_num | 1.0 | 29197.0 | 14624.5 | True
Done! Time (s): 25.8
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 46.4 | 7.8
Automatically estimating depth for 15 chunks:
Using Zheng et al. 2021 method. Loading model: bedpick_ZhengApproach_20220629.h5[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 0s 361ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 0s 359ms/step [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 41.5s remaining: 4.5min 1/1 [==============================] - 0s 423ms/step 1/1 [==============================] - 1s 595ms/step 1/1 [==============================] - 1s 611ms/step 1/1 [==============================] - 0s 314ms/step [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 42.7s remaining: 2.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 43.1s remaining: 1.1min 1/1 [==============================] - 0s 425ms/step 1/1 [==============================] - 0s 425ms/step [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 43.9s remaining: 38.4s 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 0s 261ms/step 1/1 [==============================] - 1s 910ms/step 1/1 [==============================] - 1s 987ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 0s 313ms/step 1/1 [==============================] - 2s 2s/step [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.2min remaining: 35.6s 1/1 [==============================] - 0s 235ms/step 1/1 [==============================] - 0s 219ms/step 1/1 [==============================] - 0s 297ms/step 1/1 [==============================] - 0s 391ms/step [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.2min remaining: 18.3s 1/1 [==============================] - 0s 204ms/step [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 1.2min finished Done! Time (s): 76.0
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 53.6 | 9.0
Exporting bedpick plots... .jpg [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 25.8s remaining: 2.8min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 26.1s remaining: 1.2min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 26.2s remaining: 39.4s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 26.6s remaining: 23.2s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 45.0s remaining: 22.4s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 47.5s remaining: 11.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 48.2s finished Done! Time (s): 48.3
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 53.0 | 8.9
Automatically detecting shadows for 15 chunks: MODE: 1 | Remove all shadows... [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 854ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 5s 5s/step 1/1 [==============================] - 5s 5s/step 1/1 [==============================] - 6s 6s/step 1/1 [==============================] - 6s 6s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 44.1s remaining: 4.8min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 44.6s remaining: 2.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 44.8s remaining: 1.1min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 45.2s remaining: 39.5s 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 957ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 879ms/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.3min remaining: 39.5s 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 1s 939ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 893ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 894ms/step [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.4min remaining: 21.3s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 1.4min finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 54.6 | 9.1
Exporting sonogram tiles:
Exporting 15 sonograms for ds_highfreq[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 14.7s remaining: 1.6min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 14.8s remaining: 40.9s [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 15.1s remaining: 22.6s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 15.2s remaining: 13.3s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 27.0s remaining: 13.4s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 27.4s remaining: 6.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 27.7s finished
Exporting 30 sonograms for ss_port[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 1.1min remaining: 7.2min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 1.1min remaining: 3.1min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 1.1min remaining: 1.7min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 1.1min remaining: 59.4s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 2.0min remaining: 1.0min [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 2.0min remaining: 30.1s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 2.0min finished
Exporting 30 sonograms for ss_star[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 1.0min remaining: 6.6min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 1.1min remaining: 3.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 1.1min remaining: 1.7min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 1.1min remaining: 59.4s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.9min remaining: 57.0s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.9min remaining: 28.5s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 2.0min finished
Exporting 15 sonograms for ds_vhighfreq[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 14.1s remaining: 1.5min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 14.2s remaining: 39.3s [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 14.5s remaining: 21.9s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 14.6s remaining: 12.8s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 27.2s remaining: 13.6s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 27.6s remaining: 6.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 27.8s finished Done! Time (s): 296.5
BGunnett8
commented
1 year ago Sorry my prior paste didn't show the whole procedure for the first recording. This one shows everything all the way to the end of the process. +++++++++++++++++++++++++++++++++++++++++++ +++++++++++++++++++++++++++++++++++++++++++ Working On ./BenData2\R00021.DAT Start Time: 2023-06-12 17:11
READING
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 38.5 | 6.4
Getting DAT Metadata... Done! Time (s): 0.1
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 38.5 | 6.4
Getting Header Structure... Header Length: 72 ds_highfreq : Done! ss_port : Done! ss_star : Done! ds_vhighfreq : Done! Time (s): 0.0
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
3.5 | 38.5 | 6.4
Getting SON file header metadata... [Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers. [Parallel(n_jobs=4)]: Done 1 tasks | elapsed: 6.2s [Parallel(n_jobs=4)]: Done 2 out of 4 | elapsed: 6.2s remaining: 6.2s [Parallel(n_jobs=4)]: Done 4 out of 4 | elapsed: 6.3s remaining: 0.0s [Parallel(n_jobs=4)]: Done 4 out of 4 | elapsed: 6.3s finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
7.8 | 42.4 | 7.1
Locating missing pings and adding NoData... [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 8 | elapsed: 15.2s remaining: 45.8s [Parallel(n_jobs=8)]: Done 3 out of 8 | elapsed: 15.3s remaining: 25.5s [Parallel(n_jobs=8)]: Done 4 out of 8 | elapsed: 15.3s remaining: 15.3s [Parallel(n_jobs=8)]: Done 5 out of 8 | elapsed: 15.3s remaining: 9.2s [Parallel(n_jobs=8)]: Done 6 out of 8 | elapsed: 15.4s remaining: 5.1s [Parallel(n_jobs=8)]: Done 8 out of 8 | elapsed: 15.7s remaining: 0.0s [Parallel(n_jobs=8)]: Done 8 out of 8 | elapsed: 15.7s finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 46.4 | 7.8
Summary of Ping Metadata:
B001 : ds_highfreq Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 0 | 29460 | 14730.0 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 194.1 | 289.7 | 252.144 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 200.0 | 200.0 | 200.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 67027781.0 | 33508641.331 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.32 | 1417.718 | True orig_record_num | 0.0 | 29196.0 | 14623.5 | True
B002 : ss_port Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 2 | 29461 | 14732.001 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.139 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1050.0 | 1050.0 | 1050.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 65683064.0 | 32841751.489 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1388.906 | True orig_record_num | 1.0 | 29197.0 | 14625.48 | True
B003 : ss_star Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 3 | 29463 | 14732.991 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.146 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1050.0 | 1050.0 | 1050.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 65683064.0 | 32841751.489 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1388.953 | True orig_record_num | 1.0 | 29197.0 | 14626.46 | True
B004 : ds_vhighfreq Ping Count: 7366
Attribute | Minimum | Maximum | Average | Valid
record_num | 1 | 29462 | 14731.008 | True time_s | 0.0 | 1914.285 | 956.995 | True utm_e | -9546711.0 | -9542996.0 | -9544784.037 | True utm_n | 5272306.0 | 5273488.0 | 5272768.695 | True gps1 | 1.0 | 1.0 | 1.0 | True instr_heading | 192.0 | 289.7 | 252.139 | True gps2 | 1.0 | 1.0 | 1.0 | True speed_ms | 0.8 | 2.3 | 1.512 | True inst_dep_m | 0.8 | 3.5 | 2.053 | True volt_scale | 31.0 | 31.0 | 31.0 | True f | 1100.0 | 1100.0 | 1100.0 | True ping_cnt | 8927.0 | 9608.0 | 9020.227 | True index | 0.0 | 67027781.0 | 33508641.331 | True chunk_id | 0 | 14 | 6.873 | True lon | -85.756 | -85.723 | -85.739 | True lat | 42.93 | 42.938 | 42.933 | True e | 601496.469 | 604206.467 | 602903.848 | True n | 4753798.261 | 4754705.459 | 4754158.494 | True tempC | 23.5 | 23.5 | 23.5 | True t | 0.108 | 0.108 | 0.108 | True pix_m | 0.019 | 0.019 | 0.019 | True date | 2022-08-31 | 2022-08-31 | - | True time | 10:35:11 | 11:07:05 | - | True trk_dist | 0.0 | 2893.424 | 1417.77 | True orig_record_num | 1.0 | 29197.0 | 14624.5 | True
Done! Time (s): 25.8
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 46.4 | 7.8
Automatically estimating depth for 15 chunks:
Using Zheng et al. 2021 method. Loading model: bedpick_ZhengApproach_20220629.h5[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 0s 361ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 0s 359ms/step [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 41.5s remaining: 4.5min 1/1 [==============================] - 0s 423ms/step 1/1 [==============================] - 1s 595ms/step 1/1 [==============================] - 1s 611ms/step 1/1 [==============================] - 0s 314ms/step [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 42.7s remaining: 2.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 43.1s remaining: 1.1min 1/1 [==============================] - 0s 425ms/step 1/1 [==============================] - 0s 425ms/step [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 43.9s remaining: 38.4s 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 0s 261ms/step 1/1 [==============================] - 1s 910ms/step 1/1 [==============================] - 1s 987ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 0s 313ms/step 1/1 [==============================] - 2s 2s/step [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.2min remaining: 35.6s 1/1 [==============================] - 0s 235ms/step 1/1 [==============================] - 0s 219ms/step 1/1 [==============================] - 0s 297ms/step 1/1 [==============================] - 0s 391ms/step [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.2min remaining: 18.3s 1/1 [==============================] - 0s 204ms/step [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 1.2min finished Done! Time (s): 76.0
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 53.6 | 9.0
Exporting bedpick plots... .jpg [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 25.8s remaining: 2.8min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 26.1s remaining: 1.2min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 26.2s remaining: 39.4s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 26.6s remaining: 23.2s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 45.0s remaining: 22.4s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 47.5s remaining: 11.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 48.2s finished Done! Time (s): 48.3
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 53.0 | 8.9
Automatically detecting shadows for 15 chunks: MODE: 1 | Remove all shadows... [Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 854ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 5s 5s/step 1/1 [==============================] - 5s 5s/step 1/1 [==============================] - 6s 6s/step 1/1 [==============================] - 6s 6s/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 2s 2s/step [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 44.1s remaining: 4.8min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 44.6s remaining: 2.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 44.8s remaining: 1.1min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 45.2s remaining: 39.5s 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 957ms/step 1/1 [==============================] - 2s 2s/step 1/1 [==============================] - 1s 879ms/step 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 3s 3s/step [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.3min remaining: 39.5s 1/1 [==============================] - 3s 3s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 4s 4s/step 1/1 [==============================] - 1s 939ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 893ms/step 1/1 [==============================] - 1s 1s/step 1/1 [==============================] - 1s 894ms/step [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.4min remaining: 21.3s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 1.4min finished
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 54.6 | 9.1
Exporting sonogram tiles:
Exporting 15 sonograms for ds_highfreq[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 14.7s remaining: 1.6min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 14.8s remaining: 40.9s [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 15.1s remaining: 22.6s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 15.2s remaining: 13.3s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 27.0s remaining: 13.4s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 27.4s remaining: 6.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 27.7s finished
Exporting 30 sonograms for ss_port[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 1.1min remaining: 7.2min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 1.1min remaining: 3.1min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 1.1min remaining: 1.7min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 1.1min remaining: 59.4s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 2.0min remaining: 1.0min [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 2.0min remaining: 30.1s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 2.0min finished
Exporting 30 sonograms for ss_star[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 1.0min remaining: 6.6min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 1.1min remaining: 3.0min [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 1.1min remaining: 1.7min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 1.1min remaining: 59.4s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 1.9min remaining: 57.0s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 1.9min remaining: 28.5s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 2.0min finished
Exporting 15 sonograms for ds_vhighfreq[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 14.1s remaining: 1.5min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 14.2s remaining: 39.3s [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 14.5s remaining: 21.9s [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 14.6s remaining: 12.8s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 27.2s remaining: 13.6s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 27.6s remaining: 6.8s [Parallel(n_jobs=8)]: Done 15 out of 15 | elapsed: 27.8s finished Done! Time (s): 296.5
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 51.2 | 8.6
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
0.0 | 51.0 | 8.5
RECTIFYING
Smoothing trackline... Done! Time (s): 0.4
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
4.9 | 51.1 | 8.5
Calculating, smoothing, and interpolating range extent coordinates... [Parallel(n_jobs=2)]: Using backend LokyBackend with 2 concurrent workers. [Parallel(n_jobs=2)]: Done 2 out of 2 | elapsed: 3.2s remaining: 0.0s [Parallel(n_jobs=2)]: Done 2 out of 2 | elapsed: 3.2s finished Done! Time (s): 4.4
Current CPU/RAM Usage:
CPU % | RAM % | RAM [GB]
1.2 | 37.0 | 6.2
Rectifying and exporting GeoTiffs:
Exporting 15 GeoTiffs for ss_port[Parallel(n_jobs=8)]: Using backend LokyBackend with 8 concurrent workers. [Parallel(n_jobs=8)]: Done 2 out of 15 | elapsed: 20.7s remaining: 2.3min [Parallel(n_jobs=8)]: Done 4 out of 15 | elapsed: 21.5s remaining: 59.4s [Parallel(n_jobs=8)]: Done 6 out of 15 | elapsed: 44.0s remaining: 1.1min [Parallel(n_jobs=8)]: Done 8 out of 15 | elapsed: 44.2s remaining: 38.7s [Parallel(n_jobs=8)]: Done 10 out of 15 | elapsed: 48.8s remaining: 24.3s [Parallel(n_jobs=8)]: Done 12 out of 15 | elapsed: 54.2s remaining: 13.5s Could not process: ./BenData2\R00021.DAT
Total Processing Time: 0:15:29
CameronBodine
commented
1 year ago Ok, it appears there is an issue during the rectification process. This is usually due to some issue with the recording itself but there may be an issue with the script also. Can you provide me with the problematic recording? You can upload it to my gdrive:
https://drive.google.com/open?id=1WdkzIydG67zWYf5rW7g1fWFQWnM8BsLx&usp=drive_fs
BGunnett8
commented
1 year ago I sent you the first recording of the batch that I've been trying to work with. I can send more of them if you like, but I figured since none of them are working that just sending the first one over would be good enough for you to analyze.
CameronBodine
commented
1 year ago Hi @BGunnett8, you were experiencing an issue rectifying and mosaicing your imagery because the range was set to a very large value (looks like 180 meters?). An example of a bedpick plot is shown below.
This makes the image prohibitively large to rectify and can't be fit into memory. I added a new option called cropRange to crop the range to a value, in meters, you can specify.
I tested on a value of 30 and was able to successfully export mosaics. Here's the same bedpick plot I showed above cropped to 30 m:
To update PING-Mapper, navigate to the PING-Mapper directory and run git pull in the console.
When you collect recordings in the future, I recommend setting the range to the maximum extent that you are able to see returns from the bed. When we scan rivers, I find a value of 30-50 meters per side is about the maximum usable extent.
BGunnett8
commented
1 year ago Thank you very much for making this change? Was it an error that we made when recording with the Helix 7? There's a setting we should change? Also, when I ran git pull in the console, I got this which seems to be an error.
error: Your local changes to the following files would be overwritten by merge: main.py main_batchDirectory.py Please commit your changes or stash them before you merge. Aborting Updating 73aca8b..a019b55
CameronBodine
commented
1 year ago Try running git stash then git pull.
It is a setting you make while recording with the Helix. It's possible you have the unit set to auto range or maybe auto depth. I recommend looking at the manual for instructions on how to fix the range to a specific value. In the screen shot below, the range is set to 150 ft per side.
BGunnett8
commented
1 year ago That worked, I'll run all my files again with crop range and let you know how it goes. Thanks for the info on the helix as well.
BGunnett8
commented
1 year ago Most of them worked this time but a few still didn't produce mosaics. I posted recordings 22, 24, 29, 40, 41 and 42 in your google drive. These still had the same error of showing all of the data separately but failing to produce mosaics. The whole batch was run with CropRange = 30.
CameronBodine
commented
1 year ago Can you run only one of the problematic recordings with main.py and post the error? Thanks!
CameronBodine
commented
1 year ago I was just able to process mosaics from the new set of recordings you provided. I believe you are experiencing an OOM (out of memor) error. I noticed from your original console output that your computer has 16 GB of RAM. My computer has more and is thus able to export the mosaics. There are two parameters you can adjust on your computer to successfully export these mosaics:
The first is the the nchunk parameter. Specifying a smaller nchunk value will reduce the number of pings that are loaded into memory at once. You could try a value of 150 or 250.
The second parameter is threadCnt. PING-Mapper uses multithreading to speed up the export process. A value of 0 will use all threads available on your computer, meaning loading more data into memory at once. Using a negative value will reduce the number of threads to use by that much, or you can fix it at a smaller value, say 1 to 4, to specify the exact number of threads.
If you run into further issues, please try processing the problematic recording with main.py instead of the batch processing script. There will be an error printed which you can provide to help me identify the issue. The batch script is designed to process what it can, even if there is an error, and will attempt to process all the recordings in the parent directory.
Let me know how it goes for you!
BGunnett8
commented
1 year ago Sorry for the delay in response. I used main.py for all 6 recordings and it worked for 2 of them. This is the error that I get for two of them. What does this mean?
Traceback (most recent call last):
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\site-packages\joblib\externals\loky\process_executor.py", line 428, in _process_worker
r = call_item()
^^^^^^^^^^^
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\site-packages\joblib\externals\loky\process_executor.py", line 275, in call
return self.fn(*self.args, self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\site-packages\joblib_parallel_backends.py", line 620, in call
return self.func(*args, *kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 288, in call
return [func(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 288, in
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\BGunn\PingMapper\main.py", line 134, in
CameronBodine
commented
1 year ago Thank you for the console output! It's an out of memory issue, as I suspected. Did you try adjusting the nchunk or threadCnt parameters? If you did, I recommend trying smaller values to see if you can get the rectified images to export.
CameronBodine
commented
1 year ago FYI, the last portion of your console output indicates that there is not enough memory available to allocate the array:
File "C:\Users\BGunn\miniconda3\envs\ping\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.20 GiB for an array with shape (80553616, 2) and data type float64I made the alterations and I was able to get images for everything! Thank you so much for all of your help and responsiveness!
CameronBodine
commented
1 year ago Glad to hear it! Happy scanning and mapping!
CameronBodine
commented
1 year ago I found the memory leak! You should be able to process the problematic recordings without modifying nchunk or threadCnt after updating with git pull. It is possible you may still get OOM error, but this fix should lower the chances of that.
Hello,
This software has worked wonderfully so far but with the last batch of data it failed to give me the mosaic images for both water column present and absent. Everything else is there. In the miniconda window I do the following:
1.cd PingMapper
and it begins running my batch of files like normal but no mosaics. I've attached images of the files that it does produce and the code that I'm using. Usually under the files that you see in the screenshot is where it puts the mosaic images but not this time. Any help would be appreciated. Thank you!
