We have messaged before about this same issue however back then it was with PingMapper not producing mosaics. You had me alter the nchunk to 250 and croprange to 30 and the issue was solved. Now that PingMapper 2.0 is out, I decided to run the same files to get substrate classification. The same exact set of files that were not producing mosaics are now not returning any substrate data, it doesn't even make a folder labelled substrate like every other file. I did alter the nchunk to 250 in the substrate section of code. I'm using main_batchDirectory.py for these files. Thanks for your help in the past and hopefully you know the solution to this problem as well. This is what the code looks like for me:
Part of PING-Mapper software
#
Co-Developed by Cameron S. Bodine and Dr. Daniel Buscombe
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
#
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
#
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
import sys
sys.path.insert(0, 'src')
from funcs_common import *
from main_readFiles import read_master_func
from main_rectify import rectify_master_func
from main_mapSubstrate import map_master_func
import time
import datetime
Get processing script's dir so we can save it to file
scriptDir = os.getcwd()
For the logfile
oldOutput = sys.stdout
============================================
#######################
Start User Parameters
#######################
Path to data/output
inDir = r'./BenData4' # Parent folder of sonar recordings
outDir = r'./BenDataOut4' # Parent folder for export files
* IMPORTANT **
Export Mode: project_mode
0==NEW PROJECT: Create a new project. [DEFAULT]
If project already exists, program will exit without any project changes.
1==OVERWRITE MODE: Create new project, regardless of previous project state.
If project exists, it will be DELETED and reprocessed.
If project does not exist, a new project will be created.
project_mode = 1
General Parameters
tempC = 23.5 #Temperature in Celsius
nchunk = 250 #Number of pings per chunk
cropRange = 30 #Crop imagery to specified range [in meters]; 0.0==No Cropping
exportUnknown = False #Option to export Unknown ping metadata
fixNoDat = True # Locate and flag missing pings; add NoData to exported imagery.
threadCnt = 0 #Number of compute threads to use; 0==All threads; <0==(Total threads + threadCnt); >0==Threads to use up to total threads
Position Corrections
Provide an x and y offset to account for position offset between
control head (or external GPS) and transducer.
Origin (0,0) is the location of control head (or external GPS)
X-axis runs from bow (fore, or front) to stern (aft, or rear) with positive offset towards the bow, negative towards stern
Y-axis runs from portside (left) to starboard (right), with negative values towards the portside, positive towards starboard
egn = True
egn_stretch = 1 # 0==Min-Max; 1==% Clip
egn_stretch_factor = 0.5 # If % Clip, the percent of histogram tails to clip (1.0 == 1%)
Sonogram Exports
tileFile = '.jpg' # Img format for plots and sonogram exports
wcp = True #Export tiles with water column present: 0==False; 1==True, side scan channels only; 2==True, all available channels.
wcr = True #Export Tiles with water column removed (and slant range corrected): 0==False; 1==True, side scan channels only; 2==True, all available channels.
Speed corrected sonogram Exports
lbl_set = 2 # Export images for labeling: 0==False; 1==True, keep water column & shadows; 2==True, remove water column & shadows
spdCor = 1 # Speed correction: 0==No Speed Correction; 1==Stretch by GPS distance; !=1 or !=0 == Stretch factor.
maxCrop = False # True==Ping-wise crop; False==Crop tile to max range.
Depth Detection and Shadow Removal Parameters
remShadow = 2 # 0==Leave Shadows; 1==Remove all shadows; 2==Remove only bank shadows
detectDep = 1 #0==Use Humminbird depth; 1==Auto detect depth w/ Zheng et al. 2021;
2==Auto detect depth w/ Thresholding
smthDep = True #Smooth depth before water column removal
adjDep = 0 #Aditional depth adjustment (in pixels) for water column removaL
pltBedPick = True #Plot bedpick on sonogram
Rectification Parameters
rect_wcp = True #Export rectified tiles with water column present
rect_wcr = True #Export rectified tiles with water column removed/slant range corrected
son_colorMap = 'Greys_r' # Specify colorramp for rectified imagery. '_r'==reverse the ramp: https://matplotlib.org/stable/tutorials/colors/colormaps.html
Substrate Mapping
pred_sub = 1 # Automatically predict substrates and save to npz: 0==False; 1==True, SegFormer Model
pltSubClass = True # Export plots of substrate classification and predictions
map_sub = 1 # Export substrate maps (as rasters): 0==False; 1==True. Requires substrate predictions saved to npz.
export_poly = True # Convert substrate maps to shapefile: map_sub must be > 0 or raster maps previously exported
map_class_method = 'max' # 'max' only current option. Take argmax of substrate predictions to get final classification.
Mosaic Exports
pix_res = 0 # Pixel resolution [meters]: 0 = Default (~0.02 m). ONLY APPLIES TO MOSAICS
mosaic_nchunk = 250 # Number of chunks per mosaic: 0=All chunks. Specifying a value >0 generates multiple mosaics if number of chunks exceeds mosaic_nchunk.
mosaic = 1 #Export sonar mosaic; 0==Don't Mosaic; 1==Do Mosaic - GTiff; 2==Do Mosaic - VRT
map_mosaic = 1 #Export substrate mosaic; 0==Don't Mosaic; 1==Do Mosaic - GTiff; 2==Do Mosaic - VRT
#####################
End User Parameters
#####################
============================================
Find all DAT and SON files in all subdirectories of inDir
inFiles=[]
for root, dirs, files in os.walk(inDir):
for file in files:
if file.endswith('.DAT'):
inFiles.append(os.path.join(root, file))
inFiles = sorted(inFiles)
for i, f in enumerate(inFiles):
print(i, ":", f)
for datFile in inFiles:
try:
copied_scriptname = os.path.basename(file).split('.')[0]+''+time.strftime("%Y-%m-%d_%H%M")+'.py'
script = os.path.join(scriptDir, os.path.basename(file))
logfilename = 'log_'+time.strftime("%Y-%m-%d_%H%M")+'.txt'
start_time = time.time()
inPath = os.path.dirname(datFile)
humFile = datFile
recName = os.path.basename(humFile).split('.')[0]
sonPath = humFile.split('.DAT')[0]
sonFiles = sorted(glob(sonPath+os.sep+'*.SON'))
projDir = os.path.join(outDir, recName)
params = {
'logfilename':logfilename,
'project_mode':project_mode,
'script':[script, copied_script_name],
'humFile':humFile,
'sonFiles':sonFiles,
'projDir':projDir,
'tempC':tempC,
'nchunk':nchunk,
'exportUnknown':exportUnknown,
'fixNoDat':fixNoDat,
'threadCnt':threadCnt,
'x_offset':x_offset,
'y_offset':y_offset,
'egn':egn,
'egn_stretch':egn_stretch,
'egn_stretch_factor':egn_stretch_factor,
'tileFile':tileFile,
'wcp':wcp,
'wcr':wcr,
'lbl_set':lbl_set,
'spdCor':spdCor,
'maxCrop':maxCrop,
'USE_GPU':False,
'remShadow':remShadow,
'detectDep':detectDep,
'smthDep':smthDep,
'adjDep':adjDep,
'pltBedPick':pltBedPick,
'rect_wcp':rect_wcp,
'rect_wcr':rect_wcr,
'son_colorMap':son_colorMap,
'pred_sub':pred_sub,
'map_sub':map_sub,
'export_poly':export_poly,
'pltSubClass':pltSubClass,
'map_class_method':map_class_method,
'pix_res':pix_res,
'mosaic_nchunk':mosaic_nchunk,
'mosaic':mosaic,
'map_mosaic':map_mosaic
}
globals().update(params)
# =========================================================
# Determine project_mode
print(project_mode)
if project_mode == 0:
# Create new project
if not os.path.exists(projDir):
os.mkdir(projDir)
else:
projectMode_1_inval()
elif project_mode == 1:
# Overwrite existing project
if os.path.exists(projDir):
shutil.rmtree(projDir)
os.mkdir(projDir)
elif project_mode == 2:
# Update project
# Make sure project exists, exit if not.
if not os.path.exists(projDir):
projectMode_2_inval()
# =========================================================
# For logging the console output
logdir = os.path.join(projDir, 'meta', 'logs')
if not os.path.exists(logdir):
os.makedirs(logdir)
logfilename = os.path.join(logdir, logfilename)
sys.stdout = Logger(logfilename)
for k,v in params.items():
print(k, v)
print('sonPath',sonPath)
print('\n\n\n+++++++++++++++++++++++++++++++++++++++++++')
print('+++++++++++++++++++++++++++++++++++++++++++')
print('***** Working On *****')
print(humFile)
print('Start Time: ', datetime.datetime.now().strftime('%Y-%m-%d %H:%M'))
print('\n===========================================')
print('===========================================')
print('***** READING *****')
read_master_func(**params)
# read_master_func(sonFiles, humFile, projDir, t, nchunk, exportUnknown, wcp, wcr, detectDepth, smthDep, adjDep, pltBedPick, threadCnt)
if rect_wcp or rect_wcr:
print('\n===========================================')
print('===========================================')
print('***** RECTIFYING *****')
rectify_master_func(**params)
# rectify_master_func(sonFiles, humFile, projDir, nchunk, rect_wcp, rect_wcr, mosaic, threadCnt)
#==================================================
#==================================================
if pred_sub or map_sub or export_poly or pltSubClass:
print('\n===========================================')
print('===========================================')
print('***** MAPPING SUBSTRATE *****')
print("working on "+projDir)
map_master_func(**params)
sys.stdout.log.close()
except Exception as Argument:
unableToProcessError(logfilename)
print('\n\nCould not process:', datFile)
sys.stdout = oldOutput
gc.collect()
print("\n\nTotal Processing Time: ",datetime.timedelta(seconds = round(time.time() - start_time, ndigits=0)))
Hello,
We have messaged before about this same issue however back then it was with PingMapper not producing mosaics. You had me alter the nchunk to 250 and croprange to 30 and the issue was solved. Now that PingMapper 2.0 is out, I decided to run the same files to get substrate classification. The same exact set of files that were not producing mosaics are now not returning any substrate data, it doesn't even make a folder labelled substrate like every other file. I did alter the nchunk to 250 in the substrate section of code. I'm using main_batchDirectory.py for these files. Thanks for your help in the past and hopefully you know the solution to this problem as well. This is what the code looks like for me:
Part of PING-Mapper software
#
Co-Developed by Cameron S. Bodine and Dr. Daniel Buscombe
#
Inspired by PyHum: https://github.com/dbuscombe-usgs/PyHum
#
MIT License
#
Copyright (c) 2022-23 Cameron S. Bodine
#
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
#
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
#
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
import sys sys.path.insert(0, 'src')
from funcs_common import * from main_readFiles import read_master_func from main_rectify import rectify_master_func from main_mapSubstrate import map_master_func
import time import datetime
Get processing script's dir so we can save it to file
scriptDir = os.getcwd()
For the logfile
oldOutput = sys.stdout
============================================
#######################
Start User Parameters
#######################
Path to data/output
inDir = r'./BenData4' # Parent folder of sonar recordings outDir = r'./BenDataOut4' # Parent folder for export files
* IMPORTANT **
Export Mode: project_mode
0==NEW PROJECT: Create a new project. [DEFAULT]
If project already exists, program will exit without any project changes.
1==OVERWRITE MODE: Create new project, regardless of previous project state.
If project exists, it will be DELETED and reprocessed.
If project does not exist, a new project will be created.
project_mode = 1
General Parameters
tempC = 23.5 #Temperature in Celsius nchunk = 250 #Number of pings per chunk cropRange = 30 #Crop imagery to specified range [in meters]; 0.0==No Cropping exportUnknown = False #Option to export Unknown ping metadata fixNoDat = True # Locate and flag missing pings; add NoData to exported imagery. threadCnt = 0 #Number of compute threads to use; 0==All threads; <0==(Total threads + threadCnt); >0==Threads to use up to total threads
Position Corrections
Provide an x and y offset to account for position offset between
control head (or external GPS) and transducer.
Origin (0,0) is the location of control head (or external GPS)
X-axis runs from bow (fore, or front) to stern (aft, or rear) with positive offset towards the bow, negative towards stern
Y-axis runs from portside (left) to starboard (right), with negative values towards the portside, positive towards starboard
Z-offsets can be provided with
adjDep
below.x_offset = 0.0 # [meters] y_offset = 0.0 # [meters]
Sonar Intensity Corrections
egn = True egn_stretch = 1 # 0==Min-Max; 1==% Clip egn_stretch_factor = 0.5 # If % Clip, the percent of histogram tails to clip (1.0 == 1%)
Sonogram Exports
tileFile = '.jpg' # Img format for plots and sonogram exports wcp = True #Export tiles with water column present: 0==False; 1==True, side scan channels only; 2==True, all available channels. wcr = True #Export Tiles with water column removed (and slant range corrected): 0==False; 1==True, side scan channels only; 2==True, all available channels.
Speed corrected sonogram Exports
lbl_set = 2 # Export images for labeling: 0==False; 1==True, keep water column & shadows; 2==True, remove water column & shadows spdCor = 1 # Speed correction: 0==No Speed Correction; 1==Stretch by GPS distance; !=1 or !=0 == Stretch factor. maxCrop = False # True==Ping-wise crop; False==Crop tile to max range.
Depth Detection and Shadow Removal Parameters
remShadow = 2 # 0==Leave Shadows; 1==Remove all shadows; 2==Remove only bank shadows detectDep = 1 #0==Use Humminbird depth; 1==Auto detect depth w/ Zheng et al. 2021;
2==Auto detect depth w/ Thresholding
smthDep = True #Smooth depth before water column removal adjDep = 0 #Aditional depth adjustment (in pixels) for water column removaL pltBedPick = True #Plot bedpick on sonogram
Rectification Parameters
rect_wcp = True #Export rectified tiles with water column present rect_wcr = True #Export rectified tiles with water column removed/slant range corrected son_colorMap = 'Greys_r' # Specify colorramp for rectified imagery. '_r'==reverse the ramp: https://matplotlib.org/stable/tutorials/colors/colormaps.html
Substrate Mapping
pred_sub = 1 # Automatically predict substrates and save to npz: 0==False; 1==True, SegFormer Model pltSubClass = True # Export plots of substrate classification and predictions map_sub = 1 # Export substrate maps (as rasters): 0==False; 1==True. Requires substrate predictions saved to npz. export_poly = True # Convert substrate maps to shapefile: map_sub must be > 0 or raster maps previously exported map_class_method = 'max' # 'max' only current option. Take argmax of substrate predictions to get final classification.
Mosaic Exports
pix_res = 0 # Pixel resolution [meters]: 0 = Default (~0.02 m). ONLY APPLIES TO MOSAICS mosaic_nchunk = 250 # Number of chunks per mosaic: 0=All chunks. Specifying a value >0 generates multiple mosaics if number of chunks exceeds mosaic_nchunk. mosaic = 1 #Export sonar mosaic; 0==Don't Mosaic; 1==Do Mosaic - GTiff; 2==Do Mosaic - VRT map_mosaic = 1 #Export substrate mosaic; 0==Don't Mosaic; 1==Do Mosaic - GTiff; 2==Do Mosaic - VRT
#####################
End User Parameters
#####################
============================================
Find all DAT and SON files in all subdirectories of inDir
inFiles=[] for root, dirs, files in os.walk(inDir): for file in files: if file.endswith('.DAT'): inFiles.append(os.path.join(root, file))
inFiles = sorted(inFiles)
for i, f in enumerate(inFiles): print(i, ":", f)
for datFile in inFiles: try: copied_scriptname = os.path.basename(file).split('.')[0]+''+time.strftime("%Y-%m-%d_%H%M")+'.py' script = os.path.join(scriptDir, os.path.basename(file))