dwcaress / MB-System

MB-System is an open source software package for the processing and display of bathymetry and backscatter imagery data derived from multibeam, interferometry, and sidescan sonars.
https://www.mbari.org/products/research-software/mb-system/
Other
124 stars 42 forks source link

Update Kongsberg kmall format to rev. G #995

Closed anbj closed 4 years ago

anbj commented 4 years ago

The Kongsberg kmall format has been revised and is now in rev G (01.11.2019). As far as I can tell the implementation of the KMall-format in MB-System is based on rev. F of the datagram description.

I've tried to use MB-System 5.7.6beta31 with kmall-files in presumably rev. G, and getting into trouble. Sample file can be supplied if necessary.

E.g. 1

$ mbnavlist -I file.kmall

The MBF_KEMKMALL module skipped data between identified
data records. Something is broken, most likely the data...
However, the data may include a data record type that we
haven't seen yet, or there could be an error in the code.
If skipped data are reported multiple times, we recommend
you post a problem description through the discussion list
available at https://listserver.mbari.org/sympa/arc/mbsystem
and make a data sample available.
Have a nice day...
MBF_KEMKMALL skipped 25568 bytes before record #CHE at file pos 28818
Segmentation fault (core dumped)

e.g. 2

$ mbnavlist -I file2.kmall

The MBF_KEMKMALL module skipped data between identified
data records. Something is broken, most likely the data...
However, the data may include a data record type that we
haven't seen yet, or there could be an error in the code.
If skipped data are reported multiple times, we recommend
you post a problem description through the discussion list
available at https://listserver.mbari.org/sympa/arc/mbsystem
and make a data sample available.
Have a nice day...
MBF_KEMKMALL skipped 45870 bytes before record #SKM at file pos 52582

The datagram description may be downloaded from Kongsbergs website; https://www.kongsberg.com/maritime/support/document-and-downloads/software-downloads/; "KMALL - Datagram description Rev:G". The changes between rev. F to rev. G is (quote from the document):

Rev G. 01 NOV 2019
Code changes:

#FCF datagram, new datagram for backscatter calibration

#MRZ datagram, updated to version 1
Line 880: float bsCorrectionOffset_dB - Backscatter offset set in the installation menu.
Line 881: uint8_t lambertsLawApplied - Beam intensity data corrected as seabed image data (Lambert and normal incidence corrections).
Line 882: uint8_t iceWindow - Ice window installed.
Line 883: uint16_t padding4 - Padding for byte alignment.
Line 905: Description changed: float txNominalSourceLevel_dB; - Actual SL = txNominalSourceLevel_dB + highVoltageLevel_dB. Unit dB re 1 microPascal.
Line 916: float highVoltageLevel_dB - 20log(Measured high voltage power level at TX pulse or Nominal high voltage power level). This parameter will also include the effect of user selected transmit power reduction (transmitPower_dB) and mammal protection. Actual SL = txNominalSourceLevel_dB + highVoltageLevel_dB. Unit dB.
Line 917: float sectorTrackingCorr_dB - Backscatter correction added in sector tracking mode. Unit dB.
Line 918: float effectiveSignalLength_sec - Signal length used for backscatter footprint calculation. This compensates for the TX pulse tapering and the RX filter bandwidths. Unit second.

#MWC datagram, updated to version 1
Line 1163: float detectedRangeInSamplesHighResolution - The same information as in detectedRangeInSamples with higher resolution. Two way range in samples. Approximation to calculated distance from tx to bottom detection [meters] = soundVelocity_mPerSec * detectedRangeInSamples / (sampleFreq_Hz * 2). The detected range is set to zero when the beam has no bottom detection.

#SVT datagram
Line 515, 516: Kongsberg sound velocity datagrams KSSIS 80 and KSSIS 43 added.
dwcaress commented 4 years ago

Thanks for raising this issue. Please go ahead and make a data sample available. What system was used to collect the problem data?

Cheers, Dave

On Mar 2, 2020, at 3:40 AM, anbj notifications@github.com wrote:

The Kongsberg kmall format has been revised and is now in rev G (01.11.2019). As far as I can tell the implementation of the KMall-format in MB-System is based on rev. F of the datagram description.

I've tried to use MB-System 5.7.6beta31 with kmall-files in presumably rev. G, and getting into trouble. Sample file can be supplied if necessary.

E.g. 1

$ mbnavlist -I file.kmall

The MBF_KEMKMALL module skipped data between identified data records. Something is broken, most likely the data... However, the data may include a data record type that we haven't seen yet, or there could be an error in the code. If skipped data are reported multiple times, we recommend you post a problem description through the discussion list available at https://listserver.mbari.org/sympa/arc/mbsystem and make a data sample available. Have a nice day... MBF_KEMKMALL skipped 25568 bytes before record #CHE at file pos 28818 Segmentation fault (core dumped)

e.g. 2

$ mbnavlist -I file2.kmall

The MBF_KEMKMALL module skipped data between identified data records. Something is broken, most likely the data... However, the data may include a data record type that we haven't seen yet, or there could be an error in the code. If skipped data are reported multiple times, we recommend you post a problem description through the discussion list available at https://listserver.mbari.org/sympa/arc/mbsystem and make a data sample available. Have a nice day... MBF_KEMKMALL skipped 45870 bytes before record #SKM at file pos 52582

The datagram description may be downloaded from Kongsbergs website; https://www.kongsberg.com/maritime/support/document-and-downloads/software-downloads/; "KMALL - Datagram description Rev:G". The changes between rev. F to rev. G is (quote from the document):

Rev G. 01 NOV 2019 Code changes:

FCF datagram, new datagram for backscatter calibration

MRZ datagram, updated to version 1

Line 880: float bsCorrectionOffset_dB - Backscatter offset set in the installation menu. Line 881: uint8_t lambertsLawApplied - Beam intensity data corrected as seabed image data (Lambert and normal incidence corrections). Line 882: uint8_t iceWindow - Ice window installed. Line 883: uint16_t padding4 - Padding for byte alignment. Line 905: Description changed: float txNominalSourceLevel_dB; - Actual SL = txNominalSourceLevel_dB + highVoltageLevel_dB. Unit dB re 1 microPascal. Line 916: float highVoltageLevel_dB - 20log(Measured high voltage power level at TX pulse or Nominal high voltage power level). This parameter will also include the effect of user selected transmit power reduction (transmitPower_dB) and mammal protection. Actual SL = txNominalSourceLevel_dB + highVoltageLevel_dB. Unit dB. Line 917: float sectorTrackingCorr_dB - Backscatter correction added in sector tracking mode. Unit dB. Line 918: float effectiveSignalLength_sec - Signal length used for backscatter footprint calculation. This compensates for the TX pulse tapering and the RX filter bandwidths. Unit second.

MWC datagram, updated to version 1

Line 1163: float detectedRangeInSamplesHighResolution - The same information as in detectedRangeInSamples with higher resolution. Two way range in samples. Approximation to calculated distance from tx to bottom detection [meters] = soundVelocity_mPerSec detectedRangeInSamples / (sampleFreq_Hz 2). The detected range is set to zero when the beam has no bottom detection.

SVT datagram

Line 515, 516: Kongsberg sound velocity datagrams KSSIS 80 and KSSIS 43 added.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.


David W. Caress Principal Engineer Seafloor Mapping Lab

Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039

caress@mbari.org http://www.mbari.org/~caress/

Phone: 831-775-1775

anbj commented 4 years ago

The following KMall-file was collected by EM712. The file also contains WC. I've been told that the data is not perfect, but that it will capable of verifying the ability to read the KMall-format.

Link to file (3.65 GB): https://www.dropbox.com/s/9z0ko010em24reh/0003_20191128_085424_ShipName.kmall?dl=0

dwcaress commented 4 years ago

We just received revision H of the Kongsberg format spec plus new data samples. We will be working on addressing both the prior changes raised above and the new additions.

anbj commented 4 years ago

Great!

anbj commented 4 years ago

Any thoughts on when this will be implemented? I’m hoping before June (cruise time)!

dwcaress commented 4 years ago

I'm trying to get to it, but other things keep coming up. Updates for both the Kongsberg and Teledyne formats are a requirement for the next non-beta release, so it is a priority. I will commit to before the end of May.

dwcaress commented 4 years ago

Thanks for letting me know your timeline - it's probably best to bug me again in two weeks.

anbj commented 4 years ago

A bit early, but...

ping!

anbj commented 4 years ago

ping!

anbj commented 4 years ago

ping!

(Sorry David!)

dwcaress commented 4 years ago

No problem, I’m actively working on it.

Dave

On May 28, 2020, at 12:59 AM, anbj notifications@github.com wrote:

ping!

(Sorry David!)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.


David W. Caress Principal Engineer Seafloor Mapping Lab

Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039

caress@mbari.org http://www.mbari.org/~caress/

Phone: 831-775-1775

anbj commented 4 years ago

Cheers!

dwcaress commented 4 years ago

Pre-release 5.7.6beta38 following commit #1103 brings the kmall (format 261) i/o module up to date with Kongsberg format specification revision G. Most processing tasks are working correctly, but there are still problems with recalculating bathymetry by raytracing through a sound speed model. Apologies that this has taken so long (and is still not fully complete). The complexity results from trying to make it all work on all types of platforms (vessels and submerged vehicles) with all the different sonars. I'll continue working on the bathymetry recalculation - at least some of the problem is getting the time latencies and bias parameters right to correct the the attitude and navigation used in the calculation - in some data samples the values that work seem to conflict with those found in the installation parameters datagram.

Here is a sample workflow:

-------------------------------------------------------------------------------

#

Example MB-System processing of Kongsberg kmall format data

#

David W. Caress

Monterey Bay Aquarium Research Institute

8 June 2020

#

-------------------------------------------------------------------------------

#

These notes pertain to Kongsberg multibeam data logged in files with suffixes

of ".kmall". These data are supported by MB-System format:

MBF_KEMKMALL 261 => *.kmall files, new Kongsberg format for EM124, EM304, EM712, EM2040

#

-------------------------------------------------------------------------------

Preprocess Stage: Format 261 files with *.kmall suffix

-------------------------------------------------------------------------------

#

MB-System now recognizes files with the *.kmall suffix as format 261.

Preprocessing is required, but there is no format conversion - the raw files

are still in format 261. Also, there is no need to extract or specify a

platform model.

Get datalist of the logged swath files using

chmod -x *kmall mbm_makedatalist -S.kmall -Odatalistl.mb-1 -P mbdatalist -I datalistl.mb-1 -O -V #

Preprocess the data - merge asynchronous data (position, depth, heading, attitude)

from navigation post processing

mbpreprocess \ --input=datalistl.mb-1 \ --verbose #

Get datalist of raw *.mb261 files

mbm_makedatalist -S.mb261 -P -V mbdatalist -Z

-------------------------------------------------------------------------------

Processing: tide correction, bathymetry editing, backscatter correction

-------------------------------------------------------------------------------

Get tide models and set for use by mbprocess

mbotps -I datalist.mb-1 -M -D60.0 -V

Set recalculation of bathymetry by raytracing using SVPs embedded in files

- recalculation is currently not working correctly

mbsvplist -P -V

Process the data

mbm_multiprocess

Edit bathymetry

mbedit

Autofilter bathymetry (-Y-100/100 trims swath outside 100 meters from nadir)

mbclean -Y-100/100

3D bathymetry editor

mbeditviz -I datalist.mb-1

Process the data

mbm_multiprocess -X2

mbgrid -I datalistp.mb-1 -A2 -F5 -N -C100/3 -O ZTopoInt -V mbgrdviz -I ZTopoInt.grd &

process amplitude and sidescan

mbbackangle -I datalist.mb-1 \ -A1 -A2 -Q -V \ -N87/86.0 -R50 -G2/85/1500.0/85/100 \ -T ZTopoInt.grd mbset -PAMPCORRFILE:datalist.mb-1_tot.aga mbset -PSSCORRFILE:datalist.mb-1_tot.sga mbset -PAMPSSCORRTOPOFILE:ZTopoInt.grd

Process the data

mbm_multiprocess -X2

Filter the sidescan

mbfilter -Idatalistp.mb-1 -S2/5/3 -V

-------------------------------------------------------------------------------

#

Generate first cut grid and plots

mbgrid -I datalistl.mb-1 -A2 -F5 -N -C2 -O ZTopoLog -V mbgrdviz -I ZTopoLog.grd &

mbgrid -I datalist.mb-1 -A2 -F5 -N -O ZTopoRaw -V mbgrdviz -I ZTopoRaw.grd &

mbgrid -I datalistp.mb-1 -A2 -F5 -N -O ZTopo -V mbgrdviz -I ZTopo.grd &

mbm_grdplot -I ZTopo.grd \ -O ZTopoSlopeNav \ -G5 -D0/1 -A1 \ -L"NTU - 31 January 2020 - EM304 Testing":"Topography (meters)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -MNIdatalistp.mb-1 \ -Pc -MIE300 -MITg -V ZTopoSlopeNav.cmd

Topo slope map

mbm_grdplot -I ZTopo.grd \ -O ZTopoSlope \ -G5 -D0/1 -A1 \ -L"NTU - 31 January 2020 - EM304 Testing":"Topography (meters)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZTopoSlope.cmd

Topo shade map

mbm_grdplot -I ZTopo.grd \ -O ZTopoShade \ -G2 -A0.2/090/05 \ -L"NTU - 31 January 2020 - EM304 Testing":"Topography (meters)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZTopoShade.cmd

mbm_grdplot -I ZTopo.grd \ -O ZTopoCont \ -G1 -C1 -A1 -MCW0p \ -L"NTU - 31 January 2020 - EM304 Testing":"Topography (meters)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZTopoCont.cmd

Topo shade map

mbm_grd3dplot -I ZTopo.grd \ -O ZTopo3DShade \ -G2 -A0.2/90/05 -E150/15 \ -L"NTU - 31 January 2020 - EM304 Testing":"Topography (meters)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZTopo3DShade.cmd

-------------------------------------------------------------------------------

#

Generate first cut amplitude mosaic and plot

mbmosaic -I datalistp.mb-1 -A3 -N -Y6 -F0.05 \ -O ZAmpC -V mbgrdviz -I ZTopo.grd -J ZAmpC.grd &

mbm_grdplot -I ZAmpC.grd \ -O ZAmpCPlot \ -G1 -W1/4 -D \ -L"NTU - 31 January 2020 - EM304 Testing":"Amplitude (dB)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZAmpCPlot.cmd #

Generate first cut sidescan mosaic and plot

mbmosaic -I datalistp.mb-1 -A4F -N -Y6 -F0.05 \ -O ZSsCF -V mbgrdviz -I ZTopo.grd -J ZSsCF.grd &

mbm_grdplot -I ZSsCF.grd \ -O ZSsCFPlot \ -G1 -W1/4 -D \ -L"NTU - 31 January 2020 - EM304 Testing":"Amplitude (dB)" \ -MGLfx4/1/-20.85/0.5+l"km" \ -Pc -MIE300 -MITg -V ZSsCFPlot.cmd

-------------------------------------------------------------------------------

anbj commented 4 years ago

This is great Dave, thank you. I'll try it out as soon as I can. I met a problem compiling this, see #1104.

anbj commented 4 years ago

I'll close this one.