Extract 3d data from Lowrance StructureScan3d

[Whateverman2012]

Hi,

First of all, thanks for your effort to make sl file useful outside major brands ecosystem. We try to use those files to create a visual art project.

I'm trying to Export3D from an Sl3 file produced by a Lowrance FS7 connected to a StructureScan3D transducer. The console output tells me that I have more than 2000 3d frames but unfortunately with 3dm or 3dg option the csv output is empty. Only the headers are available. I traced it and header.NumberOfLeftBytes in Export3D seems to be 0. Any suggestion to what to look for ?

Regards.

Jean-François

[halmaia]

Hi,

It is strange, maybe a different file version. Can you send me a small, but problematic file?

Best regards, Ákos Halmai

On Wed, Aug 2, 2023, 1:08 AM Whateverman2012 @.***> wrote:

Hi,

First of all, thanks for your effort to make sl file useful outside major brands ecosystem. We try to use those files to create a visual art project.

I'm trying to Export3D from an Sl3 file produced by a Lowrance FS7 connected to a StructureScan3D transducer. The console output tells me that I have more than 2000 3d frames but unfortunately with 3dm or 3dg option the csv output is empty. Only the headers are available. I traced it and header.NumberOfLeftBytes in Export3D seems to be 0. Any suggestion to what to look for ?

Regards.

Jean-François

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4EOMRAD67Z5V3ETUIDXTGD5ZANCNFSM6AAAAAA3AN44GY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Whateverman2012]

Hi,

Thank you for the quick feedback. Digging deeper I found that all my data were marked as unreliable. It may indicate that the file is fine but the survey is not. Some sort of good news. I'll create another survey in few days to see if we get better results. Maybe we did something wrong with the wiring or the transducer settings. Sorry to have bothered you.

Hopefully I discovered the "includeUnreliable" feature in Export3D ! I added 2 more command line options : -3dm+ and -3dp+ which "includeUnreliable" data.

We have the latest firmware of Lowrance. So once I get that new survey I'll let you know if Lowrance messed with their sl3 format if you want.

Regards

Jean-François Lahos

[Whateverman2012]

Hi M. Halmai,

I just made a scan. Unfortunately it seems that all data are marked "Unreliable". I know there are a lot of vegetation in the lake but the sonar screen was able to show me interesting images during the survey. (link : https://www.dropbox.com/t/ZAPyuOS0JU81534j) May I ask you if I can get something out those data or it is unusable. As you said to me, maybe it is due to the newer firmware...

Beside, with a csv file from export3d, is there a formula to to get 3d points from X, Y, Zwgs, depth, angle and distance ?

Regards.

Jean-François Lahos

[halmaia]

Hi,

Thank you for the file, now it is easier to mitigate. I checked the file structure. I have a good and a bad news:

1. There is no mounting issue, almost all bytes are reliable.
2. The file structure is different. The values are shifted to the left with 12 bytes, and some previously empty fields are populated now.

I try to correct the shift and do a new release.

Best regards, Ákos Halmai

Whateverman2012 @.***> ezt írta (időpont: 2023. aug. 7., H, 2:19):

Hi M. Halmai,

I just made a scan. Unfortunately it seems that all data are marked "Unreliable". I know there are a lot of vegetation in the lake but the sonar screen was able to show me interesting images during the survey. (link : https://www.dropbox.com/t/GRn9dQD7vHz3Hz8N) May I ask you if I can get something out those data or it is unusable. As you said to me, maybe it is due to the newer firmware...

Beside, with a csv file from export3d, is there a formula to to get 3d points from X, Y, Zwgs, depth, angle and distance ? (Angle of 45 degres seems to be my average. Does 45 degres mean -Z axis ?)

Regards.

Jean-François Lahos

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5#issuecomment-1667017570, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4HQWRLGZSNCJ5333P3XUAYAXANCNFSM6AAAAAA3AN44GY . You are receiving this because you commented.Message ID: @.***>

[halmaia]

Got it: there is a version number at the beginning of the frame at byte 4: my files are 1.0, your file was recorded in 1.3.

Ákos Halmai @.***> ezt írta (időpont: 2023. aug. 7., H, 13:53):

Hi,

Thank you for the file, now it is easier to mitigate. I checked the file structure. I have a good and a bad news:

1. There is no mounting issue, almost all bytes are reliable.
2. The file structure is different. The values are shifted to the left with 12 bytes, and some previously empty fields are populated now.

I try to correct the shift and do a new release.

Best regards, Ákos Halmai

Whateverman2012 @.***> ezt írta (időpont: 2023. aug. 7., H, 2:19):

Hi M. Halmai,

I just made a scan. Unfortunately it seems that all data are marked "Unreliable". I know there are a lot of vegetation in the lake but the sonar screen was able to show me interesting images during the survey. (link : https://www.dropbox.com/t/GRn9dQD7vHz3Hz8N) May I ask you if I can get something out those data or it is unusable. As you said to me, maybe it is due to the newer firmware...

Beside, with a csv file from export3d, is there a formula to to get 3d points from X, Y, Zwgs, depth, angle and distance ? (Angle of 45 degres seems to be my average. Does 45 degres mean -Z axis ?)

Regards.

Jean-François Lahos

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5#issuecomment-1667017570, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4HQWRLGZSNCJ5333P3XUAYAXANCNFSM6AAAAAA3AN44GY . You are receiving this because you commented.Message ID: @.***>

[halmaia]

Hi,

I published a new release to handle the new file version. Check the resulting image:

Best regards, Ákos Halmai

[halmaia]

Hi,

Check out this video: https://youtu.be/NkwaxYtHqxw

Best regards, Ákos Halmai

[halmaia]

The SideScan is also repaired: (The SideScan corrected version will be bublished in a few hours...)

[halmaia]

I published the release for imagery too.

[halmaia]

Regarding to the question, how to convert these points into a meaningful dataset:

1. This is a CSV file with Lowrance's X & Y coordinates with an additional WGS84 Z value. All coordinates are metric. The X & Y is a Mercator projection, similar to the Web Mercator.
2. To visualize these points, you need a Geographical Information System, like the open source QGIS, or the proprietary ArcGIS Pro. These applications are capable to read these files and project them into the well-known WGS84 (GPS's Lat & Lon). It is also possible to convert these points into raster, to use the result as a Digital Bathymetry Model. I used the ArcGIS Pro's Topo To Raster function to create the DBM in the video. It is also available in the open source QGIS.
3. You also need a GIS software to correctly visualize sidescan & downscan data: these dataset are highly distorted otherwise.

[Whateverman2012]

Hi,

What an amazing work ! Thank you so much for those lightning fast updates.

I don't know if it is the right place to write all this...

I think I finally understood the exported data. I was working with -3dm option which in my case has a constant zero as heading. Thus all my frames were aiming the same direction. As soon I switched to -3dg (gnss) 3d points were aligned according to gnss heading.

Sadly, the 3d data seems to have been filtered in the sl3 file in a way trees are completely gone. I guess as soon the sonar system can't figure out a 3d shape from points they are ignored even if shown on monitor. If you know something about StructureScan3d "noise" rejection please feel free to share ;)

Thank you again for all your useful work.

Best regards.

Jean-François

[halmaia]

Hi,

The exported CSV file works like a series of cross sections: around the sonar, the ground sampling distance is about 1 foot, but after reaching a threshold it samples the "important" points only. Magnetic heading is designed to counteract drifting, but to export the dataset with that type of heading your need to have a digital compass attached to Lowrance. For example a Precision-9. Yes, the StructureScan is designed to survey the bottom. On older HDS devices I saw an option to change the noise rejection level, but I think it belongs to the standard sonar imagery. Non-ground points used to appear as white spots hanging in the water column. If you see such points on the screen, they may appear in the unreliable point set. Today, I can not check the description, nor the sonar, but if I go home, I will do it. However, the resolution might be too low to get something useful. There is a dataframe in the file which contains the raw interferometric readings, but I was unable to decode it. Maybe we can find something interesting in that dataset above the bottom. The logs are clearly visible on sidescan and downscan imagery. If you have more data to share, it is welcome, because I still have unknown fields in the file. Do not forget to adjust the Z values with the offset between your transducer and the touchscreen. You can re-adjust the SS imagery in a GIS software to get some extra brightness.

Best regards, Ákos Halmai

On Tue, Aug 8, 2023, 6:06 PM Whateverman2012 @.***> wrote:

Hi,

What an amazing work ! Thank you so much for those lightning fast updates.

I don't know if it is the right place to write all this...

I think I finally understood the exported data. I was working with -3dm option which in my case has a constant zero as heading. Thus all my frames were aiming the same direction. As soon I switched to -3dg (gnss) 3d points were aligned according to gnss heading.

Sadly, the 3d data seems to have been filtered in the sl3 file in a way trees are completely gone. I guess as soon the sonar system can't figure out a 3d shape from points they are ignored even if shown on monitor. If you know something about StructureScan3d "noise" rejection please feel free to share ;)

Thank you again for all your useful work.

Best regards.

Jean-François

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5#issuecomment-1669900895, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4GY4MJ4XM6FWONPXVTXUJPZLANCNFSM6AAAAAA3AN44GY . You are receiving this because you modified the open/close state.Message ID: @.***>

[Whateverman2012]

Hi,

Thank you very much for sharing all that information.

Here is another scan : https://www.dropbox.com/t/MYRTTBwhV1k1sktG

Do you know a commercial software using 3d frames ? It is so unclear why those data are exported but not really used by anyone...

Regards

Jean-François Lahos

[halmaia]

Yes, definitely! Chesapeake Technology's SonarWiz ( https://chesapeaketech.com/products/). However, the pricing might be disappointing :-). If you will be able to get one, I'm really interested in the output.

Whateverman2012 @.***> ezt írta (időpont: 2023. aug. 15., K, 18:03):

Hi,

Thank you very much for sharing all that information.

Here is another scan : https://www.dropbox.com/t/MYRTTBwhV1k1sktG

Do you know a commercial software using 3d frames ? It is so unclear why those data are exported but not really used by anyone...

Regards

Jean-François Lahos

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5#issuecomment-1679201882, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4FR2ZCBJHXHNHSNJ33XVOMUXANCNFSM6AAAAAA3AN44GY . You are receiving this because you modified the open/close state.Message ID: @.***>

[halmaia]

You can also try https://www.genesismaps.com/.

[halmaia]

Hi!

The files have a "World File" for position and scaling. If you put these DS and SS files into a GIS application (e.g. QGIS) you're gonna see the same result as in ReefMaster. (ReefMaster does the same). If you add multiple files to QGIS you will be able to follow the whole record without jumps. Simple image viewers can not process this auxiliary world file.

Best regards, Ákos Halmai

Whateverman2012 @.***> ezt írta (időpont: 2023. okt. 7., Szo, 0:03):

Hi !

I am actually trying to export downscan data to an image. It definitely works but I want to unify the different scale between each breakpoints. I played with water depth and max_range but nothing really convincing. Do you have an idea how to calculate the scale of each parts within a frame ?

Reefmaster seems to know what to do... Here's a snapshot of one of its windows. [image: image] https://user-images.githubusercontent.com/19954470/273339464-e81d9bfc-9759-4174-9aad-6835f5ca47c6.png

Regards

— Reply to this email directly, view it on GitHub https://github.com/halmaia/SL3Reader/issues/5#issuecomment-1751448780, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEBCY4BG5ZS2Q3IMOU6AIOTX6B54BAVCNFSM6AAAAAA3AN44G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRGQ2DQNZYGA . You are receiving this because you modified the open/close state.Message ID: @.***>