search

noaa-ocs-hydrography / kluster

A distributed multibeam processing system built using the Pangeo ecosystem
Creative Commons Zero v1.0 Universal
46 stars 12 forks source link

Think about GSF as an interchange format #23

Open ericgyounkin opened 3 years ago

ericgyounkin commented 3 years ago

Might be a better option out there, but GSF would probably be better than exporting to csv

This might already support reading/writing https://github.com/schwehr/generic-sensor-format

Spec can be found here https://www.leidos.com/products/ocean-marine

giumas commented 3 years ago

SONAR-netCDF4? Then the files could be imported in QGis and browsed in Panoply.

CC: @GlenRice-NOAA, @cyrilleponcelet

ericgyounkin commented 3 years ago

I found these folks that seem to be doing roughly what I am doing but for water column, which is really exciting Echopype and they seem to be something like SONAR-netcdf. Out of university of washington. I found netCDF/xarray functionality to be unreliable, especially with distributed writes, which is why I moved to Zarr. But for an interchange format that wouldn't matter necessarily. I'll have to look into this a little more.

For basic visualization, I plan to try

cyrilleponcelet commented 3 years ago

Hi @ericgyounkin , @giumas

I'm very excited to see your project and can't wait to have a deeper look at it since we are developping too some python algorithm with the same target (Build a multibeam processing sandbox for scientists/engineers, Build a multibeam cloud processing system for field use/production, ...)

SONAR-netcdf4 is a water column dedicated format right now, but it was meant to describe all kind of sounder data. It supports Omni-Sonar, single beams, an extension is done by Simrad to support ADCP. On my side, I'm currently extending it to support bathymetric echo sounders (kmall, all, and s7k). I planned to release this by the end of the year but can send you some preliminary version.

Nevertheless I would be really interesting to see if you use this SONAR-netcdf4 format (or not but it would be interesting to know why)

ericgyounkin commented 3 years ago

Hi @cyrilleponcelet,

Thanks, I'd like to hear more about your team's approach to the multibeam processing problem. For an interchange data format, all I really care about is how widely adopted it is. I don't plan on moving away from Zarr for my internal data storage, as it is very flexible and integrated into xarray. But i'm open to anything for exporting.

cyrilleponcelet commented 3 years ago

For water column data, the format is being widely adopted. But of course bathymetry being not defined yet I do not know any software (except mine) importing this format. I hope it is a matter of time but right now GSF is probably more widely used.

Great, I'd like to talk about your approach too.

ericgyounkin commented 3 years ago

@cyrilleponcelet I've been in contact with AusSeabed as well about multibeam processing, it seems they are also developing some cloud oriented processing routines. It would be good for us all to discuss to see if there was some common ground for a little collaboration. Would you be the right contact for that? If so, could you share your email? I guess I can't get that from GitHub.

monocilindro commented 2 years ago

This project by @UKHO might be connected to the solution to this issue https://github.com/UKHO/gsfpy

giumas commented 2 years ago

@monocilindro, that's currently a Linux-only solution. See https://github.com/UKHO/gsfpy/issues/127