keelymc commented 3 months ago

General description of problem

I was attempting to convert AZFP echosounder data from engineering format to netCDF. The frequency values for the echosounder are 130kHz, 200kHz, 455kHz, and 769kHz. Using the code, ed = open_raw('FILENAME.01A', sonar_model='AZFP', xml_path='XMLFILENAME.xml') ed.to_netcdf(save_path='./unpacked_files') I received an error which is attached to this inquiry. The error suggests that there is something wrong with the use of the 130kHz frequency in echopype. I was wondering if anyone knew why this was occurring and if there is something that can be done to fix the error? Thank you in advanced.

Echopype version

Echopype v0.9.0

How did you install Echopype (e.g., conda or pip)

I installed Echopype by pip

What is your operating system

Windows

Minimal code example

I had to use this code so I could choose the files within my directory because when I attempted to manually put in the pathways to my files, the XML file was not working so the code is quite long.

import tkinter as tk from tkinter import filedialog import echopype as ep from echopype import open_raw

Function to select the XML file

def select_xml_file(): root = tk.Tk() root.withdraw() # Hide the main window xml_path = filedialog.askopenfilename( title="Select XML File", filetypes=[("XML files", "*.xml")] ) return xml_path

Function to select the raw data file

def select_raw_data_file(): root = tk.Tk() root.withdraw() # Hide the main window raw_data_path = filedialog.askopenfilename( title="Select Raw Data File", filetypes=[("01A files", "*.01A")] ) return raw_data_path

Function to select the output directory

def select_output_directory(): root = tk.Tk() root.withdraw() # Hide the main window save_path = filedialog.asksaveasfilename( title="Save NetCDF File As", defaultextension=".nc", filetypes=[("NetCDF files", "*.nc")] ) return save_path

Select files and directory

xml_path = select_xml_file() raw_data_path = select_raw_data_file() save_path = select_output_directory()

Check if paths are selected

if not xml_path or not raw_data_path or not save_path: print("File selection was cancelled.") else:

Open the raw data file and convert it to NetCDF format

ed = open_raw(raw_data_path, sonar_model='AZFP', xml_path=xml_path)
ed.to_netcdf(save_path=save_path)
print(f"NetCDF file saved to {save_path}")

Error message printouts

Traceback (most recent call last):
  File "C:/Users/Admin/OneDrive/Documents/1.Honours/Python Practice/Converting echosounder data from 01A to netCDF_23070918.py", line 47, in <module>
    ed = open_raw(raw_data_path, sonar_model='AZFP', xml_path=xml_path)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python312\Lib\site-packages\echopype\utils\prov.py", line 237, in inner
    dataobj = func(*args, **kwargs)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python312\Lib\site-packages\echopype\convert\api.py", line 472, in open_raw
    parser.parse_raw()
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python312\Lib\site-packages\echopype\convert\parse_azfp.py", line 371, in parse_raw
    self.Sv_offset[ind] = self._calc_Sv_offset(
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python312\Lib\site-packages\echopype\convert\parse_azfp.py", line 557, in _calc_Sv_offset
    raise ValueError(
ValueError: Frequency 130000.0 Hz is not in the Sv offset dictionary! Please contact AZFP Environmental Sciences and raise an issue in the echopype repository.

Example data

Related existing issues or PRs

I have not seen a related issue.

Troubleshooting steps

No response

leewujung commented 3 months ago

Hey @keelymc - thanks for reporting this! This is because in Echopype the function to retrieve Sv offset look for matching frequency values only. A few months ago from talking with the AZFP sales person they recommended using the 125 kHz Sv offset value for 130 kHz transducers. It is a matter of adding this small expansion in to the code, but we haven't gotten around to do it.

The relevant section is here: https://github.com/OSOceanAcoustics/echopype/blob/be2b86cc668b59b0b856789de29614bcc25000bd/echopype/convert/parse_azfp.py#L28-L52

Would you be interested in putting 130 kHz into this dictionary and throw in a test for this?

keelymc commented 3 months ago

Yes, I can certainly add the 130kHz to the dictionary. I have created a separate fork to do so and can send it once I work out and test the code.

leewujung commented 2 months ago

Sounds good, thanks @keelymc ! Here are some tips from our contributing page.

keelymc commented 2 months ago

Hello, my apologies, I am having another technical issue. There seems to be a conflict between two packages required for the conversion of the 01A file to netCDF. Specifically, the packages pathlib and datatree seem to be causing problems. It seems that the datatree package is not available for later versions of python and that the new package is referred to as xarray_datatree. I have installed xarray_datatree. I have also attempted to fix the error by installing xarray_datatree and removing the package 'datatree', however the issue seemed to persist. I have attached the error code I have been receiving along with the code I am using, and the edits I have made in the dictionary. To the dictionary, I have added the 130kHz as well as the addition of 320us pulse length for the 200kHz frequency which was used for my AZFP glider (I have circled in red the parts which I have added to the dictionary).