DFO-Ocean-Navigator / Ocean-Data-Map-Project

The Ocean Navigator is an online tool that is used to help visualise scientific research data. a users guide is available at https://dfo-ocean-navigator.github.io/Ocean-Navigator-Manual/ and the tool is live at
http://navigator.oceansdata.ca
GNU General Public License v3.0
49 stars 20 forks source link

Cmems data_importers scripts #1128

Closed Neha-Thakare1995 closed 6 months ago

Neha-Thakare1995 commented 6 months ago

Background

This pull request addresses the need to import data from NetCDF files from cmems into a database using Python. The code leverages the xarray library to handle NetCDF files and SQLAlchemy for database operations. The script aims to provide a flexible and efficient solution for importing observational data into a database.

Why did you take this approach?

  1. Database operations are optimized by using SQLAlchemy's session management. Sessions are committed after each step to minimize the risk of data loss in case of failures.
  2. To import data from NetCDF files from cmems into a database using Python.

Anything in particular that should be highlighted?

  1. The script iterates over each NetCDF file, extracting relevant data variables defined in the VARIABLE_MAPPING. It then iterates over the rows of the DataFrame to create corresponding database records for stations, samples, data types, and platforms.
  2. Users can specify the database URI as an input parameter, allowing them to connect to different databases without modifying the code.
  3. The defopt library is utilized to automatically generate a CLI for the script, simplifying its invocation and usage. Users can run the script from the command line with minimal effort.

Screenshot(s)

Checks

Hint To run all python unit tests run the following command from the root repository directory:

bash run_python_tests.sh
Neha-Thakare1995 commented 6 months ago

@JustinElms The script has been as per the suggestion

JustinElms commented 6 months ago

Great work Neha!