search

aodn / python-aodntools

Repository for templates and code relating to generating standard NetCDF files for the Australia Ocean Data Network
GNU Lesser General Public License v3.0
10 stars 3 forks source link

Aggregated timeseries nc4 #97

Closed diodon closed 4 years ago

diodon commented 4 years ago

This is a complete new version of the code, using netCDF4 standard library for streaming aggregated data into the output file, with very low memory demand. As an example, aggregating TEMP from 650 WQM files will use less than 400MB of RAM. Still uses xarray to open and manipulate the input files, but pandas is no longer required.

diodon commented 4 years ago

in the last version the dependency on the package was removed (redundancy). Also added the timeseries_tools.py that contains a set of general functions used by the timeseries code generating scripts.

codecov[bot] commented 4 years ago

Codecov Report

Merging #97 into master will increase coverage by 0.12%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #97      +/-   ##
==========================================
+ Coverage   80.38%   80.51%   +0.12%     
==========================================
  Files           6        6              
  Lines         775      780       +5     
  Branches      124      125       +1     
==========================================
+ Hits          623      628       +5     
  Misses        125      125              
  Partials       27       27
Impacted Files Coverage Δ
...tools/timeseries_products/aggregated_timeseries.py 87.17% <100%> (ø) :arrow_up:
aodntools/timeseries_products/hourly_timeseries.py 89.05% <100%> (+0.21%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 22e8fa5...be63ef0. Read the comment docs.