search

VForWaTer / metacatalog

Modular metadata management platform for environmental data.
https://vforwater.github.io/metacatalog
GNU General Public License v3.0
3 stars 1 forks source link

Array type datatable #126

Closed mmaelicke closed 3 years ago

mmaelicke commented 3 years ago

@AlexDo1

codecov[bot] commented 3 years ago

Codecov Report

Merging #126 (58962cc) into master (c0b539b) will increase coverage by 0.97%. The diff coverage is 70.62%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #126      +/-   ##
==========================================
+ Coverage   58.63%   59.61%   +0.97%     
==========================================
  Files          59       61       +2     
  Lines        2715     2808      +93     
==========================================
+ Hits         1592     1674      +82     
- Misses       1123     1134      +11
Flag Coverage Δ
e2e 59.61% <70.62%> (+0.97%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
metacatalog/db/revisions/__init__.py 100.00% <ø> (ø)
metacatalog/models/details.py 97.77% <ø> (ø)
metacatalog/models/timeseries_legacy.py 0.00% <0.00%> (ø)
metacatalog/db/revisions/rev6.py 60.00% <60.00%> (ø)
metacatalog/models/datasource.py 54.80% <60.00%> (-0.10%) :arrow_down:
metacatalog/ext/io/interface.py 80.48% <66.66%> (ø)
metacatalog/ext/io/importer.py 58.41% <83.63%> (+18.76%) :arrow_up:
metacatalog/ext/io/reader.py 74.50% <92.30%> (+11.65%) :arrow_up:
metacatalog/api/add.py 48.27% <100.00%> (+2.95%) :arrow_up:
metacatalog/api/db.py 81.73% <100.00%> (+0.73%) :arrow_up:
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c0b539b...58962cc. Read the comment docs.

mmaelicke commented 3 years ago

Hey @AlexDo1,

I just want to verify something: I pulled a copy of this branch into the export/rest stuff I am currently working on earlier last week. This branch was still pretty much work-in-progress. At the time, the get_data method worked for the new ARRAY datatypes, but yielded Dataframes like this:

image

Can you verify, that in the final version of this PR:

  1. the entry_id column will be dropped (because it is redundant and not needed)
  2. the data and precision columns are un-stacked into flat-array columns
  3. the columns originating from data are renamed floowing the Entry.datasource.data_names content
  4. the columns originating from precision get some kind of naming too. I don't really care: precision1, precision2 etc.

Is that all implemented in the final PR, or do you need some help cause some stuff is missing? Sorry for pushing like this, but I would like to merge the PR soon...

AlexDo1 commented 3 years ago

I´m not sure if all these problems are solved with my latest commit, but this is a Dataframe I got from get_data:

  1. Input for entry.import_data(data=data, precision=precision, force_data_names=False) data image precision image

(note that data and precission column names contain underscores)

  1. View of the imported data in pgAdmin image

  2. Output from entry.get_data image

Maybe you could check again with the latest commit?

mmaelicke commented 3 years ago

I merged the latest commit into the export-functions branch and it works like a charm now! Really nice work! The export functions, which will be added by #132 already support JSON, XML, Python dict, netCDF and xarray export....