search

dtcenter / METdataio

https://metdataio.readthedocs.io/en/latest/index.html
Apache License 2.0
5 stars 3 forks source link

Error loading ECLV(ECON) line type from VSDB files #32

Closed TatianaBurek closed 4 years ago

TatianaBurek commented 4 years ago

mv_dev.txt

Lines with ECLV(ECON) line type from VSDB files are not in eclv tables. Count comparison: line_data_eclv 1339 83 line_data_eclv_pnt 24165 1557 File for testing: met_data/ens_vsdb/2011070100/sref/sref_20141004.vsdb line #1101

Expected Behavior

Both databases should have the same count

To Reproduce

Describe the steps to reproduce the behavior: 1. run query on both databases: `select from line_data_eclv, line_data_eclv_pnt, data_file where line_data_eclv.line_num=1101 and line_data_eclv.data_file_id=data_file.data_file_id and data_file.filename="sref_20141004.vsdb" and line_data_eclv.line_data_id=line_data_eclv_pnt.line_data_id` 2. The result should be similar to the data from the attached file*

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

Labels

Projects and Milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow for details.

venitahagerty commented 4 years ago

Hi Tatiana,

Using the test data I received in May 2020, I get line counts of 1307 for line_data_eclv and 23557 from line_data_eclv_pnt for both Java and Python. Here is the XML from the load_spec after the connection tag. Could you please send me your XML?

2011070100 2011070812 129600 yyyyMMddHH false 1 true true true true vhagerty test data true true false true true false /Users/venita.hagerty/metmay2020/met_data/{config}/{fcst_init}/{config1} mtd afwaoc noahmp ensemble_stat point_stat grid_stat meso_vsdb ens_vsdb precip_vsdb metprd sref vsdb_phy0134 vsdb_phy0135 true Database containing grid-stat and point-stat data for AFWAOC (v3.5.1) and NoahMP (v3.5.1) for the test period of 2011070100 - 2012062912. All three were run with Noahv2.7.1 LIS input data. Loaded on 10/28/13 (after re-running with ADPSFC instead of ONLYSF).

I will look at that specific VSDB file.

Thank you, Venita

On Fri, Jul 31, 2020 at 1:14 PM Tatiana Burek notifications@github.com wrote:

mv_dev.txt https://github.com/NCAR/METdb/files/5008653/mv_dev.txt

*Lines with ECLV(ECON) line type from VSDB files are not in eclv tables. Count comparison: line_data_eclv 1339 83 line_data_eclv_pnt 24165 1557 File for testing: met_data/ens_vsdb/2011070100/sref/sref_20141004.vsdb line

1101*

Expected Behavior

Both databases should have the same count To Reproduce

Describe the steps to reproduce the behavior:

1. run query on both databases: select from line_data_eclv, line_data_eclv_pnt, data_file where line_data_eclv.line_num=1101 and line_data_eclv.data_file_id=data_file.data_file_id and data_file.filename="sref_20141004.vsdb" and line_data_eclv.line_data_id=line_data_eclv_pnt.line_data_id 2. The result should be similar to the data from the attached file*

List relevant project deadlines here or state NONE. Funding Source

Define the source of funding and account keys here or state NONE. Define the Metadata Assignee

  • Select engineer(s) or no engineer required
  • Select scientist(s) or no scientist required

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Review projects and select relevant Repository and Organization ones
  • Select milestone

Define Related Issue(s)

Consider the impact to the other METplus components.

Bugfix Checklist

See the METplus Workflow https://ncar.github.io/METplus/Contributors_Guide/github_workflow.html for details.

  • Complete the issue definition above.
  • Fork this repository or create a branch of master_. Branch name: bugfix_master_
  • Fix the bug and test your changes.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into master_ and link the pull request to this issue. Pull request: bugfix master_
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Complete the steps above to fix the bug on the develop branch and link the pull request to this issue. Branch name: bugfix_develop Pull request: bugfix develop
  • Close this issue.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NCAR/METdb/issues/32, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJGI4XWCPTVFJF6L6TXBFK3R6MJYHANCNFSM4PQYCRFA .

-- Venita Hagerty, PhD Research Associate CIRA at NOAA, GSL/AVID 2B152, x6098

TatianaBurek commented 4 years ago

I use XML form METviewer-test repository: https://github.com/NCAR/METviewer-test/blob/master/test_cases/loading/load_test.xml I added 'tc_gen' a few days ago but Java ignores it.

The number of loaded files (excluding mtd) in data_file is the same in both databases. I run this query: select count() from data_file where filename not like 'mtd%'; and got +----------+ | count() | +----------+ | 1707 | +----------+ for Java and Python

venitahagerty commented 4 years ago

This error was caused by having NA values in fcst_lead and obs_lead, which had not been seen before. Added code to convert these NA values to 0.

This revealed the issue that when there is an error in a batch of files, none of the files in that batch are loaded into the database. This is a more complicated issue to solve. Will add another issue for that.