search

GeoscienceAustralia / agdc

Repository for Australian Geoscience Data Cube (AGDC) code
BSD 3-Clause "New" or "Revised" License
29 stars 24 forks source link

Added class to test Modis database metadata, database data & tiles #63

Closed petewa closed 9 years ago

petewa commented 9 years ago

This is an updated version of the code I used to test jeremy’s water branch from April.

It performs 3 checks:

Checks Database Metadata against Alex/Jeremy Database creation scripts for MODIS.
Compares ingested data between a test database and the reference base database
Compares tiles and directory structure between the test tile directory and the reference base tile directory.

It may be reusable to test other ingesters, especially test 3 which compares tiles and directory structure between the test tile directory and the reference base tile directory. Test 1 and 2 are hard-coded for MODIS.

There are 3 modes of operation:

Mode 0 – Run all tests (1,2,3) Mode 1 – Checks Database Metadata against Alex/Jeremy Database creation scripts for MODIS. 1.1. Check tile_type metadata 1.2. Check satellite metadata 1.3. Check sensor metadata 1.4. Check processing_level metadata 1.5. Check band metadata 1.6. Check band_source metadata 1.7. Check band_lookup_scheme metadata 1.8. Check band_equivalent metadata Mode 2 – Compare ingested data between a test database and the reference base database 2.1. Test dataset table. 2.2. Test acquisition table. 2.3. Test tile table. 2.4. Test tile_footprint table. 2.5. Test flattened table. Mode 3 – Compare tiles and directory structure between the test tile directory and the reference base tile directory. 3.1. compares directory structure 3.1.1. checks files names are identical 3.1.2. checks directory structure are identical 3.2. compare tiles files 3.2.1. Compare file format 3.2.2. Compare Geo Transform 3.2.3. Compare Projection 3.2.4. Compare Number of bands 3.2.5. Compare number of pixels in the Height dimension 3.2.6. Compare number of pixels in the Width dimension 3.2.7. Compare metadata 3.2.8. Compare bands 3.2.8.1. Compare Data Type 3.2.8.2. Compare Block Size 3.2.8.3. Compare Offset 3.2.8.4. Compare Scale 3.2.8.5. Compare Mask Flags 3.2.8.6. Compare Mask band 3.2.8.6.1. Compare Data 3.2.8.7. Compare Data.

Example execution:

module load python module load psycopg2 module load gdal/1.11.1-python

cd tests/modis

Mode 0: python test_modis_ingester.py --test 0 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test"

Mode 1: python test_modis_ingester.py --test 1 --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" = 1. Testing Database Metadata =

1.1 Testing type_type metadata .............. Pass 1.2 Testing satellite metadata .............. Pass 1.3 Testing sensor metadata ................. Pass 1.4 Testing band_type metadata .............. Pass 1.5 Testing processing_level metadata ....... Pass 1.6 Testing band metadata ................... Pass 1.7 Testing band_source metadata ............ Pass 1.8 Testing band_lookup_scheme metadata ..... Pass 1.9 Testing band_equivalent metadata ........ Pass

Mode 2: python test_modis_ingester.py --test 2 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" = 2. Testing Database =

2.1 Testing dataset table ................... Pass 2.2 Testing acquisition table ............... Pass 2.3 Testing tile table ...................... Pass 2.4 Testing tile_footprint table ............ Pass 2.5 Testing flattened tables ................ Pass

Mode 3: python test_modis_ingester.py --test 3 --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test" = 3. Testing Tile Files =

3.1 Testing directory structure ............. Pass 3.2 Testing tiles ...........................[ ] 6/160 3%

wenjunwu commented 9 years ago

Great! Thanks Peter.

From: Peter Wang [mailto:notifications@github.com] Sent: Friday, 5 June 2015 10:41 AM To: GeoscienceAustralia/agdc Subject: [agdc] Added class to test Modis database metadata, database data & tiles (#63)

This is an updated version of the code I used to test jeremy’s water branch from April.

It performs 3 checks:

Checks Database Metadata against Alex/Jeremy Database creation scripts for MODIS.

Compares ingested data between a test database and the reference base database

Compares tiles and directory structure between the test tile directory and the reference base tile directory.

It may be reusable to test other ingesters, especially test 3 which compares tiles and directory structure between the test tile directory and the reference base tile directory. Test 1 and 2 are hard-coded for MODIS.

There are 3 modes of operation:

Mode 0 – Run all tests (1,2,3) Mode 1 – Checks Database Metadata against Alex/Jeremy Database creation scripts for MODIS. 1.1. Check tile_type metadata 1.2. Check satellite metadata 1.3. Check sensor metadata 1.4. Check processing_level metadata 1.5. Check band metadata 1.6. Check band_source metadata 1.7. Check band_lookup_scheme metadata 1.8. Check band_equivalent metadata Mode 2 – Compare ingested data between a test database and the reference base database 2.1. Test dataset table. 2.2. Test acquisition table. 2.3. Test tile table. 2.4. Test tile_footprint table. 2.5. Test flattened table. Mode 3 – Compare tiles and directory structure between the test tile directory and the reference base tile directory. 3.1. compares directory structure 3.1.1. checks files names are identical 3.1.2. checks directory structure are identical 3.2. compare tiles files 3.2.1. Compare file format 3.2.2. Compare Geo Transform 3.2.3. Compare Projection 3.2.4. Compare Number of bands 3.2.5. Compare number of pixels in the Height dimension 3.2.6. Compare number of pixels in the Width dimension 3.2.7. Compare metadata 3.2.8. Compare bands 3.2.8.1. Compare Data Type 3.2.8.2. Compare Block Size 3.2.8.3. Compare Offset 3.2.8.4. Compare Scale 3.2.8.5. Compare Mask Flags 3.2.8.6. Compare Mask band 3.2.8.6.1. Compare Data 3.2.8.7. Compare Data.

Example execution:

module load python module load psycopg2 module load gdal/1.11.1-python

cd tests/modis

Mode 0: python test_modis_ingester.py --test 0 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test"

Mode 1: python test_modis_ingester.py --test 1 --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" = 1. Testing Database Metadata =

1.1 Testing type_type metadata .............. Pass 1.2 Testing satellite metadata .............. Pass 1.3 Testing sensor metadata ................. Pass 1.4 Testing band_type metadata .............. Pass 1.5 Testing processing_level metadata ....... Pass 1.6 Testing band metadata ................... Pass 1.7 Testing band_source metadata ............ Pass 1.8 Testing band_lookup_scheme metadata ..... Pass 1.9 Testing band_equivalent metadata ........ Pass

Mode 2: python test_modis_ingester.py --test 2 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" = 2. Testing Database =

2.1 Testing dataset table ................... Pass 2.2 Testing acquisition table ............... Pass 2.3 Testing tile table ...................... Pass 2.4 Testing tile_footprint table ............ Pass 2.5 Testing flattened tables ................ Pass

Mode 3: python test_modis_ingester.py --test 3 --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test" = 3. Testing Tile Files =

3.1 Testing directory structure ............. Pass 3.2 Testing tiles ...........................[ ] 6/160 3%

You can view, comment on, or merge this pull request online at:

https://github.com/GeoscienceAustralia/agdc/pull/63

Commit Summary

File Changes

Patch Links:

— Reply to this email directly or view it on GitHubhttps://github.com/GeoscienceAustralia/agdc/pull/63.

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.

landscape-bot commented 9 years ago

Code Health Code quality remained the same when pulling 05e6571 on petewa:peter/test_modis into 787a7a5 on GeoscienceAustralia:develop.

jeremyh commented 9 years ago

Thanks Peter.