search

GeoscienceAustralia / agdc

Repository for Australian Geoscience Data Cube (AGDC) code
BSD 3-Clause "New" or "Revised" License
29 stars 24 forks source link

Peter/test modis #59

Closed petewa closed 9 years ago

petewa commented 9 years ago

Note: I forked this from agdc/master by mistake a while ago instead of agdc/develop, however I have merged changes from agdc/develop and it is up date with agdc/develop.

This is an updated version of the code I used to test jeremy’s water branch from April.

It performs 3 checks:

It may be reusable to test other ingesters, especially test 3 which compares tiles and directory structure between the test tile directory and the reference base tile directory. Test 1 and 2 are hard-coded for MODIS.

There are 3 modes of operation:

Mode 0 – Run all tests (1,2,3) Mode 1 – Checks Database Metadata against Alex/Jeremy Database creation scripts for MODIS. 1.1. Check tile_type metadata 1.2. Check satellite metadata 1.3. Check sensor metadata 1.4. Check processing_level metadata 1.5. Check band metadata 1.6. Check band_source metadata 1.7. Check band_lookup_scheme metadata 1.8. Check band_equivalent metadata Mode 2 – Compare ingested data between a test database and the reference base database 2.1. Test dataset table. 2.2. Test acquisition table. 2.3. Test tile table. 2.4. Test tile_footprint table. 2.5. Test flattened table. Mode 3 – Compare tiles and directory structure between the test tile directory and the reference base tile directory. 3.1. compares directory structure 3.1.1. checks files names are identical 3.1.2. checks directory structure are identical 3.2. compare tiles files 3.2.1. Compare file format 3.2.2. Compare Geo Transform 3.2.3. Compare Projection 3.2.4. Compare Number of bands 3.2.5. Compare number of pixels in the Height dimension 3.2.6. Compare number of pixels in the Width dimension 3.2.7. Compare metadata 3.2.8. Compare bands 3.2.8.1. Compare Data Type 3.2.8.2. Compare Block Size 3.2.8.3. Compare Offset 3.2.8.4. Compare Scale 3.2.8.5. Compare Mask Flags 3.2.8.6. Compare Mask band 3.2.8.6.1. Compare Data 3.2.8.7. Compare Data.

Example execution:

module load python module load psycopg2 module load gdal/1.11.1-python

cd tests/modis

Mode 0: python test_modis_ingester.py --test 0 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'" --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test"

Mode 1: python test_modis_ingester.py --test 1 --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'"

= 1. Testing Database Metadata =

1.1 Testing type_type metadata .............. Pass 1.2 Testing satellite metadata .............. Pass 1.3 Testing sensor metadata ................. Pass 1.4 Testing band_type metadata .............. Pass 1.5 Testing processing_level metadata ....... Pass 1.6 Testing band metadata ................... Pass 1.7 Testing band_source metadata ............ Pass 1.8 Testing band_lookup_scheme metadata ..... Pass 1.9 Testing band_equivalent metadata ........ Pass

Mode 2: python test_modis_ingester.py --test 2 --base_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test_base' user='cube_user' password='GAcube0'" --test_db_string "host='130.56.244.224' port='6432' dbname='hypercube_modis_test' user='cube_user' password='GAcube0'"

= 2. Testing Database =

2.1 Testing dataset table ................... Pass 2.2 Testing acquisition table ............... Pass 2.3 Testing tile table ...................... Pass 2.4 Testing tile_footprint table ............ Pass 2.5 Testing flattened tables ................ Pass

Mode 3: python test_modis_ingester.py --test 3 --base_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-base" --test_tile_directory "/g/data/u39/public/data/modis/datacube/mod09-tiled-test"

= 3. Testing Tile Files =

3.1 Testing directory structure ............. Pass 3.2 Testing tiles ...........................[ ] 6/160 3%

simonoldfield commented 9 years ago

A quick comment/preference - the fact that this pull request originated from a fork from master rather than develop seems to convolute the history unnecessarily. My preference would be for Peter to rebase from develop and them re-submit the pull request so that we get a clearer history of what has actually changed.

wenjunwu commented 9 years ago

Agree with Simon’s comment. It will be cleaner.

From: Simon Oldfield [mailto:notifications@github.com] Sent: Thursday, 4 June 2015 12:03 PM To: GeoscienceAustralia/agdc Subject: Re: [agdc] Peter/test modis (#59)

A quick comment/preference - the fact that this pull request originated from a fork from master rather than develop seems to convolute the history unnecessarily. My preference would be for Peter to rebase from develop and them re-submit the pull request so that we get a clearer history of what has actually changed.

— Reply to this email directly or view it on GitHubhttps://github.com/GeoscienceAustralia/agdc/pull/59#issuecomment-108680639.

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.