search

GeoscienceAustralia / agdc

Repository for Australian Geoscience Data Cube (AGDC) code
BSD 3-Clause "New" or "Revised" License
29 stars 24 forks source link

missing files? #94

Open juan-guerschman opened 8 years ago

juan-guerschman commented 8 years ago

Hi DataCubers, A quick question about discovering what data are available in the datacube (Landsat). First let me clarify that I am not using the Python API at the moment.

What I do is search by filename. For example if I want L8 for a given tile and a given year and month, I go to the corresponding folder and look all the filenames that match my requirements. In most cases I get a right answer but in some cases there are files missing

Example: for L8 in the tile 145 -35 and January 2015, I go to /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLITIRS/145-035/2015/ And then search using the string _2015-01-

In that particular case I find 3 NBAR : /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLITIRS/145-035/2015/LS8_OLI_TIRS_NBAR145-035_2015-01-08T00-08-18.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLITIRS/145-035/2015/LS8_OLI_TIRS_NBAR145-035_2015-01-17T00-02-28.tif

Another example, LS7, 2015, December, tile 145, -36 I find only two NBAR /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_NBAR145-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_NBAR145-036_2015-12-02T00-09-58.tif

But 5 FC (which need NBAR to be done first) /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_FC145-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_FC145-036_2015-12-02T00-09-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_FC145-036_2015-12-11T00-03-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_FC145-036_2015-12-11T00-03-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_FC145-036_2015-12-27T00-03-47.tif

And 8 PQA!! /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-02T00-09-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-02T00-09-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-11T00-03-34.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-11T00-03-58.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-18T00-09-51.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-18T00-10-15.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-27T00-03-47.tif /g/data2/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7ETM/145-036/2015/LS7_ETM_PQA145-036_2015-12-27T00-04-11.tif

Any idea why and if I am doing anything wrong? Is there a python function that does this? Or is there any other easier way to find what’s available?

Cheers Juan

juan-guerschman commented 8 years ago

in addition to my previous post. I'm running a test and it looks like the issue I reported applies only to LS7 and LS8 and from 2013 to current (i.e. no problem with 2012 or earlier and/or LS5).

sixy6e commented 8 years ago

Take a look at the examples: https://github.com/GeoscienceAustralia/agdc/tree/develop/api-examples/notebooks

The docs page has a lot more information on data discovery and command line tools: http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/index.html http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/api_overview.html http://geoscienceaustralia.github.io/agdc/api/0.1.0/html/api_tools.html

Also you're better off using the api, as in your example you're finding files that need to be mosaiced, eg:

from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list

satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]

tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)

for tile in tiles:
    print "Acquisition time: {}".format(tile.start_datetime)
    nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
    print "NBAR path: {}".format(nbar.path)
    print "PQ path: {}".format(pq.path)
    print "FC path: {}\n".format(fc.path)
Acquisition time: 2015-12-02 00:09:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-02T00-09-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-02T00-09-34.vrt

Acquisition time: 2015-12-11 00:03:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-11T00-03-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-11T00-03-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-11T00-03-34.vrt

Acquisition time: 2015-12-18 00:09:51
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-18T00-09-51.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-18T00-09-51.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-18T00-09-51.vrt

Acquisition time: 2015-12-27 00:03:47
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-27T00-03-47.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-27T00-03-47.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-27T00-03-47.vrt```

PQ data may not exist for some equivalent NBAR datasets, as the ingest requirements were relaxed so that NBAR could be ingested without an accompanying PQ or FC dataset.

Hope that helps
juan-guerschman commented 8 years ago

Thanks Josh, I didn't know about all these documentation and tools.

Juan

juan-guerschman commented 8 years ago

Josh, your example retrieves data ok. However, when I try to retrieve the same info using the API command Line Tools (which I found very useful) I don't get anything from 2015 or 2016 back.

example: 

$ retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2015-01 --acq-max 2016-12 --dataset-type ARG25  --list-only --output-directory /tmp
2016-01-28 15:27:01,442 INFO
        acq = 2015-01-01 to 2016-12-31
        satellites = LS5 LS7 LS8
        PQA mask =
        WOFS mask =

2016-01-28 15:27:01,442 INFO
        x = 145
        y = -036
        VECTOR mask =

2016-01-28 15:27:01,442 INFO
        datasets to retrieve = ARG25
        output directory = /tmp
        over write existing = False
        list only = True
        output format = GEOTIFF
$

same example but starting in 2014 returns existing data for 2014 but nothing for 2015 onwards.

$ retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2014-01 --acq-max 2016-12 --dataset-type ARG25  --list-only --output-directory /tmp
2016-01-28 16:11:54,611 INFO
        acq = 2014-01-01 to 2016-12-31
        satellites = LS5 LS7 LS8
        PQA mask =
        WOFS mask =

2016-01-28 16:11:54,612 INFO
        x = 145
        y = -036
        VECTOR mask =

2016-01-28 16:11:54,612 INFO
        datasets to retrieve = ARG25
        output directory = /tmp
        over write existing = False
        list only = True
        output format = GEOTIFF

2016-01-28 16:11:55,215 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-05T00-09-59.vrt]
2016-01-28 16:11:55,215 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-01-05T23-58-54.048441.tif]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-01-13T00-05-09.401713.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-14T00-03-38.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-01-29T00-05-28.166291.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-01-30T00-03-33.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-02-06T00-09-38.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-02-06T23-59-20.672816.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-02-22T00-09-25.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-02-22T23-59-10.182069.vrt]
2016-01-28 16:11:55,216 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-03-02T00-05-23.197369.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-03T00-03-02.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-10T00-09-12.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-03-10T23-59-16.320070.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-03-18T00-05-31.689687.tif]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-19T00-02-55.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-03-26T00-08-56.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-04-11T00-09-07.tif]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-11T23-59-29.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-19T00-05-46.vrt]
2016-01-28 16:11:55,217 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-04-20T00-02-22.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-04-27T00-08-27.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-04-27T23-59-40.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-05-06T00-02-29.tif]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-05-13T00-08-13.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-05-13T23-59-46.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-05-29T23-59-50.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-06-06T00-06-03.vrt]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-06-15T00-00-16.tif]
2016-01-28 16:11:55,218 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-06-22T00-06-04.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-09T00-02-18.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-16T00-08-31.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-07-16T23-59-59.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-07-25T00-02-20.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-08-01T00-08-36.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-08-02T00-00-06.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-08-09T00-06-18.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-08-10T00-02-30.vrt]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-08-17T00-09-06.tif]
2016-01-28 16:11:55,219 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-08-18T00-00-06.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-08-25T00-06-15.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-08-26T00-02-32.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-02T00-08-45.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-09-03T00-00-06.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-11T00-02-36.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-18T00-08-46.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-09-19T00-00-10.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-09-26T00-06-24.tif]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-09-27T00-02-34.vrt]
2016-01-28 16:11:55,220 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-04T00-08-49.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-05T00-00-22.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-12T00-06-36.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-13T00-02-41.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-20T00-08-53.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-10-21T00-00-31.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-10-28T00-06-45.tif]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-10-29T00-02-38.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-05T00-08-51.vrt]
2016-01-28 16:11:55,221 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-06T00-00-41.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-11-13T00-06-54.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-14T00-02-41.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/mosaic_cache/LS8_OLI_TIRS_NBAR_145_-036_2014-11-21T00-08-47.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-22T00-00-44.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-11-29T00-06-59.vrt]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-07T00-08-47.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-15T00-07-07.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-16T00-02-58.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2014/LS8_OLI_TIRS_NBAR_145_-036_2014-12-23T00-08-45.tif]
2016-01-28 16:11:55,222 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-24T00-01-00.tif]
2016-01-28 16:11:55,223 INFO Would retrieve datasets [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-12-31T00-07-08.vrt]
$

Any ideas?

Thanks

Juan

juan-guerschman commented 8 years ago

by the way, none of the command line API tools gets anything after 2014...

juan-guerschman commented 8 years ago

last comment:

I tried to replicate what you suggested and couldn't get a proper answer either (for data in 2015). Example: 

from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list

satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]

tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)

for tile in tiles:
    print "Acquisition time: {}".format(tile.start_datetime)
    nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
    print "NBAR path: {}".format(nbar.path)
    print "PQ path: {}".format(pq.path)
    print "FC path: {}\n".format(fc.path)

result is empty

try this now: 

from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list

satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2014, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]

tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)

for tile in tiles:
    print "Acquisition time: {}".format(tile.start_datetime)
    nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
    print "NBAR path: {}".format(nbar.path)
    print "PQ path: {}".format(pq.path)
    print "FC path: {}\n".format(fc.path)

and get 

Acquisition time: 2014-12-15 00:07:07
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_NBAR_145_-036_2014-12-15T00-07-07.tif
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_PQA_145_-036_2014-12-15T00-07-07.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/LS7_ETM_FC_145_-036_2014-12-15T00-07-07.tif

Acquisition time: 2014-12-31 00:07:08
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_NBAR_145_-036_2014-12-31T00-07-08.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_PQA_145_-036_2014-12-31T00-07-08.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2014/mosaic_cache/LS7_ETM_FC_145_-036_2014-12-31T00-07-08.vrt

looks like some file needs to be utdated?

Juan

sixy6e commented 8 years ago

Your first example works for me:

retrieve_dataset.py --x 145 --y -36 --satellite LS5 LS7 LS8 --acq-min 2015-01 --acq-max 2016-12 --dataset-type ARG25  --list-onl
y --output-directory $PWD
2016-01-29 08:54:44,470 INFO
        acq = 2015-01-01 to 2016-12-31
        satellites = LS5 LS7 LS8
        LS7 SLC OFF = INCLUDED
        LS8 PRE WRS2 = INCLUDED
        PQA mask =
        WOFS mask =
        season =

2016-01-29 08:54:44,470 INFO
        x = 145
        y = -036
        VECTOR mask =

2016-01-29 08:54:44,470 INFO
        datasets to retrieve = ARG25
        bands to retrieve = ['BLUE', 'GREEN', 'RED', 'NEAR_INFRARED', 'SHORT_WAVE_INFRARED_1', 'SHORT_WAVE_INFRARED_2', 'COASTAL_AEROSOL']
        output directory = /short/v10/jps547/tmp/dcube-test
        over write existing = False
        list only = True
        output format = GEOTIFF

2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/LS8_OLI_TIRS_NBAR_145_-036_2015-
01-01T00-02-53.tif]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-01-16T00-07-12.vrt]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/LS8_OLI_TIRS_NBAR_145_-036_2015-
01-17T00-02-28.tif]
2016-01-29 08:54:50,975 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-01-25T00-01-04.vrt]
2016-01-29 08:54:50,976 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/LS7_ETM_NBAR_145_-036_2015-02-01T00-0
7-19.tif]
.
.
.
.
2016-01-29 08:54:50,981 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-11T00-03-34.vrt]
2016-01-29 08:54:50,981 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-18T00-09-51.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2015/mosaic_cache/LS8_OLI_TIRS_NBAR_1
45_-036_2015-12-19T00-02-36.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_20
15-12-27T00-03-47.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS8_OLI_TIRS/145_-036/2016/mosaic_cache/LS8_OLI_TIRS_NBAR_1
45_-036_2016-01-11T00-08-43.vrt]
2016-01-29 08:54:50,982 INFO Would retrieve dataset [/g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2016/mosaic_cache/LS7_ETM_NBAR_145_-036_20
16-01-12T00-04-03.vrt]

Maybe you're using a different version???

From the u46 project modules I used the following module to produce the above output: agdc-api/0.1.0-b20160111

This might resolve your other questions regarding data not being found post 2014.

sixy6e commented 8 years ago

Your Python example also works for me.

from datetime import date
from datacube.api.model import DatasetType, Satellite
from datacube.api.query import list_tiles_as_list

satellites = [Satellite(i) for i in ['LS7']]
dataset_types = [DatasetType.ARG25, DatasetType.PQ25, DatasetType.FC25]
min_date = date(2015, 12, 01)
max_date = date(2015, 12, 31)
x_cell = [145]
y_cell = [-36]

tiles = list_tiles_as_list(x=x_cell, y=y_cell, acq_min=min_date, acq_max=max_date, satellites=satellites, dataset_types=dataset_types)

for tile in tiles:
    print "Acquisition time: {}".format(tile.start_datetime)
    nbar, pq, fc = [tile.datasets[ds] for ds in dataset_types]
    print "NBAR path: {}".format(nbar.path)
    print "PQ path: {}".format(pq.path)
    print "FC path: {}\n".format(fc.path)

Acquisition time: 2015-12-02 00:09:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-02T00-09-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-02T00-09-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-02T00-09-34.vrt

Acquisition time: 2015-12-11 00:03:34
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-11T00-03-34.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-11T00-03-34.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-11T00-03-34.vrt

Acquisition time: 2015-12-18 00:09:51
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-18T00-09-51.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-18T00-09-51.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-18T00-09-51.vrt

Acquisition time: 2015-12-27 00:03:47
NBAR path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_NBAR_145_-036_2015-12-27T00-03-47.vrt
PQ path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_PQA_145_-036_2015-12-27T00-03-47.tif
FC path: /g/data/rs0/tiles/EPSG4326_1deg_0.00025pixel/LS7_ETM/145_-036/2015/mosaic_cache/LS7_ETM_FC_145_-036_2015-12-27T00-03-47.vrt

So could be the version that you're using is pointing to a different database.

juan-guerschman commented 8 years ago

I actually tried loading a newer module yesterday, but I get the following error message: 

[jpg599@raijin2 ~]$ module use /projects/u46/opt/modules/modulefiles
[jpg599@raijin2 ~]$ module load agdc-api/0.1.0-b20160111
utility.c(2165):ERROR:50: Cannot open file '/projects/u46/opt/modules/modulefiles/eo-tools/dev' for 'reading'
[jpg599@raijin2 ~]$