SMAP Downloading Issue

[akssinghe]

I am having an issue when populating the db with SMAP.

INFO: Updating database! None [92.0, 5.5, 109.75, 28.75] None WARNING: Missing options for local dataset smap. Nothing ingested!

data.conf [domain] minlat: 5.5 maxlat: 28.75 minlon: 92.0 maxlon: 109.75

[smap] startdate: 2015-1-1 enddate: 2016-12-31

[kandread]

Ok, there's a couple of things going on. 1) h5py needs to be installed. I will add it as a requirement in the install docs. 2) Unfortunately the download will still not work because NASA decided to switch from FTP to HTPPS. The SMAP code needs to be completely rewritten, since there is no easy way to automatically figure the full path to the the filename so a web scraper needs to be written that can then select the files to download.

[akssinghe]

Thanks Kostas for quick information.

[kandread]

Tested and fixed with 532873d1d7ae7f0961d622405fdd5156926836f9 Need to run ./bin/buildout install earthdata beforehand to install the netrc file for authentication.

[akssinghe]

It seems authentication issue still persists for SMAP, AMSRE and MOD16.

SMAP: INFO: Starting new HTTP connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org INFO: Resetting dropped connection: n5eil01u.ecs.nsidc.org WARNING: No SMAP data available for 2016-10-02.

AMSRE: INFO: Starting new HTTP connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org INFO: Resetting dropped connection: n5eil01u.ecs.nsidc.org INFO: Imported 2009-10-03 in soilmoist.amsre WARNING: AMSR-E data not available for 20091003. Skipping download!

MOD16: INFO: Updating database! INFO: Starting new HTTP connection (1): files.ntsg.umt.edu WARNING: MOD16 data not available for 2009-10-01. Skipping download! INFO: Starting new HTTP connection (1): files.ntsg.umt.edu WARNING: MOD16 data not available for 2009-10-02. Skipping download!

data.conf: [domain] minlat: 5.5 maxlat: 28.75 minlon: 92.0 maxlon: 109.75

[smap] startdate: 2016-10-1 enddate: 2016-10-5

[kandread]

This is not an authentication issue. Regarding SMAP, the variable names have changed upstream. The warning you're getting with AMSRE is incorrect, I've removed it. MOD16 is an 8-day product hence why you're not getting data on 2009-10-1 and 2009-10-2.

[akssinghe]

Same issue for SMAP and AMSRE is still there. I feel that the issue is related to url of the datasets.

BTW, A suggestion; is it possible to add a url to SMAP to access the AP (active passive combined) dataset?

SMAP: INFO: Starting new HTTP connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org INFO: Resetting dropped connection: n5eil01u.ecs.nsidc.org WARNING: No SMAP data available for 2016-11-11.

AMSRE: INFO: Updating database! INFO: Starting new HTTP connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): urs.earthdata.nasa.gov INFO: Starting new HTTPS connection (1): n5eil01u.ecs.nsidc.org INFO: Resetting dropped connection: n5eil01u.ecs.nsidc.org WARNING: AMSR-E data not available for 20091001. Skipping download!

[kandread]

Can't reproduce it, I just tried it on those dates and it works here. Are you sure you have the latest code? By active/passive do you mean the enhanced product?

[akssinghe]

Yes. Pulled from github today. Anyway I will try again to pull. correct, enhanced product.

[kandread]

enhanced product was added, check #67. Just use smape instead of smap

[akssinghe]

Thanks... Is it necessary to move .netrc file to machine's home directory? or to keep in RHEAS directory?

[kandread]

Yes, absolutely. The buildout installation should take care of that, but if it's not there you can move it manually.