This repository is no longer updated, any development is now happening in the new [[https://github.com/GEUS-SICE/SICE][SICE repository]] of the [[https://github.com/GEUS-SICE][GEUS-SICE organization]].
Table of Contents :toc_2:noexport:
[[#sentinel-3-s3-albedo-processing-pipeline][Sentinel-3 (S3) Albedo Processing Pipeline]]
[[#debugging--testing][Debugging & Testing]]
[[#development-environment][Development Environment]]
[[#misc-notes--code-snippets][Misc Notes & Code Snippets]]
Sentinel-3 (S3) Albedo Processing Pipeline
The steps above are encapsulated in [[./S3_wrapper.sh]].
In more detail:
** Fetch S3 OLCI & SLSTR products
** Process with SNAP
** pySICE
** Mosaic
** Outputs | File Name | Description | Units | |--------------------------------------+------------------------------------------------------------------------+-------------| | grain_diameter.tif | Snow grain diameter | mm | | snow_specific_surface_area.tif | Snow specific surface area | m^{2}kg^{-1}| | NDSI.tif | Normalized Difference Snow Index | | | NDBI.tif | Normalized Difference Blue ice Index | | | SCDA_final.tif | Simple Cloud Detection Algorithm | | | SZA.tif | Solar Zenith Angle | | | sza_lut.tif | Solar Zenith Angle LookUp Table | | | SAA.tif | Solar Azimuth Angle | | | OZA.tif | Observation Zenith Angle | | | OAA.tif | Observation Azimtuh Angle | | | albedo_bb_planar_nir.tif | near infrared broadband planar albedo | 700-2400 nm | | albedo_bb_planar_sw.tif | shortwave broadband planar albedo | 300-2400 nm | | albedo_bb_planar_vis.tif | visible broadband planar albedo | 300-700 nm | | albedo_bb_spherical_nir.tif | near infrared broadband spherical albedo | 700-2400 nm | | albedo_bb_spherical_sw.tif | shortwave broadband spherical albedo | 300-2400 nm | | albedo_bb_spherical_vis.tif | visible broadband spherical albedo | 300-700 nm | | r_TOA_01..21.tif | Top Of the Atmosphere reflectance (OLCI bands 1 through 21) | | | rBRR_01..21.tif | bottom of the atmosphere surface reflectance (OLCI bands 1 through 21) | | | BT_S7..9.tif | Brightness Temperatures (SLSTR bands 7 to 9) | | | r_TOA_S1..S5.tif | Top of the Atmosphere reflectance (SLSTR bands 1 to 5) | | | albedo_spectral_planar_01..21.tif | spectral planar albedo (OLCI bands 1 to 11 and 16 to 21) | | | albedo_spectral_spherical_01..21.tif | spectral spherical albedo (OLCI bands 1 to 11 and 16 to 21) | | | diagnostic_retrieval.tif | See next table | | | height.tif | Altitude from the SRAL instrument (Ku/C Radar Altimeter) | | | conc.tif | pollutant concentration (volumetric concentration) | | | al.tif | effective absorption length(mm) | | | r0.tif | reflectance of a semi-infinite non-absorbing snow layer | | | 03.tif | total ozone product (OLCI) | | | 03_SICE.tif | total ozone product (OLCI) corrected for ozone scattering | | | WV.tif | Total column Water Vapor (OLCI) | | | num_scenes.tif | number of scenes in daily mosaic | | | nume_scenes_cloudfree.tif | number of cloud free scenes in daily mosaic | |
| Diagnostic Code | Description | |-----------------+-----------------------------------------------------------------------------------------------| | 0 | clean snow | | 1 | polluted snow | | 6 | polluted snow for which r0 was calculated and not derived from observations | | 7 | polluted snow of calculated spherical albedo in bands 1 and 2 >0.98 reprocessed as clean snow | | 100 | sza<75, no retrival | | 102 | TOA reflectance at band 21 < 0.1, no retrieval | | 104 | grain_diameter < 0.1, no retrieval, potential cloud flag | | -n | impossible to solve polluted snow albedo equation at band n |
The Slopey option computes effective solar/viewing angles based on ArcticDEM to obtain reflectance products accounting for slopes. This terrain correction is an option of the SICE toolchain that can be enabled in [[./S3_wrapper.sh]]. The associated outputs have the same file names as in the table presented above.
Change the year and day for loops in [[./S3_wrapper.sh]] to one year and one day. E.g.
for year in 2017; do # one year for doy in 227 180; do # two example days
done done
This work is developed with the following software versions
** Bash
bash --version
: GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu) : Copyright (C) 2016 Free Software Foundation, Inc. : License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html : : This is free software; you are free to change and redistribute it. : There is NO WARRANTY, to the extent permitted by law.
** Python
Run the following using [[./SICE.yml]] in order to set up an identical Anaconda Python environment.
conda env create -f SICE.yml
And then run =conda activate SICE= to activate the SICE environment.
=ResolvePackageNotFound= error can be raised. In that case, run =conda env export --no-builds > environment.yml= instead.
** Parallel
parallel --version
GNU parallel 20161222 Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016 Ole Tange and Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication please cite as described in 'parallel --citation'.
** GRASS
grass --version 2>&1
GRASS GIS 7.4.0
Geographic Resources Analysis Support System (GRASS) is Copyright, 1999-2018 by the GRASS Development Team, and licensed under terms of the GNU General Public License (GPL) version >=2.
This GRASS GIS 7.4.0 release is coordinated and produced by the GRASS Development Team with contributions from all over the world.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
** GPT
~/local/snap/bin/gpt --diag
SNAP Release version 7.0 SNAP home: /home/kdm/local/snap/bin//.. SNAP debug: null SNAP log level: null Java home: /home/kdm/local/snap/jre Java version: 1.8.0_202 Processors: 8 Max memory: 18.7 GB Cache size: 1024.0 MB Tile parallelism: 8 Tile size: 512 x 512 pixels
To configure your gpt memory usage: Edit snap/bin/gpt.vmoptions
To configure your gpt cache size and parallelism: Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism}
** SNAP
** Generate ice mask
grass72 -c EPSG:3413 ./Gtmp r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask
g.region raster=icemask g.region res=500 -ap g.region zoom=icemask
d.mon start=wx0 d.erase d.rast icemask
r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o d.rast mask g.region zoom=mask
r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o exit trash Gtmp
*** Buffered ice mask
grass -c mask.tif ./Gtmp r.in.gdal input=mask.tif output=mask
r.mapcalc "ice = if(mask == 2, 1, null())" r.grow input=ice output=ice_grow radius=50 new=1
r.null ice_grow null=100 r.clump input=ice_grow output=clumps r.stats -c clumps sort=asc for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o done r.null ice_grow setnull=100
r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE" exit trash Gtmp
** Footprint
Get GL outline by drawing in Google Earth, export KML, then:
ogrinfo -al GL_outline.kml | grep LINESTRING | sed s/\ 0//g