Incompatibilities between NCI and DEA Sandbox environments

[robbibt]

Issue to record any inconsistencies we find between the NCI and DEA Sandbox environments, including:

• Any modules/python packages not installed on both environments
• Any missing Jupyter lab extensions
• Any key datasets not indexed on both environments
• etc

[cbur24]

• [x] The python module scikit-learn is not installed on the DEA Sandbox - Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/12

• [x] datacube_stats is not installed on the DEA Sandbox (work is currently being done to make it compatible, a local install will work for some stats) - Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/16

[robbibt]

• [x] I've now thoroughly tested the OTPS tidal model that @emmaai recently installed on the Sandbox, and it looks like everything is working well

[robbibt]

• [ ] Sentinel 2 Near-Real-Time data isn't currently available on the NCI

[BexDunn]

• [x] module digitalearthau is not available on the DEA Sandbox. This module has the wofs fusing function in it - Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/15

Temporary fix is to run this code from the terminal:

pip install --user 'git+https://github.com/GeoscienceAustralia/digitalearthau.git'

Update: resolved by moving wofs_fuser to DEA data handling: https://github.com/GeoscienceAustralia/dea-notebooks/pull/403

[robbibt]

• [x] wofs_albers currently needs the user to specify output_crs and resolution on the Sandbox, which isn't necessary on the NCI (see https://github.com/GeoscienceAustralia/dea-config/issues/405)

Resolved on 23/09/2019

[robbibt]

• [ ] There is no DEM available on the sandbox (on the NCI, we have the 30 m SRTM DEM e.g. dc.load(product='srtm_dem1sv1_0', **query))

[uchchwhash]

If there's no security concerns that prevent us from doing so, having ssh would be nice on the sandbox.

[Kirill888]

@uchchwhash one can always download whatever binary and run it, so...

[CEKrause]

• [x] The sandbox is missing descartes, which is required to plot shapefiles using geopandas. - Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/13

Fixed by https://github.com/GeoscienceAustralia/dea-notebooks/issues/344#issuecomment-543428699

[cbur24]

The remote sensing package rsgislib cannot be installed on the Sandbox as it requires a conda installation. https://www.rsgislib.org/download.html

[cbur24]

The version of GDAL on the Sandbox does not appear to have the .kea driver. https://gdal.org/drivers/raster/kea.html

[robbibt]

• [x] The seaborn module for producing scientific plots is currently missing from the Sandbox (e.g. import seaborn as sns)

Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/22

Fixed by https://github.com/GeoscienceAustralia/dea-notebooks/issues/344#issuecomment-543428699

[robbibt]

• [x] ffmpeg is missing from the Sandbox. This is needed for generating animations based on satellite data

Issue: https://github.com/GeoscienceAustralia/dea-sandbox/issues/23

Fixed by https://github.com/GeoscienceAustralia/dea-notebooks/issues/344#issuecomment-543428699

[robbibt]

• [x] Virtual products functionality requires a version of datacube-core > 1.7. This is currently not the case on the sandbox

Fixed by https://github.com/GeoscienceAustralia/dea-notebooks/issues/344#issuecomment-543428699

[uchchwhash]

Happy to report that the incompatibilities are now less in number:

1. datacube-core has been updated to virtual products' liking
2. ffmpeg seems to work
3. seaborn, scikit-learn, datacube-stats should be there
4. I may have missed descartes, apologies
5. the .kea and rsgislib story is more complicated, sorry @cbur24 !

[uchchwhash]

Update!

1. descartes should now also be available

Please restart your servers in the sandbox if you need this.

[robbibt]

• %load_ext line_profiler is not installed on the DEA Sandbox. This is really useful for trouble-shooting code, and it would be great to have it available.

[robbibt]

• [ ] Radar products such as s1_gamma0_geotif_scene are not currently available on the NCI datacube

[robbibt]

This has served its purpose, closing for now