wri / gfw_forest_loss_geotrellis

Global Tree Cover Loss Analysis using Geotrellis and SPARK
MIT License
10 stars 8 forks source link

GTC-2683 Upgrade to GDAL 3.8.3 and Miniconda3 #242

Closed danscales closed 3 months ago

danscales commented 3 months ago

GTC-2683 Upgrade to GDAL 3.8.3.

This allows upgrading to Miniconda3 as well (GTC-2774), which is much more recent that the old Miniconda that we've been using and seems to run quite a bit faster. GDAL 3.8.3 is also compatible with EMR-serverless, if we want to use it for certain jobs.

solomon-negusse commented 3 months ago

@danscales: recommending again testing out libmamba solver that may give you more speed up.. i got an order of magnitude speed up on personal project (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community).

danscales commented 3 months ago

@danscales: recommending again testing out libmamba solver that may give you more speed up.. i got an order of magnitude speed up on personal project (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community).

@solomon-negusse I think the much newer conda (downloaded by the much newer Miniconda3) may be using libmamba. The solver is much faster and only takes about 15 seconds. There is only one conda line now, since we load everything from the same conda-forge repo (don't use the default anaconda repo). The total time for the update is now only a minute, of which the 45 seconds remaining (i.e. which is not the solving) is downloading and unpacking the 2GB of packages.