[Bug] g.extensions failing in various versions within containers running python 3

Describe the bug Trying to add extensions using g.extension running various versions in containers results in the following,

Fetching <extension.name.here> from GRASS GIS Addons repository (be
patient)...
Compiling...
<extension.name.here.html> is missing the descriptionInstalling...
Updating extensions metadata file...
Updating extension modules metadata file...

WARNING: No metadata available for module 'extension.name.here': Unable to
         fetch interface description for command '<extension.name.here'.

         Details: <[Errno 2] No such file or directory:
         'extension.name.here'>
Installation of <extension.name.here> successfully finished

Note that despite concluding with the line about the extension installing, when I try to use the extension I get,

r.skyview: command not found

This is true of the extensions:

r.skyview
v.percolate
v.rast.bufferstats
r.valley.bottom

In the following containers and versions:

mundialis/grass-py3-pdal:latest, GRASS 8.3.dev
mundialis/grass-py3-pdal:stable-ubuntu, GRASS 7.8.8dev
mundialis/grass-py3-pdal:8.0.1-ubuntu, GRASS 8.0.1
mundialis/grass-py3-pdal:8.0.1RC1-debian GRASS 8.0.1RC1

To Reproduce 1. Using a minimal Dockerfile to override the entrypoint:

from mundialis/grass-py3-pdal:whatever_tag

ENTRYPOINT [ "bash" ]

Start like,

data=/media/data/grassdata

docker build . -t grass_container
docker run -it --rm --user=$(id -u):$(id -g) --volume $PWD:/work --volume $data:/data --env HOME=/work grass_container /work/test.sh

Where test.sh is,

#!/bin/sh

# add GRASS binaries and libraries to PATH
export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts

# adons location
export GRASS_ADDON_BASE=$HOME/.grass8/addons

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# path to GRASS settings file
export GISRC=/tmp/grass7-${USER}-$GIS_LOCK/gisrc
rm -fr /tmp/grass7-${USER}-$GIS_LOCK
mkdir /tmp/grass7-${USER}-$GIS_LOCK
export TMPDIR="/tmp/grass7-${USER}-$GIS_LOCK"

# set GISDBASE, LOCATION_NAME, and/or MAPSET
echo "GISDBASE: /data" >>$GISRC
echo "LOCATION_NAME: bishops_peak" >>$GISRC
echo "MAPSET: bishops_peak" >>$GISRC

# start in text mode
echo "GRASS_GUI: text" >>$GISRC

# this should print the GRASS version used:
g.version

# check GISDBASE, LOCATION_NAME, and MAPSET
g.gisenv

# add extensions
#g.extension extension=r.skyview operation=add
g.extension extension=v.rast.bufferstats operation=add

Expected behavior I would expect the extension to install.

System description:

In the following containers and versions:

mundialis/grass-py3-pdal:latest, GRASS 8.3.dev
mundialis/grass-py3-pdal:stable-ubuntu, GRASS 7.8.8dev
mundialis/grass-py3-pdal:8.0.1-ubuntu, GRASS 8.0.1
mundialis/grass-py3-pdal:8.0.1RC1-debian GRASS 8.0.1RC1

Containers are being run on Ubuntu 20.04.4 LTS

Thanks

I have been checking this (using the tags from https://grass.osgeo.org/download/docker/ ; see also GRASS GIS docker matrix: https://github.com/OSGeo/grass/blob/main/docker/README.md) and think to have identified the problem. The problem arises IMHO from the way your test.sh is written, i.e. how you initialize the GRASS GIS session. There seems to be something missing in your script above.

However, there is a much easier (and less error prone) way to handle addon installation. Here the modified test.sh script:

#!/bin/sh

echo "--- Linux distro used:"
cat /etc/os-release | grep 'NAME\|VERSION'

eval `cat /etc/os-release`
if [ "$NAME" == "Alpine Linux" ] ; then
   # add a few missing packages in case of Alpine
   apk add make gcc
fi

echo "--- GRASS version used:"
grass --tmp-location epsg:4326 --exec g.version -rge

echo "--- GISDBASE, LOCATION_NAME, and MAPSET:"
grass --tmp-location epsg:4326 --exec g.gisenv

## check interfaces
#grass --tmp-location epsg:4326 --exec g.version --interface-description
#grass --tmp-location epsg:4326 --exec g.version --html-description

# add extensions
grass --tmp-location epsg:4326 --exec g.extension extension=r.skyview operation=add
grass --tmp-location epsg:4326 --exec g.extension extension=v.rast.bufferstats operation=add

Next two examples for Ubuntu and Alpine:

Ubuntu

Dockerfile:

from mundialis/grass-py3-pdal:8.0.1-ubuntu
ENTRYPOINT ["bash"]

Build the image: docker build . -t grass_container

Run the test.sh script in docker (I actually use podman which doesn't need root access):

podman run -it --rm --volume .:/work --volume .:/data --env HOME=/work grass_container /work/test.sh
--- Linux distro used:
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

--- GRASS version used:
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.version -rge> ...
version=8.0.1
date=2022
revision=bf43378
build_date=2022-02-24
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=2022-02-24T21:25:18+00:00
libgis_date=2022-02-23T16:00:00+00:00
proj=6.3.1
gdal=3.0.4
geos=3.8.0
sqlite=3.31.1
Execution of <g.version -rge> finished.
Cleaning up temporary files...

--- GISDBASE, LOCATION_NAME, and MAPSET:
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.gisenv> ...
GISDBASE=/tmp/grass8-root-31
LOCATION_NAME=tmploc
MAPSET=PERMANENT
Execution of <g.gisenv> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...

Executing <g.extension extension=r.skyview operation=add> ...
Fetching <r.skyview> from GRASS GIS Addons repository (be patient)...
Compiling...
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...
Installation of <r.skyview> successfully finished
Execution of <g.extension extension=r.skyview operation=add> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.extension extension=v.rast.bufferstats operation=add> ...
Fetching <v.rast.bufferstats> from GRASS GIS Addons repository (be
patient)...
Compiling...
<v.rast.bufferstats.html> is missing the descriptionInstalling...
Updating extensions metadata file...
Updating extension modules metadata file...
Installation of <v.rast.bufferstats> successfully finished
Execution of <g.extension extension=v.rast.bufferstats operation=add> finished.
Cleaning up temporary files...

Alpine Linux

Dockerfile:

from mundialis/grass-py3-pdal:8.0.1-alpine
ENTRYPOINT ["bash"]

Build the image: docker build . -t grass_container

Run the test.sh script in docker (again using podman which doesn't need root access):

podman run -it --rm --volume .:/work --volume .:/data --env HOME=/work grass_container /work/test.sh
--- Linux distro used:
NAME="Alpine Linux"
VERSION_ID=3.12.9
PRETTY_NAME="Alpine Linux v3.12"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/8) Installing binutils (2.34-r2)
(2/8) Installing isl (0.18-r0)
(3/8) Installing libatomic (9.3.0-r2)
(4/8) Installing libgphobos (9.3.0-r2)
(5/8) Installing mpfr4 (4.0.2-r4)
(6/8) Installing mpc1 (1.1.0-r1)
(7/8) Installing gcc (9.3.0-r2)
(8/8) Installing make (4.3-r0)
Executing busybox-1.31.1-r21.trigger
OK: 465 MiB in 140 packages
--- GRASS version used:
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.version -rge> ...
version=8.0.1
date=2022
revision=bf43378
build_date=2022-02-24
build_platform=x86_64-pc-linux-musl
build_off_t_size=8
libgis_revision=2022-02-24T21:02:14+00:00
libgis_date=2022-02-24T21:02:14+00:00
proj=7.0.1
gdal=3.1.4
geos=3.8.1
sqlite=3.32.1
Execution of <g.version -rge> finished.
Cleaning up temporary files...
--- GISDBASE, LOCATION_NAME, and MAPSET:
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.gisenv> ...
GISDBASE=/tmp/grass8-root-23
LOCATION_NAME=tmploc
MAPSET=PERMANENT
Execution of <g.gisenv> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.extension extension=r.skyview operation=add> ...
WARNING: Extension <r.skyview> already installed. Re-installing...
Fetching <r.skyview> from GRASS GIS Addons repository (be patient)...
Compiling...
Installing...
Updating extensions metadata file...
Updating extension modules metadata file...
Installation of <r.skyview> successfully finished
Execution of <g.extension extension=r.skyview operation=add> finished.
Cleaning up temporary files...
Starting GRASS GIS...
Creating new GRASS GIS location <tmploc>...
Cleaning up temporary files...
Executing <g.extension extension=v.rast.bufferstats operation=add> ...
WARNING: Extension <v.rast.bufferstats> already installed. Re-installing...
Fetching <v.rast.bufferstats> from GRASS GIS Addons repository (be
patient)...
Compiling...
<v.rast.bufferstats.html> is missing the descriptionInstalling...
Updating extensions metadata file...
Updating extension modules metadata file...
Installation of <v.rast.bufferstats> successfully finished
Execution of <g.extension extension=v.rast.bufferstats operation=add> finished.
Cleaning up temporary files...

Overall, it works both in Ubuntu and Alpine, hence I think that your session creation approach misses some steps. So I suggest to use the official --exec way which simplifies it a lot. See also:

https://grass.osgeo.org/grass-stable/manuals/grass.html#batch-jobs-with-the-exec-interface

OSGeo / grass

[Bug] g.extensions failing in various versions within containers running python 3 #2368

Ubuntu

Alpine Linux