Proposal to Add Environment Variable for Extension Installation

[Llewellynvdm]

We have been experimenting with the idea of adding a new environment variable to the Joomla Docker image: JOOMLA_EXTENSIONS_URLS. This variable will allow users to specify extension package URLs, separated by semicolons, which will be installed into the Joomla website immediately after the auto-deployment is completed at the initial deployment of the container.

Implementation Details

We have tested this feature in our personal images on Octoleo, and it works as intended. Below are the key parts of the implementation:

URL Validation (Line 181)

We use the following function to validate each URL provided:

validate_url() {
        if [[ $1 =~ ^http(s)?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}(/.*)?$ ]]; then
                return 0
        else
                return 1
        fi
}

Extension Installation (Line 219)

The installation logic is as follows:

# Install any extensions found in the extensions URLs environment variable
if [[ -n "${JOOMLA_EXTENSIONS_URLS}" && "${#JOOMLA_EXTENSIONS_URLS}" -gt 2 ]]; then
        IFS=';' read -ra ADDR <<< "$JOOMLA_EXTENSIONS_URLS"
        for extension_url in "${ADDR[@]}"; do
                # Validate each URL
                if validate_url "$extension_url"; then
                        # Try to install the Joomla extension from the URL
                        if php cli/joomla.php extension:install --url "${extension_url}"; then
                                echo >&2 "Successfully installed ${extension_url}"
                        else
                                echo >&2 "Failed to install ${extension_url}"
                        fi
                else
                        echo >&2 "Invalid URL: ${extension_url}"
                fi
        done
fi

If the JOOMLA_EXTENSIONS_URLS variable is not set or is empty, this installation step will be skipped. If any URL is invalid or if an installation fails, an error message will be output to the standard error, and the script will proceed. Joomla will still fully deploy regardless of these errors.

Benefits

This feature will allow for the deployment of a Joomla website with pre-installed extensions, significantly enhancing the usability and flexibility of the Docker image.

Request for Feedback

We have received approval from the CMS maintenance team to add this feature. However, we would appreciate feedback on how to implement this in the best possible way, ensuring it adheres to all Docker and Joomla standards.

Thank you for considering this proposal. We look forward to your feedback and suggestions.

[Fedik]

I think it is a good idea. However it probably need to offer also install from local? I think need both JOOMLA_EXTENSIONS_URLS and JOOMLA_EXTENSIONS_PATHS.

[Llewellynvdm]

@Fedik I will move forward with the JOOMLA_EXTENSIONS_URLS as an initial step, and then we can start looking the paths, as this will of necessity imply mounting a local file or folder to the container right? This is a far larger process and while I am willing to go that direction, I would like to get this initial part implemented asap.

@J0WI what do you think, and is there any changes to the script you might suggest?

[Fedik]

For beginning maybe It could be just as customisation for own containers. Kind of:

FROM joomla

COPY ./path/to/local-extensions /tmp/joomla/extensions

ENV JOOMLA_EXTENSIONS_URLS=https://example.com/extension1.zip https://example.com/extension2.zip
ENV JOOMLA_EXTENSIONS_PATHS=/tmp/joomla/extensions/local-extension1.zip /tmp/joomla/extensions/local-extension2.zip

In theory, but maybe I missing something. It is not require mounting a local file or folder.

[Llewellynvdm]

Yes that is the gist, but that is not the script, and the needed validation that the paths exist.

since there will always be those who will load the JOOMLA_EXTENSIONS_PATHS with values and not COPY ./path/to/local-extensions /tmp/joomla/extensions in which case we must give the correct notice.

On a side note, the value must be semicolon separated not spaces, and very important, you must add all the values needed for auto-deployment else the container will not act on the JOOMLA_EXTENSIONS_URLS value.

[Llewellynvdm]

Here is an update to the script:

# Function to log messages
joomla_log() {
    local msg="$1"
    echo >&2 " $msg"
}

# Function to log info messages
joomla_log_info() {
    local msg="$1"
    echo >&2 "[INFO] $msg"
}

# Function to log warning messages
joomla_log_warning() {
    local msg="$1"
    echo >&2 "[WARNING] $msg"
}

# Function to log error messages
joomla_log_error() {
    local msg="$1"
    echo >&2 "[ERROR] $msg"
}

# Function to set a line
joomla_line() {
    echo >&2 "========================================================================"
}

# Function to set a line at end
joomla_line_start() {
    joomla_line
    echo >&2
}

# Function to set a line at end
joomla_line_end() {
    echo >&2
    joomla_line
}

# Function to give final success message (1)
joomla_success() {
    joomla_log "This server is now configured to run Joomla!"
}

# Function to give final success message (2)
joomla_success_need_db() {
    joomla_success
    echo >&2
    joomla_log " NOTE: You will need your database server address, database name,"
    joomla_log "       and database user credentials to install Joomla."
}

# Function to validate URLs
joomla_validate_url() {
    if [[ $1 =~ ^http(s)?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}(/.*)?$ ]]; then
        return 0
    else
        return 1
    fi
}

# Function to validate paths
joomla_validate_path() {
    if [[ -f $1 ]]; then
        return 0
    else
        return 1
    fi
}

# Function to split values by semicolon
joomla_split_values() {
    local input=$1
    local -n arr=$2
    IFS=';' read -ra arr <<< "$input"
}

# Function to install extension from URL
joomla_install_from_url() {
    local url=$1
    if joomla_validate_url "$url"; then
        if php cli/joomla.php extension:install --url "$url" --no-interaction; then
            joomla_log_info "Successfully installed $url"
        else
            joomla_log_error "Failed to install $url"
        fi
    else
        joomla_log_error "Invalid URL: $url"
    fi
}

# Function to install extension from path
joomla_install_from_path() {
    local path=$1
    if joomla_validate_path "$path"; then
        if php cli/joomla.php extension:install --path "$path" --no-interaction; then
            joomla_log_info "Successfully installed $path"
        else
            joomla_log_error "Failed to install $path"
        fi
    else
        joomla_log_error "Invalid Path: $path"
    fi
}

# Install any extensions found in the extensions urls env
if [[ -n "${JOOMLA_EXTENSIONS_URLS}" && "${#JOOMLA_EXTENSIONS_URLS}" -gt 2 ]]; then
    joomla_split_values "$JOOMLA_EXTENSIONS_URLS" JURLS
    for extension_url in "${JURLS[@]}"; do
        joomla_install_from_url "$extension_url"
    done
fi

# Install any extensions found in the extensions paths env
if [[ -n "${JOOMLA_EXTENSIONS_PATHS}" && "${#JOOMLA_EXTENSIONS_PATHS}" -gt 2 ]]; then
    joomla_split_values "$JOOMLA_EXTENSIONS_PATHS" JPATHS
    for extension_path in "${JPATHS[@]}"; do
        joomla_install_from_path "$extension_path"
    done
fi

It can be tested with:

FROM llewellyn/joomla

COPY ./path/to/local-extensions /tmp/joomla/extensions

ENV JOOMLA_EXTENSIONS_URLS=https://example.com/extension1.zip;https://example.com/extension2.zip
ENV JOOMLA_EXTENSIONS_PATHS=/tmp/joomla/extensions/local-extension1.zip;/tmp/joomla/extensions/local-extension2.zip

Note the separator is: ;

The whole entrypoint.sh script has been refactored, I am testing it to see if its stable.

[rdeutz]

Good idea

[Fedik]

Looks good to me.

IFS=';' read -ra arr <<< "$input"

Not that the code should reset IFS to default, after this? (with unset IFS)

Note the separator is: ;

Is it something common on the Docker lands? :) Personaly I would keep default bash behavior, however maybe I missing something :)

[Llewellynvdm]

Is it something common on the Docker lands? :)

Not really a docker thing, just a safer thing :)

Here is an update on the function:

# Function to split values by semicolon
joomla_split_values() {
    local input=$1   # The input string to be split
    local -n arr=$2  # The array to store the split values (passed by reference)
    local old_IFS=$IFS  # Save the original IFS value

    IFS=';' read -ra arr <<< "$input"  # Split the input by semicolon and store in array

    IFS=$old_IFS  # Restore the original IFS value
}

Using a specific delimiter like a semicolon reduces the chance of accidental splitting. For example, URLs and paths often contain spaces, but using semicolons as delimiters ensures that these elements remain intact when splitting.

I considered handling both space and semicolon delimiters (that it will first try semicolon, and if not found try space), but experience has taught me that this can lead to misunderstandings in the long run. Therefore, we stick with the semicolon (;) and avoid misunderstanding.

[Fedik]

Okay, then I would suggest to take in to account a new line, at least. Because with default behavior we also can do:

VARIABLE="foobar1
foobar2
foobar3"

(Not sure if it working in .Docker file, but it is valid for bash)

I think following IFS should work for both ; and new line:

IFS=";"$'\n'

At least in my quick test

[Llewellynvdm]

Passing environment variables with newlines directly in Docker commands, Dockerfiles, or Docker Compose files requires specific handling because Docker's environment variable processing can interpret newlines differently.

Docker Command

When using the docker run command, you can pass environment variables with newlines using printf or similar methods. For example:

docker run -e "MULTILINE_VAR=$(printf 'line1\nline2\nline3')" your_image

Dockerfile

In a Dockerfile, you can use the ENV instruction to define an environment variable with newlines. Make sure to use a method that preserves the newlines. For example:

FROM ubuntu:latest

# Define the environment variable with newlines
ENV MULTILINE_VAR="line1\nline2\nline3"

# Or using a heredoc syntax
RUN echo -e "line1\nline2\nline3" > /tmp/multiline.txt && \
    export MULTILINE_VAR=$(cat /tmp/multiline.txt)

Docker Compose

In a Docker Compose file, you can define environment variables with newlines by using a block scalar in YAML. Here's how to do it:

services:
  web:
    image: nginx
    environment:
      - MULTILINE_VAR=|-
        line1
        line2
        line3

Practical Example

Docker Command

docker run -e "MULTILINE_VAR=$(printf 'line1\nline2\nline3')" nginx

Dockerfile

FROM ubuntu:latest

# Define the environment variable with newlines
ENV MULTILINE_VAR="line1\nline2\nline3"

CMD ["bash", "-c", "echo -e \"$MULTILINE_VAR\""]

Docker Compose File

services:
  web:
    image: nginx
    environment:
      - MULTILINE_VAR=|-
        line1
        line2
        line3
    command: ["bash", "-c", "echo -e \"$MULTILINE_VAR\""]

Conclusion

You can pass environment variables with newlines in Docker commands, Dockerfiles, and Docker Compose files by using appropriate techniques to preserve the newline characters. But they all seem kinda overkill and prone to complicate stuff, so while we can in BASH deal with this easy, docker will work far better with just ; separated string.

I am really not trying to be difficult about this, but I know getting this official images merged upstream we need to avoid complicating thing even more than they already are.

We are already pushing up against our limits as some consider these changes to effect the immutability of the images, so getting this feature accepted would be my first goal.

Then we have to add update functionality to the images #161 (another big one)

[Fedik]

Thanks for explanation