Plugin installation not working

[ericpre]

When trying to figure out https://github.com/nomad-coe/nomad-schema-plugin-example/issues/4, I tried to implement the installation instructions from https://nomad-lab.eu/prod/v1/staging/docs/plugins/plugins.html#add-a-plugin-to-your-nomad into a bash script to try to make it reproducible but I can't get the plugin installation to work. Here are the scripts:

#!/bin/bash

# Set variable
# #############
git_repo=https://github.com/nomad-coe/nomad-schema-plugin-example.git

basename=$(basename $git_repo)
foldername=${basename%.*}

# clean current folder
rm nomad-oasis.zip
rm -rf nomad-oasis

# Download/unzip configuration file, set permissions
# ##################################################
wget https://nomad-lab.eu/prod/v1/staging/docs/assets/nomad-oasis.zip
unzip nomad-oasis.zip

cd nomad-oasis
sudo chown -R 1000 .volumes

# Clone plugins and set permission
# ################################
git clone $git_repo
sudo chown -R 1000 $foldername

# Update `nomad.yaml` with plugin metadata 
# ########################################

plugin_config_file_name=$(find $foldername -name "nomad.yaml")
echo "###################"
echo "Plugin config path:"
echo $plugin_config_file_name

cat $plugin_config_file_name >> configs/nomad.yaml

# get plugin name and pluging module name
plugin_name=$(shyaml get-value plugins.include < $plugin_config_file_name)
echo "############"
echo "Plugin name:"
echo $plugin_name

plugin_module_name=$(shyaml get-value plugins.options.$plugin_name.python_package < $plugin_config_file_name)
echo "###################"
echo "Plugin module name:"
echo $plugin_module_name

# Create `docker-compose.plugins.yaml`
# ####################################

cat >> docker-compose.plugins.yaml << EOF
services:
    worker:
        volumes:
            - ./$foldername/$plugin_module_name:/app/plugins/$plugin_module_name
    app:
        volumes:
            - ./$foldername/$plugin_module_name:/app/plugins/$plugin_module_name
EOF

and to start docker:

#!/bin/bash
cd nomad-oasis
docker compose down
export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml
docker compose up -d
curl localhost/nomad-oasis/alive

Any idea what is wrong? When I use nomad-oasis-with-plugins.zip, it works fine!

[lauri-codes]

Hi @ericpre!

Sorry for the late reply. Have you had any luck with this? There could be an issue with the permissions, but with a quick glance, you seem to be setting permissions just fine. I will try to reproduce the problem.

[ericpre]

No, I am very much stuck!

[lauri-codes]

I may have found the issue: your script sets up nomad.yaml like this:

plugins:
  include: 'schemas/example'
  options:
    schemas/example:
      python_package: nomadschemaexample

include should here be a list instead of a string value. So using this instead should work:

plugins:
  include:
    - 'schemas/example'
  options:
    schemas/example:
      python_package: nomadschemaexample

Let me know if this works out.

[ericpre]

I have tried by changing the nomad.yaml file from the plugin code (https://github.com/ericpre/nomad-schema-plugin-example/blob/main/nomad.yaml) and I get the same - the example schema is not added to the "other" build-in schema.

The content of the configs/nomad.yaml is now:

services:
  api_host: 'localhost'
  api_base_path: '/nomad-oasis'

oasis:
  is_oasis: true
  uses_central_user_management: true

north:
  jupyterhub_crypt_key: '978bfb2e13a8448a253c629d8dd84ff89587f30e635b753153960930cad9d36d'

meta:
  deployment: 'oasis'
  deployment_url: 'https://my-oasis.org/api'
  maintainer_email: 'me@my-oasis.org'

logstash:
  enable: false

mongo:
    db_name: nomad_oasis_v1

elastic:
    entries_index: nomad_oasis_entries_v1
    materials_index: nomad_oasis_materials_v1
normalize:
  normalizers:
    include:
      - MetainfoNormalizer
plugins:
  # We only include our schema here. Without the explicit include, all plugins will be
  # loaded. Many build in plugins require more dependencies. Install nomad-lab[parsing]
  # to make all default plugins work.
  include:
    - 'schemas/example'
  options:
    schemas/example:
      python_package: nomadschemaexample

[lauri-codes]

I tried this locally, and I do see the schema in the GUI. I just ran your script with the addition of fixing the include to a list.

Note that you should see the schema as Other->ExampleSection. If you still can't get it working, I would look if there are some errors in the logs. You can get some of the logs by removing the -d flag from the docker compose up call.

[ericpre]

Thank you @lauri-codes for your help, there was indeed an error in the log. After changing the nomad.yaml file to a list as suggested above, the bash script were broken and the docker-compose.plugins.yaml generated by this script was incorrect.

The updated script is below - the fix is the .0 when parsing the "plugin name".

#!/bin/bash

# Set variable
# #############
#git_repo=https://github.com/nomad-coe/nomad-schema-plugin-example.git
# plugin with list in `nomad.yaml`
git_repo=https://github.com/ericpre/nomad-schema-plugin-example.git

basename=$(basename $git_repo)
foldername=${basename%.*}

# clean current folder
rm nomad-oasis.zip
rm -rf nomad-oasis

# Download/unzip configuration file, set permissions
# ##################################################
wget https://nomad-lab.eu/prod/v1/staging/docs/assets/nomad-oasis.zip
unzip nomad-oasis.zip

cd nomad-oasis
sudo chown -R 1000 .volumes

# Clone plugins and set permission
# ################################
git clone $git_repo
sudo chown -R 1000 $foldername

# Update `nomad.yaml` with plugin metadata 
# ########################################

plugin_config_file_name=$(find $foldername -name "nomad.yaml")
echo "###################"
echo "Plugin config path:"
echo $plugin_config_file_name

cat $plugin_config_file_name >> configs/nomad.yaml

# get plugin name and pluging module name
plugin_name=$(shyaml get-value plugins.include.0 < $plugin_config_file_name)
echo "############"
echo "Plugin name:"
echo $plugin_name

plugin_module_name=$(shyaml get-value plugins.options.$plugin_name.python_package < $plugin_config_file_name)
#plugin_module_name=$(shyaml get-value plugins.options.schemas/example.python_package < nomad-schema-plugin-example/nomad.yaml)

echo "###################"
echo "Plugin module name:"
echo $plugin_module_name

# Create `docker-compose.plugins.yaml`
# ####################################

cat >> docker-compose.plugins.yaml << EOF
services:
    worker:
        volumes:
            - ./$foldername/$plugin_module_name:/app/plugins/$plugin_module_name
    app:
        volumes:
            - ./$foldername/$plugin_module_name:/app/plugins/$plugin_module_name
EOF

And the script to start the docker image:

#!/bin/bash
cd nomad-oasis
docker compose down

if [ "$1" = "prune" ]
then
  docker system prune --volumes
  docker system prune -a
fi

export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml

if [ "$1" = "out" ]
then
  docker compose up > log.txt
else
  docker compose up -d
fi

curl localhost/nomad-oasis/alive