Project generation flexibility via environment variables/logic

[noelmcloughlin]

Here are issues I have been experiencing with linkml-cookiecutter:

Per https://github.com/linkml/linkml-project-cookiecutter/issues/51 we have limited control over linkml generators via gen-project. Furthermore, we cannot pass arguments to GitHub Actions to control documentation generation.Environment variables are good practice (see https://12factor.net/config) so enhanced support would be useful. To implement a good solution I propose:

• keep existing behavior and do not force anything on existing happy users.
• store environment variables in a common file named custom.env.
• Enhance Makefile and GitHub actions to read environment variables from that file.
• enhance Makefile to run custom generators if a environment variable was set.
• enhance GitHub Actions to run custom doc-generation if environment variable was set.
• during testing, update Makefile to fix any bugs noticed.

[noelmcloughlin]

The files about.yaml and utils/get-value.sh can be decommissioned if we switch to environment variables defined in config.env

LINKML_SCHEMA_NAME: {{cookiecutter.__project_slug}}
LINKML_SCHEMA_AUTHOR: {{cookiecutter.__author}}
LINKML_SCHEMA_DESCRIPTION: {{cookiecutter.project_description}}
LINKML_SCHEMA_SOURCE_PATH: {{cookiecutter.__source_path}}
LINKML_SCHEMA_GOOGLE_SHEET_ID: {{cookiecutter.google_sheet_id}}
LINKML_SCHEMA_GOOGLE_SHEET_TABS: {{cookiecutter.google_sheet_tabs}}

In the Makefile these lines can be deleted.

SCHEMA_NAME = $(shell ${SHELL} ./utils/get-value.sh name)
SOURCE_SCHEMA_PATH = $(shell ${SHELL} ./utils/get-value.sh source_schema_path)
SHEET_ID = $(shell ${SHELL} ./utils/get-value.sh google_sheet_id)
SHEET_TABS = $(shell ${SHELL} ./utils/get-value.sh google_sheet_tabs)

Finally we update Makefile to fix broken references using our environment vars.

SHEET_MODULE_PATH = $(LINKML_SCHEMA_SOURCE_PATH:)/$(SHEET_MODULE).yaml
PYMODEL = $(SRC)/$(LINKML_SCHEMA_NAME)/datamodel
$(RUN) sheets2linkml --gsheet-id $(LINKML_SCHEMA_GOOGLE_SHEET_ID) $(LINKML_SCHEMA_GOOGLE_SHEET_TABS) > $(LINKML_SCHEMA_SOURCE_PATH:).tmp && mv $(LINKML_SCHEMA_SOURCE_PATH:).tmp $(LINKML_SCHEMA_SOURCE_PATH:)

etc ,,,

If this sounds reasonable I can raise PR.

[noelmcloughlin]

In future it would be ideal to make following configurable via environment variables.

SRC = src
DEST = project
PYMODEL = $(SRC)/${SCHEMA_NAME}/datamodel
DOCDIR = docs
EXAMPLEDIR = exampledir

Makefile.project should inherit environment variables exported from Makefile ?? @matentzn said ... we use it in SSSOM for example for test file validation, and there we plan to use it also for things like generating documentation pages eg https://github.com/monarch-initiative/monarch-documentation/blob/main/project.Makefile#L15

[pkalita-lbl]

Just my two cents, but I would definitely support more consistent use of environment variables instead of the slightly non-standard about.yaml and get-value.sh. If we do go for more of a environment variable-first approach I could even imagine using more standard file names (.env instead of config.env, .env.custom instead of custom.env)

[noelmcloughlin]

Agree