Adapt cookiecutter to dynamic services

[sanderegg]

Differences to computational services:

• dynamic services need additional ENV to be defined (SIMCORE_HOST_NAME, ...)
• dynamic services are not just run like a computation service
• testing? how to test a dynamic services is running as intended?
• add alpine and ubuntu flavors
• use /bash as shell instead of sh
• move tools/ to a library such that cookie cutter can use latest?
• in windows ALL files copied to the container shall have LR endings sed -i 's/\r$//' file.txt

@odeimaiz from dash services

• travis.sh missing
• devel-config missing
• comp -> dynamic
• copy docker/scripts/healtcheck_curl_host.py
• in boot.sh
  • CREATE_DUMMY_TABLE...
  • python /home/jovyan/src/${APP_URL}
• copy Dockerfile from mattward
• copy .env-devel content from mattward
• add pylint to test requirements
• remove entrypoint.sh
• remove run_creator.py
• update test_folder_structure.py
• set -e ->

  # http://redsymbol.net/articles/unofficial-bash-strict-mode/
  set -euo pipefail
  IFS=$'\n\t'

[sanderegg]

also update the latest Makefile/docker-compose practices

[KZzizzle]

Based on an epic debug session with Sylvain and Pedro, we learned a few things

1. Line endings are VERY important. VERY VERY VERY IMPORTANT. Make sure everything is LF ant not CRLF
2. For ubuntu, we need the bash shell instead of sh, which may throw off some cookie cutter template commands
3. String formatting might be an issue depending on the service (i.e. with or without quotes when passing to the Docker container): ${VARSTRING_WITH_QUOTES//"} will give the string without quotes
4. There are problems looming about the user permissions when we create user scu. Some services may need root access: our solution was to give the absolute path of the command which would save a file into a local directory
5. In WSL, temporary folders cannot be mounted, which will be problematic for input/output folders.

[KZzizzle]

6. For some weird reason, bumpversion and WSL do not play nicely together (but perhaps only on my computer)

[KZzizzle]

7. .pylintrc is missing from the repo

[KZzizzle]

8. Travis integration is not yet in cookiecutter

[pcrespov]

1. Docker base flavors:
   • alpine (??)
   • ubuntu (e.g. opncore)
   • ubuntu-matlab-exe (e.g. ELE) with BASH
2. describe unit and integration tests based on different scenarios
3. metadata in a single yaml file
4. Makefile
   • make devenv
   • make build test
   • make publish
5. keep track two versions
6. metadata with cwl standard

[sanderegg]

• please consider using phusion baseimage instead of pure ubuntu (docker freindly image - in use for a few computational services already)
• also add github CI integration + gitlab + maybe travis

[KZzizzle]

A few "flavors" offered by codeocean - maybe we can copy them? We already have python and ubuntu, maybe R would be good to have in the future.

[pcrespov]

@GitHK once we have the pilot projects in place, we could update this cookie-cutter

[elisabettai]

@pcrespov, do you remember what you had in mind for the pilot projects you mentioned more than a year ago?

I'm about to create a new dynamic service (based on plotly dash) and I'm thinking to start with the cookiecutter for comp service and hybridize it with Jupyter Math (to get the compatibility with dy-sidecar). I'm wondering if by doing so I can help making a cookiecutter for dynamic services.

[pcrespov]

@pcrespov, do you remember what you had in mind for the pilot projects you mentioned more than a year ago?

@elisabettai pilot project? You mean the "capsules" concept?

I'm about to create a new dynamic service (based on plotly dash) and I'm thinking to start with the cookiecutter for comp service and hybridize it with Jupyter Math (to get the compatibility with dy-sidecar). I'm wondering if by doing so I can help making a cookiecutter for dynamic services.

IMO the dynamic service should not have a different cookie-cutter. Every new repository is very difficult to maintain and you need to have strong justifications to go that way.

I think it is better to talk offline