Regression in template variables handling (dollar sign)

thibaut-atempo commented 2 years ago

Bug description I've made a lot of testings and I think I found a regression between Portainer 2.15.1 and 2.16.0 (both CE and BE versions). I'm using my own App templates (using a custom template definition URL). Same templates are working fine with 2.15.1 but not with 2.16.0. Some of my templates have variables and some of these variables contains a dollar sign. The dollar sign was perfectly handled in 2.15.1 and I did not have to use a double dollar sign for a single dollar value. Now, with 2.16.0, it wont deploy stack if I have a single dollar sign in the variable value :

Expected behavior Successfully deploy a stack using a single dollar in some variables. For example : setting Password$ in variable in Poirtainer Web UI will set the environment variables correctly for the stack and I can get this exact value in container.

Portainer Logs

Steps to reproduce the issue:

Use an App template definition file with some env variables to set on a type 3 model (docker compose stack). Try to deploy with a single $ in any of the variables Get "invalid template error"

Technical details:

Portainer version: 2.16.0 (tested on both CE and BE)
Docker version (managed by Portainer): 20.10.21
Platform (windows/linux): Debian 11
Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):
Browser: Chrome latest
Use Case (delete as appropriate): Using Portainer at Home
Have you reviewed our technical documentation and knowledge base? Yes

Additional context /

tamarahenson commented 1 year ago

@thibaut-atempo

Thank you for the information. I am able to reproduce. I am going to forward to Product. Updates to follow.

Thanks!

tamarahenson commented 1 year ago

@thibaut-atempo

Apologies for delay in response. An internal issue has been logged and reviewed with Product. I do not yet have a timeline or release.

A workaround we have with the $ in environment variables is to use single quotes ' '. Will that work for you in this case?

Thanks!

thibaut-atempo commented 1 year ago

I already worked around by replacing all dollar signs in my templates. So it's OK for me. Thank you. Please note that it is also causing some issues when trying to remove a previously deployed stack having dollar signs but it's easy to fix by removing the sign before removing it.

kale1d0code commented 1 year ago

@thibaut-atempo how did you remove the dollar sign symbol from a stack which already had one?

thibaut-atempo commented 1 year ago

@thibaut-atempo how did you remove the dollar sign symbol from a stack which already had one?

Open the deployed stack and select edit in top menu then remove dollar sign from environments variables values and/or from compose YAML. It will redeploy immediately without dollar sign then you should be able to remove it. Hope it helps.

kale1d0code commented 1 year ago

I tried doing the same and got a 500 error message.

PUT https://<REDACTED>/api/stacks/174?endpointId=9 500

{
    "data": {
        "message": "failed to deploy a stack: Invalid template: \"<ypmU61hd$(dE.JeW_xYBHOWSBGz3EB6\"\n",
        "details": "failed to deploy a stack: Invalid template: \"<ypmU61hd$(dE.JeW_xYBHOWSBGz3EB6\"\n"
    },
    "status": 500,
    "config": {
        "method": "PUT",
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "jsonpCallbackParam": "callback",
        "ignoreLoadingBar": true,
        "data": {
            "id": 174,
            "StackFileContent": "version: '3.6'\n\nnetworks:\n  public:\n    external:\n      name: public\n#  frontend:\n  backend:\n  \nvolumes:\n  dbs:\n    driver: local\n  app:\n    driver: local\n  \nservices:\n  orange:\n    image: orangehrm/orangehrm:${ORANGE_VERSION}\n    restart: always\n    labels:\n    - traefik.enable=true\n    - traefik.docker.network=public\n    - traefik.http.routers.${ROUTER_NAME}.rule=${URL_CONFIG}\n    - traefik.http.routers.${ROUTER_NAME}.entrypoints=websecure,web\n    - traefik.http.routers.${ROUTER_NAME}.tls=true\n    - traefik.http.routers.${ROUTER_NAME}.tls.certresolver=default\n    - traefik.http.services.${ROUTER_NAME}.loadbalancer.server.port=80\n    environment:\n#    - ORANGEHRM_DATABASE_HOST=dbms\n#    - ORANGEHRM_DATABASE_USER=${DB_USER}\n#    - ORANGEHRM_DATABASE_NAME=${DB_NAME}\n#    - ORANGEHRM_DATABASE_PASSWORD=${DB_PASS}\n    - TIME_ZONE=${TIME_ZONE}\n    networks:\n    - public\n    - backend\n    volumes:\n    - app:/var/www/html\n  \n  dbms:\n    image: linuxserver/mariadb:${DBMS_VERSION}\n    restart: always\n    environment:\n    - MYSQL_DATABASE=${DB_NAME}\n    - MYSQL_USER=${DB_USER}\n    - MYSQL_PASSWORD=${DB_PASS}\n    - DATADIR=/var/lib/mysql\n    - TZ=${TIME_ZONE}\n    volumes:\n#    - /var/vhost/config/mysql/temptables/xlarge.cnf:/etc/my.cnf.d/override_temptables.cnf\n#    - /var/vhost/config/mysql/innodb/buffer_pool_large.cnf:/etc/my.cnf.d/override_buffer_pool.cnf\n    - dbs:/var/lib/mysql\n    healthcheck:\n      test: out=$$(mysqladmin ping -h localhost -P 3306 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD); echo $$out | grep 'mysqld is alive' || { echo $$out; exit 1; }\n      start_period: 5s\n      interval: 45s\n      timeout: 10s\n      retries: 3\n    networks:\n    - backend",
            "Env": [
                {
                    "name": "DOMAIN_NAME",
                    "value": "<REDACTED>",
                    "$$hashKey": "object:64"
                },
                {
                    "name": "URL_CONFIG",
                    "value": "Host(`${DOMAIN_NAME}`)",
                    "$$hashKey": "object:65"
                },
                {
                    "name": "SHORT_NAME",
                    "value": "hrm",
                    "$$hashKey": "object:66"
                },
                {
                    "name": "ROUTER_NAME",
                    "value": "${SHORT_NAME}",
                    "$$hashKey": "object:67"
                },
                {
                    "name": "SECRETS_FOLDER",
                    "value": "/var/vhost/secrets/${SHORT_NAME}",
                    "$$hashKey": "object:68"
                },
                {
                    "name": "TIME_ZONE",
                    "value": "Europe/London",
                    "$$hashKey": "object:69"
                },
                {
                    "name": "ORANGE_VERSION",
                    "value": "4.10.1",
                    "$$hashKey": "object:70"
                },
                {
                    "name": "DBMS_VERSION",
                    "value": "10.5.12",
                    "$$hashKey": "object:71"
                },
                {
                    "name": "DB_USER",
                    "value": "EL3C2tMAprJI",
                    "$$hashKey": "object:72"
                },
                {
                    "name": "DB_PASS",
                    "value": "<ypmU61hdt(dE.JeW_xYBHOWSBGz3EB6",
                    "$$hashKey": "object:73"
                },
                {
                    "name": "DB_NAME",
                    "value": "Pz6cqWO6H8kj",
                    "$$hashKey": "object:74"
                }
            ],
            "Prune": false,
            "PullImage": false
        },
        "url": "api/stacks/174",
        "params": {
            "endpointId": 9
        },
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json",
            "Authorization": "<REDACTED>"
        }
    },
    "statusText": "",
    "xhrStatus": "complete",
    "resource": {
        "id": 174,
        "StackFileContent": "version: '3.6'\n\nnetworks:\n  public:\n    external:\n      name: public\n#  frontend:\n  backend:\n  \nvolumes:\n  dbs:\n    driver: local\n  app:\n    driver: local\n  \nservices:\n  orange:\n    image: orangehrm/orangehrm:${ORANGE_VERSION}\n    restart: always\n    labels:\n    - traefik.enable=true\n    - traefik.docker.network=public\n    - traefik.http.routers.${ROUTER_NAME}.rule=${URL_CONFIG}\n    - traefik.http.routers.${ROUTER_NAME}.entrypoints=websecure,web\n    - traefik.http.routers.${ROUTER_NAME}.tls=true\n    - traefik.http.routers.${ROUTER_NAME}.tls.certresolver=default\n    - traefik.http.services.${ROUTER_NAME}.loadbalancer.server.port=80\n    environment:\n#    - ORANGEHRM_DATABASE_HOST=dbms\n#    - ORANGEHRM_DATABASE_USER=${DB_USER}\n#    - ORANGEHRM_DATABASE_NAME=${DB_NAME}\n#    - ORANGEHRM_DATABASE_PASSWORD=${DB_PASS}\n    - TIME_ZONE=${TIME_ZONE}\n    networks:\n    - public\n    - backend\n    volumes:\n    - app:/var/www/html\n  \n  dbms:\n    image: linuxserver/mariadb:${DBMS_VERSION}\n    restart: always\n    environment:\n    - MYSQL_DATABASE=${DB_NAME}\n    - MYSQL_USER=${DB_USER}\n    - MYSQL_PASSWORD=${DB_PASS}\n    - DATADIR=/var/lib/mysql\n    - TZ=${TIME_ZONE}\n    volumes:\n#    - /var/vhost/config/mysql/temptables/xlarge.cnf:/etc/my.cnf.d/override_temptables.cnf\n#    - /var/vhost/config/mysql/innodb/buffer_pool_large.cnf:/etc/my.cnf.d/override_buffer_pool.cnf\n    - dbs:/var/lib/mysql\n    healthcheck:\n      test: out=$$(mysqladmin ping -h localhost -P 3306 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD); echo $$out | grep 'mysqld is alive' || { echo $$out; exit 1; }\n      start_period: 5s\n      interval: 45s\n      timeout: 10s\n      retries: 3\n    networks:\n    - backend",
        "Env": [
            {
                "name": "DOMAIN_NAME",
                "value": "<REDACTED>",
                "$$hashKey": "object:64"
            },
            {
                "name": "URL_CONFIG",
                "value": "Host(`${DOMAIN_NAME}`)",
                "$$hashKey": "object:65"
            },
            {
                "name": "SHORT_NAME",
                "value": "hrm",
                "$$hashKey": "object:66"
            },
            {
                "name": "ROUTER_NAME",
                "value": "${SHORT_NAME}",
                "$$hashKey": "object:67"
            },
            {
                "name": "SECRETS_FOLDER",
                "value": "/var/vhost/secrets/${SHORT_NAME}",
                "$$hashKey": "object:68"
            },
            {
                "name": "TIME_ZONE",
                "value": "Europe/London",
                "$$hashKey": "object:69"
            },
            {
                "name": "ORANGE_VERSION",
                "value": "4.10.1",
                "$$hashKey": "object:70"
            },
            {
                "name": "DBMS_VERSION",
                "value": "10.5.12",
                "$$hashKey": "object:71"
            },
            {
                "name": "DB_USER",
                "value": "EL3C2tMAprJI",
                "$$hashKey": "object:72"
            },
            {
                "name": "DB_PASS",
                "value": "<ypmU61hdt(dE.JeW_xYBHOWSBGz3EB6",
                "$$hashKey": "object:73"
            },
            {
                "name": "DB_NAME",
                "value": "Pz6cqWO6H8kj",
                "$$hashKey": "object:74"
            }
        ],
        "Prune": false,
        "PullImage": false
    }
}

I have also tried removing the $ sign from the portainer.db file

kale1d0code commented 1 year ago

made progress there is a .env file as well as a stack.env file within each of the /compose/ replacing the dollar sign in this file has enabled me to stop, start, edit, and remove the stack. I'm not sure if it is necessary to tweak the db file

github-actions[bot] commented 1 year ago

This issue has been marked as stale as it has not had recent activity, it will be closed if no further activity occurs in the next 7 days. If you believe that it has been incorrectly labelled as stale, leave a comment and the label will be removed.

github-actions[bot] commented 1 year ago

Since no further activity has appeared on this issue it will be closed. If you believe that it has been incorrectly closed, leave a comment mentioning portainer/support and one of our staff will then review the issue. Note - If it is an old bug report, make sure that it is reproduceable in the latest version of Portainer as it may have already been fixed.

portainer / portainer

Regression in template variables handling (dollar sign) #7992