Rockons: Invalid environment variable (...)

[daniel-illi]

While developing the metadata json file for a new rockon, I can into this error with one of my defined environment variables:

Houston, we've had a problem.
Invalid environment variabled(POSTGRES_PASSWORD)

            Traceback (most recent call last):
  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run
    client, addr = self.socket.accept()
  File "/usr/lib64/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable

I found the line where this error is generated on line 142 of the file src/rockstor/storageadmin/views/rockon_id.py:

140:                    for e in env_map.keys():
141:                        if (not DContainerEnv.objects.filter(container=co, key=e).exists()):
142:                            e_msg = ('Invalid environment variabled(%s)' % e)
143:                            handle_exception(Exception(e_msg), request)

Initially I thought this happens when I define an environment variable that has no corresponding ENV set in the Dockerfile, but it didn't help when I defined the same variable in the Dockerfile, the same error was thrown.

I ran into the same problem before, as did someone else:

• https://forum.rockstor.com/t/rock-on-development-invalid-environment-variabled-mysql-root-password/2041
• https://forum.rockstor.com/t/trying-to-create-own-rockon/2140

[phillxnet]

@daniel-illi Thanks for reporting this and linking to the other report. Haven't really looked into this but did notice that both of the sighted reports involve the same env var name:

MYSQL_ROOT_PASSWORD

and your report here has an equally repeatable var name "POSTGRES_PASSWORD". I wonder if we are looking at some kind of accidental repeating env name / failure to isolate / or update type think here. The mariadb.json already uses MYSQL_ROOT_PASSWORD but I don't see any existing repo rock-on using "POSTGRES_PASSWORD". Also all sighted examples contain an underscore but I don't see how this is a problem.

Could you check to see if any of your locally defined and installed Rock-ons use the same env name. Or this could be some other bug in the container object key storage.

Also noting we have a typo in that exception message: "variabled".

[daniel-illi]

@phillxnet I'm testing the rockon in a fresh vm without any other custom rockons. No rockons are installed yet.

[Tschaul]

I get the same error with SELF_URL_PATH in the Tiny-Tiny-RSS image from https://github.com/clue/docker-ttrss

Heres my rockon:

    "TinyTinyRSS Test": {
        "container_links": {
            "ttrss-test": [
                {
                    "name": "db",
                    "source_container": "ttrss_postgres-test"
                }
            ]
        },
        "containers": {
            "ttrss-test": {
                "image": "clue/ttrss",
                "tag": "latest",
                "launch_order": 2,
                "ports": {
                    "80": {
                        "description": "Tiny-Tiny-RSS WebUI port. Suggested default: 8888",
                        "host_default": 8286,
                        "label": "WebUI port",
                        "protocol": "tcp",
                        "ui": true
                    }
                },
                "environment":{
                    "SELF_URL_PATH":{
                        "description":"Url under which ttrss will be reachable. Enables browser integration if set.",
                        "label":"Self url path",
                        "index":1
                    }
                }

            },
            "ttrss_postgres-test": {
                "image": "postgres",
                "tag": "latest",
                "launch_order": 1,
                "volumes": {
                    "/var/lib/posgresql/data/pgdata": {
                        "description": "Choose a Share for the PosgreSQL database of Tiny-Tiny-Rss.",
                        "label": "Database Storage",
                        "min_size": 1073741824
                    }
                },
                "opts":[
                    ["-e","PGDATA=/var/lib/posgresql/data/pgdata"],
                    ["-e","POSTGRES_USER=ttrss"]
                ]
            }
        },
        "version": "latest stable",
        "description": "Web-based news feed reader and aggregator",
        "icon": "https://tt-rss.org/gitlab/uploads/project/avatar/1/ic_launcher.png",
        "more_info": "<p>The default user credentials are admin/password. Make sure to change them after your first login.</p>",
        "ui": {
            "https": true,
            "slug": ""
        },
        "website": "https://tt-rss.org/"
    }
}

[daniel-illi]

Another report with the same problem: https://forum.rockstor.com/t/softether-vpn-rockon/2637/3

[alazare619]

+1 Same exact issue i'm facing with softether that was reported on the forums.

[rcastberg]

I have the same issues as mentioned here, but only if i define container_links and a second container, If i remove those the issue disappears. Could this be related to checking the wrong docker image for environmental variables?

[anatox]

A temporary fix is to duplicate environment json in all containers. It seems like rockstor tries to assign all the environment variables defined in rockon to all of its containers and fails there.

[flukejones]

This is still an issue and really needs to be fixed. The linked PR requires work.

[thailgrott]

Noticed that this is still an issue when trying to combine some containers into a single Rockon to improve the user experience of this Rockon.

I wrote the following json definition:

{
    "Nginx Reverse Proxy (jWilder)": {
        "containers": {
            "default-site": {
                "image": "nginx",
                "tag": "latest",
                "launch_order": 1,
                "ports": {},
                "volumes": {
                    "/usr/share/nginx/html": {
                        "description": "Choose a volume for static files.",
                        "label": "Default htdocs"
                    }
                },
                "environment": {
                    "VIRTUAL_HOST": {
                        "description": "Specify the virtual host name to add to the nginx proxy.",
                        "label": "Virtual Host",
                        "index": 1
                    },
                    "NGINX_HOST": {
                        "description": "Specify the virtual host again for the default host.",
                        "label": "Default Host",
                        "index": 2
                    },
                    "LETSENCRYPT_HOST": {
                        "description": "Specify the virtual host name for letsencrypt - likely the same as the virtual host.",
                        "label": "Letsencrypt host",
                        "index": 3
                    },
                    "LETSENCRYPT_EMAIL": {
                        "description": "Specify an email for letsencrypt.",
                        "label": "Letsencrypt email",
                        "index": 4
                    },
                    "LETSENCRYPT_TEST": {
                        "description": "Specify using letsencrypt test or production certificate generation.",
                        "label": "Letsencrypt test(true/false)",
                        "index": 5
                    }
                },
                "opts": [
                    ["--net", "Internet_Proxy"],
                    ["-h", "default-site"]
                ]
            },
            "letsencrypt-nginx-proxy-companion": {
                "image": "jrcs/letsencrypt-nginx-proxy-companion",
                "tag": "latest",
                "launch_order": 2,
                "ports": {},
                "volumes": {
                    "/etc/nginx/certs": {
                        "description": "Choose a volume for generated certificates and keys.",
                        "label": "Certs n Keys"
                    },
                    "/etc/nginx/vhost.d": {
                        "description": "Choose a volume for the vhost.d config directory. ",
                        "label": "Virthosts configs"
                    },
                    "/usr/share/nginx/html": {
                        "description": "Choose a volume to write the challenge files. ",
                        "label": "lets encrypt challenge"
                    }
                },
                "opts": [
                    ["--net", "Internet_Proxy"],
                    ["-v", "/var/run/docker.sock:/var/run/docker.sock:ro"],
                    ["-h", "letsencrypt-nginx-proxy-companion"]
                ]
            },
            "nginx-reverse-proxy": {
                "image": "jwilder/nginx-proxy",
                "tag": "latest",
                "launch_order": 3,
                "ports": {
                    "80": {
                        "description": "The standard HTTP port the proxy server will listen to for requests.",
                        "host_default": 8082,
                        "label": "HTTP Port",
                        "protocol": "tcp",
                        "ui": false
                    },
                    "443": {
                        "description": "The standard HTTP port the proxy server will listen to for requests.",
                        "host_default": 8083,
                        "label": "HTTPS Port",
                        "protocol": "tcp",
                        "ui": false
                    }
                },
                "volumes": {
                    "/etc/nginx/htpasswd": {
                        "description": "Choose a volume for basic authentication files.",
                        "label": "Basic Auth"
                    },
                    "/etc/nginx/certs": {
                        "description": "Choose a volume for storing certificates and keys.",
                        "label": "Certs n Keys"
                    },
                    "/etc/nginx/vhost.d": {
                        "description": "Choose a volume for the vhost.d config directory. ",
                        "label": "Virthosts configs"
                    },
                    "/usr/share/nginx/html": {
                        "description": "Choose a volume to write the challenge files. ",
                        "label": "lets encrypt challenge"
                    }
                },
                "environment": {
                    "DEFAULT_HOST": {
                        "description": "The FQDN of the default website for proxied requests.",
                        "label": "Default host",
                        "index": 1
                    }
                },
                "opts": [
                    ["--net", "Internet_Proxy"],
                    ["-v", "/var/run/docker.sock:/tmp/docker.sock:ro"],
                    ["-h", "jw-nginx-reverse-proxy"],
                    ["--label", "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy"]
                ]
            } 
        },
        "description": "jWilder nginx reverse proxy and letsencrypt companion",
        "ui": {
            "slug": ""
        },
        "volume_add_support": true,
        "website": "https://hub.docker.com/_/hello-world/",
        "version": "1.0"
    }
}

Got the following message during the attempt to install:

Houston,` we've had a problem.

Invalid environment variable (LETSENCRYPT_TEST).

        Traceback (most recent call last):

File "/opt/rockstor/eggs/gunicorn-19.7.1-py2.7.egg/gunicorn/workers/sync.py", line 68, in run_for_one self.accept(listener) File "/opt/rockstor/eggs/gunicorn-19.7.1-py2.7.egg/gunicorn/workers/sync.py", line 27, in accept client, addr = listener.accept() File "/usr/lib64/python2.7/socket.py", line 202, in accept sock, addr = self._sock.accept() error: [Errno 11] Resource temporarily unavailable

[FroggyFlox]

Hi again @KookyKrane,

I'm in the same situation as you, and actually it is when I tried to make a rock-on for the same project as you (Nginx Reverse Proxy (jWilder)) and this very issue here that I decided to get myself familiarized with the rock-on system and working on improving them. As described in the other issue you just commented on, I'm currently working on implementing docker networks (which will indirectly help with the current issue), and plan on having a look at the proposed PR (#1688) thereafter.

@anatox did a lot of work in this PR to fix this issue and there was a helpful discussion therein with @MFlyer so even though that part of the code changed quite a bit, building upon / adapting @anatox's work should be a very good start (to say the least).

[thailgrott]

Thx @FroggyFlox for pointing me to #1688 These patches are certainly motivating me to become more familiar with the system.

[FroggyFlox]

A new forum user has encountered the same issue: https://forum.rockstor.com/t/rock-on-install-fail-invalid-environment-variable-puid/7170

In addition to reinforcing the need for this issue to be fixed, let's try to update the forum thread above once this issue is closed.

[phillxnet]

Re: @anatox 's https://github.com/rockstor/rockstor-core/issues/1588#issuecomment-291024816

A temporary fix is to duplicate environment json in all containers. It seems like rockstor tries to assign all the environment variables defined in rockon to all of its containers and fails there.

Linking for context to @freaktechnik 's now closed-as-later-duplicate issue noted the same work-around https://github.com/rockstor/rockstor-core/issues/2202#issuecomment-674398879

I have found a workaround, which is to have the same set of env vars for all containers. Curiously this repeats the fields in the env var input, but will only show them once in the end table. Of course this only works if there are no env var clashes.

[phillxnet]

Title updated to reflect latest testing code behaviour - where we also no-longer have as expressive a trace-back; i.e.:

Invalid environment variable (DB_ADMIN_PASSWORD). NoneType: None

in the case of the linked draft PR in rockon-registry (https://github.com/rockstor/rockon-registry/pull/379), the most recent reproducer expressing this now long-standing limitation regarding multi-container Rock-on definitions that also involve environment elements.

I have assigned myself to this issue informed by @anatox prior work #1688 in the context of many underlying changes in the interim to our Rock-ons code - represented in current testing branch.

[phillxnet]

As-is, in current testing we pass the following from front-end to backend during a rock-on install command request POST (using the reference draft PR as reproducer):

[12/Aug/2024 14:07:06] DEBUG [storageadmin.views.rockon_id:109] install request with share_map={'bareos-backups': '/var/lib/bareos/storage', 'bareos-catalog-backup': '/var/lib/bareos', 'bareos-catalog': '/var/lib/postgresql/data', 'bareos-dir-config': '/etc/bareos', 'bareos-storage-config': '/etc/bareos', 'bareos-webui-config': '/etc/bareos-webui'}, port_map={'9100': 9100, '9101': 9101, '9103': 9103}, dev_map={}, cc_map={}, env_map={'POSTGRES_PASSWORD': 'ptt2', 'DB_ADMIN_PASSWORD': 'ptt2', 'DB_PASSWORD': 'ctt2', 'BAREOS_WEBUI_PASSWORD': 'wtt2'}
[12/Aug/2024 14:07:06] ERROR [storageadmin.util:45] Exception: Invalid environment variable (DB_ADMIN_PASSWORD).
NoneType: None

• share_map={'bareos-backups': '/var/lib/bareos/storage', 'bareos-catalog-backup': '/var/lib/bareos', 'bareos-catalog': '/var/lib/postgresql/data', 'bareos-dir-config': '/etc/bareos', 'bareos-storage-config': '/etc/bareos', 'bareos-webui-config': '/etc/bareos-webui'}
• port_map={'9100': 9100, '9101': 9101, '9103': 9103}
• dev_map={}
• cc_map={}
• env_map={'POSTGRES_PASSWORD': 'ptt2', 'DB_ADMIN_PASSWORD': 'ptt2', 'DB_PASSWORD': 'ctt2', 'BAREOS_WEBUI_PASSWORD': 'wtt2'}

I.e. as per @anatox exposition (many years ago now) we lump all config info, for all containers, together. defeating env name duplication across containers in multi-container Rock-ons: even though it is independently detailed in the source JSON that is the Rock-on definition. This applies to all env categories here also, as per @freaktechnik 's following https://github.com/rockstor/rockstor-core/issues/2202#issuecomment-972021752 :

This also appears to apply to devices.

Which limits multi-container Rock-ons to per Rock-on unique share/port/cc/env definitions.

[phillxnet]

Cherry picking front-end changes proposed by @anatox to add container info to the 'environment' input matrix, we get the following input from our Rock-on install front-end:

env_map={'85': {'POSTGRES_PASSWORD': 'ptt2'}, '86': {'DB_ADMIN_PASSWORD': 'ptt2', 'DB_PASSWORD': 'ctt2', 'BAREOS_WEBUI_PASSWORD': 'wtt2'}}

I.e. we now have a second matrix dimension associated with container id within our DB, preserving per-container env seperation.

• '85': {'POSTGRES_PASSWORD': 'ptt2'}
• '86': {'DB_ADMIN_PASSWORD': 'ptt2', 'DB_PASSWORD': 'ctt2', 'BAREOS_WEBUI_PASSWORD': 'wtt2'}

[phillxnet]

The addition, in the front-end, of our new environment matrix dimension has the following buggy artifact:

I.e. we are summarising the 'new' top level environment matrix dimension.

[EDIT]: Having completed the reproducer Rock-on install (BareOS-server-ser) we have confirmation of the cosmetic (front-end only) nature of the above: i.e. in the following (via spanner of installed draft PR reproducer Rock-on) we see all expected environment variables presented as entered:

[phillxnet]

Remaining sub-issues:

• [x] Fix Rock-on pre-install wizard env summary re now listing containers dimension, not the Env's we entered.
• [x] Confirm potential parallel issue re Share mapping requiring the same `container' awareness treatment: i.e. note the double entry in the above installed Rock-on re bareos-storage-config. [EDIT see:https://github.com/rockstor/rockstor-core/issues/1588#issuecomment-2284333987 see also @FroggyFlox https://github.com/rockstor/rockstor-core/issues/1588#issuecomment-2284283147]

[12/Aug/2024 15:57:57] DEBUG [storageadmin.views.rockon_id:109] install request with share_map={'bareos-backups': '/var/lib/bareos/storage', 'bareos-catalog-backup': '/var/lib/bareos', 'bareos-catalog': '/var/lib/postgresql/data', 'bareos-dir-config': '/etc/bareos', 'ba
reos-storage-config': '/etc/bareos', 'bareos-webui-config': '/etc/bareos-webui'}, port_map={'9100': 9100, '9101': 9101, '9103': 9103}, dev_map={}, cc_map={}, env_map={'85': {'POSTGRES_PASSWORD': 'ptt2'}, '86': {'DB_ADMIN_PASSWORD': 'ptt2', 'DB_PASSWORD': 'ctt2', 'BAREOS
_WEBUI_PASSWORD': 'wtt2'}}

[FroggyFlox]

@phillxnet, In case it helps, your second point reminds me of https://github.com/rockstor/rockstor-core/issues/2000

[phillxnet]

@FroggyFlox Thanks, that helps. We may have to tackle that separately, in a dedicated PR: still assessing how far to go with this issue that is predominantly related to env per container in multi-container Rock-ons after-all.

Confirming share entry as:

Resource type	Name	Mapped representation
Share	bareos-backups	/var/lib/bareos/storage
Share	bareos-catalog-backup	/var/lib/bareos
Share	bareos-catalog	/var/lib/postgresql/data
Share	bareos-dir-config	/etc/bareos
Share	bareos-storage-config	/etc/bareos
Share	bareos-webui-config	/etc/bareos-webui

With corresponding front-end passing to back-end the following (re shares):

[12/Aug/2024 16:40:55] DEBUG [storageadmin.views.rockon_id:109] install request with share_map={'bareos-backups': '/var/lib/bareos/storage', 'bareos-catalog-backup': '/var/lib/bareos', 'bareos-catalog': '/var/lib/postgresql/data', 'bareos-dir-config': '/etc/bareos', 'bareos-storage-config': '/etc/bareos', 'bareos-webui-config': '/etc/bareos-webui'}

And the resulting share mapping as reported by our 'spanner' on the resulting installed Rock-on:

Resource type	Name	Mapped representation
Share	bareos-backups	/var/lib/bareos/storage
Share	bareos-catalog-backup	/var/lib/bareos
Share	bareos-catalog	/var/lib/postgresql/data
Share	bareos-storage-config	/etc/bareos
Share	bareos-storage-config	/etc/bareos
Share	bareos-webui-config	/etc/bareos-webui

[EDIT See spin-off issue "Rock-on install wizard obfuscates share container info": #2886 ]

[phillxnet]

@FroggyFlox Currently looking at where/how to address:

• [x] Fix Rock-on pre-install wizard env summary re now listing containers dimension, not the Env's we entered.

[EDIT re the JS for our pre-install summary table:

https://github.com/rockstor/rockstor-core/blob/eab4ed520951577c5aacd616e53b26a3dc686efa/src/rockstor/storageadmin/static/storageadmin/js/views/rockons.js#L821

https://github.com/rockstor/rockstor-core/blob/eab4ed520951577c5aacd616e53b26a3dc686efa/src/rockstor/storageadmin/static/storageadmin/js/views/rockons.js#L835-L845

https://github.com/rockstor/rockstor-core/blob/eab4ed520951577c5aacd616e53b26a3dc686efa/src/rockstor/storageadmin/static/storageadmin/js/templates/rockons/install_summary.jst#L1-L10

]

[FroggyFlox]

Thanks a lot for taking care of all that. Making these tables aware of the containers may be more work than the scope of this issue as I believe nothing besides maybe the display of the network customization options are container aware.

Maybe we can add the container in parenthesis to the value in the table for this issue (or something along those lines)?

[phillxnet]

@FroggyFlox Agreed. Re:

Maybe we can add the container in parenthesis to the value in the table for this issue (or something along those lines)?

I'm actually working on this approach currently as it goes. We may well have to improve the tables in time thought.

[phillxnet]

Existing multi-container Rock-ons in the rockon-registry repo:

• Seafile v7+: https://github.com/rockstor/rockon-registry/blob/master/seafile7%2B.json - 3 containers: all with no environment definitions.
• Zabbix-XXL https://github.com/rockstor/rockon-registry/blob/master/zabbix-xxl.json - 3 containers: 2 with empty environment definitions. The remaining container has no environment section defined.

Proposed multi-container Rock-on:

• "bare os server set ...": https://github.com/rockstor/rockon-registry/pull/379 - 4 containers (currently): 2 with non-empty environment definitions.

[phillxnet]

Closing as: Fixed by #2887

Thanks again to @anatox for the original push (and PR) to approach this issue.

[freaktechnik]

I'm assuming I'll have to wait for this to be able to use a rockon with multiple containers (like this) that share shares? This used to work before #2064.

[phillxnet]

@freaktechnik Hello again. This issue was closed by the indicated PR. And an rpm containing this improvement was released via the testing channel under https://github.com/rockstor/rockstor-core/releases/tag/5.0.14-0 there was also another multi-container improvement added.

Form announcement of 5.0.14-0: https://forum.rockstor.com/t/v5-0-testing-channel-changelog/8898/27

Hope that helps. Not sure if it resolves your issue however. You may be able to share Shares between containers via a hard wired --volumes-from env var. Assuming you instruct the user with a must be on the owning container share name creation.