MongoDB authentication support

[stefano-cherchi]

I'm running a ClearML Server deployed on Kubernetes using the official Helm chart and everything's fine.

Now I'd like to move the document storage from the locally deployed mongo-service to an external MongoDB server or an equivalent managed service such as AWS DocumentDB or GCP Atlas but I cannot find any documentation about how to set MongoDB authentication in ClearML.

Is this scenario supported?

[jkhenning]

Hi @stefano-cherchi ,

When setting the new hostname for your external mongodb, you should simply provide a URL with a user/password, so something like mongodb://<user>:<password>@<hostname>:27017/backend and mongodb://<user>:<password>@<hostname>:27017/auth

[stefano-cherchi]

Hi @stefano-cherchi ,

When setting the new hostname for your external mongodb, you should simply provide a URL with a user/password, so something like mongodb://<user>:<password>@<hostname>:27017/backend and mongodb://<user>:<password>@<hostname>:27017/auth

Hi Jake,

unfortunately this isn't working: I have replaced mongoServiceName: mongo-service in the values.yaml file with the connection string provided by DocumentDB and this is what I get

[2021-09-17 16:15:44,164] [8] [INFO] [clearml.database] Using override mongodb host MYUSER:MYPASSWORD@REDACTEDHOST.docdb.amazonaws.com:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false
[2021-09-17 16:15:44,165] [8] [INFO] [clearml.database] Using override mongodb port 27017
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/clearml/apiserver/server.py", line 10, in <module>
    AppSequence(app).start(request_handlers=RequestHandlers())
  File "/opt/clearml/apiserver/server_init/app_sequence.py", line 40, in start
    self._init_dbs()
  File "/opt/clearml/apiserver/server_init/app_sequence.py", line 68, in _init_dbs
    db.initialize()
  File "/opt/clearml/apiserver/database/__init__.py", line 66, in initialize
    entry.host = furl(entry.host).set(host=override_hostname).url
  File "/usr/local/lib/python3.6/site-packages/furl/furl.py", line 1729, in set
    self.host = host
  File "/usr/local/lib/python3.6/site-packages/furl/furl.py", line 1889, in __setattr__
    object.__setattr__(self, attr, value)
  File "/usr/local/lib/python3.6/site-packages/furl/furl.py", line 1450, in host
    raise ValueError(errmsg % (host, INVALID_HOST_CHARS))
ValueError: Invalid host 'MYUSER:MYPASSWORD@REDACTEDHOST.docdb.amazonaws.com:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false'. Host strings must have at least one non-period character, can't contain any of '!@#$%^&'"*()+=:;/', and can't have adjacent periods.

Tried also prepending mongodb://, same error.

Apparently it expects a Kube service name or a "clean" FQDN and nothing else.

Any idea?

[jkhenning]

Hi @stefano-cherchi ,

Yeah, you're right - this seems to be missing in the server - as you've suggested, the override now can only be a proper host name, without user/password. This means you'll need to override this using the hosts.conf server configuration file, and not using the CLEARML_MONGODB_SERVICE_HOST used by the setting you've used. To forego using the config file (which might not be so easy when using helm/k8s, you can simply set the following dynamic environment variables to something like CLEARML__HOSTS__MONGO__BACKEND__HOST=mongodb://<user>:<password>@mongo-service:27017/backend and CLEARML__HOSTS__MONGO__AUTH__HOST=mongodb://<user>:<password>@mongo-service:27017/auth

[jkhenning]

BTW - I would encourage you to switch to our latest helm chart version (see https://github.com/allegroai/clearml-helm-charts)

[Mokto]

Thanks @jkhenning but I have a similar issue when I use your way ;)

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/clearml/apiserver/server.py", line 5, in <module>
    from apiserver.config_repo import config
  File "/opt/clearml/apiserver/config_repo.py", line 3, in <module>
    config = BasicConfig()
  File "/opt/clearml/apiserver/config/basic.py", line 60, in __init__
    self._config = self._reload()
  File "/opt/clearml/apiserver/config/basic.py", line 124, in _reload
    extra_config_values = self._read_extra_env_config_values()
  File "/opt/clearml/apiserver/config/basic.py", line 98, in _read_extra_env_config_values
    result, ConfigFactory.parse_string(f"{path}: {os.environ[key]}")
  File "/usr/local/lib/python3.6/site-packages/pyhocon/config_parser.py", line 201, in parse_string
    return ConfigParser().parse(content, basedir, resolve, unresolved_value)
  File "/usr/local/lib/python3.6/site-packages/pyhocon/config_parser.py", line 516, in parse
    config = config_expr.parseString(content, parseAll=True)[0]
  File "/usr/local/lib/python3.6/site-packages/pyparsing.py", line 1955, in parseString
    raise exc
  File "/usr/local/lib/python3.6/site-packages/pyparsing.py", line 3814, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected end of text, found '@'  (at char 59), (line:1, col:60

[jkhenning]

@Mokto this means you have an invalid configuration file - can you share it?

[Mokto]

Thanks for you help!

I'm using the default image from the official new helm chart. I just added those two environment variables to deployment-apiserver.yaml

 - name: CLEARML__HOSTS__MONGO__BACKEND__HOST
   value: "mongodb://clearml:XXXXXXX@yyyy.eeee.mongodb.net/backend?retryWrites=true&w=majority"
- name: CLEARML__HOSTS__MONGO__AUTH__HOST
   value: "mongodb://clearml:XXXX@yyyy.eeee.mongodb.net/auth?retryWrites=true&w=majority"

[jkhenning]

Thanks for you help!

Sure, NP 🙂

I'm using the default image from the official new helm chart. I just added those two environment variables to deployment-apiserver.yaml

Since the server takes the env var values and parses them using a HOCON string parser, I think the way to solve this is to make sure that once applied, the string will be wrapped with double quotes ("), otherwise HOCON will not consider '@' part of the string. Try this:

 - name: CLEARML__HOSTS__MONGO__BACKEND__HOST
   value: "\"mongodb://clearml:XXXXXXX@yyyy.eeee.mongodb.net/backend?retryWrites=true&w=majority\""
- name: CLEARML__HOSTS__MONGO__AUTH__HOST
   value: "\"mongodb://clearml:XXXX@yyyy.eeee.mongodb.net/auth?retryWrites=true&w=majority\""

[Mokto]

I now have an authentication error (Authentication failed) but I think everything else is solved! Thanks ;)

[Mokto]

Not sure if I should open a new issue as this is related.

Your solution theoretically worked but ClearML server doesn't seem to support SRV endpoints.

At first it complained that dnspython needed to be installed, so I added it to the dockerfile and deployed it. Now it complains about an invalid URL.

[2021-11-24 07:57:50,183] [9] [INFO] [clearml.es_factory] Using override elastic host clearml-elastic-master
[2021-11-24 07:57:50,183] [9] [INFO] [clearml.es_factory] Using override elastic port 9200
[2021-11-24 07:57:50,469] [9] [INFO] [clearml.redis_manager] Using override redis host clearml-redis-master
[2021-11-24 07:57:50,470] [9] [INFO] [clearml.redis_manager] Using override redis port 6379
[2021-11-24 07:57:50,582] [9] [WARNING] [clearml.schema_reader] failed loading cache: [Errno 2] No such file or directory: '/opt/clearml/apiserver/schema/services/_cache.json'
[2021-11-24 07:57:50,582] [9] [INFO] [clearml.schema_reader] regenerating schema cache
[2021-11-24 07:57:54,710] [9] [INFO] [clearml.app_sequence] ################ API Server initializing #####################
[2021-11-24 07:57:54,710] [9] [INFO] [clearml.database] Initializing database connections
[2021-11-24 07:57:54,711] [9] [INFO] [clearml.database] Using override mongodb host 126.126.126.126
[2021-11-24 07:57:54,711] [9] [INFO] [clearml.database] Using override mongodb port 27017
[2021-11-24 07:57:54,713] [9] [INFO] [clearml.database] Registering connection to auth-db (mongodb+srv://clearml:mypassword@126.126.126.126:27017/admin?retryWrites=true&w=majority)
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/clearml/apiserver/server.py", line 10, in <module>
    AppSequence(app).start(request_handlers=RequestHandlers())
  File "/opt/clearml/apiserver/server_init/app_sequence.py", line 40, in start
    self._init_dbs()
  File "/opt/clearml/apiserver/server_init/app_sequence.py", line 68, in _init_dbs
    db.initialize()
  File "/opt/clearml/apiserver/database/__init__.py", line 76, in initialize
    register_connection(**entry.to_struct())
  File "/usr/local/lib/python3.6/site-packages/mongoengine/connection.py", line 218, in register_connection
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/mongoengine/connection.py", line 118, in _get_connection_settings
    uri_dict = uri_parser.parse_uri(entity)
  File "/usr/local/lib64/python3.6/site-packages/pymongo/uri_parser.py", line 457, in parse_uri
    "%s URIs must not include a port number" % (SRV_SCHEME,))
pymongo.errors.InvalidURI: mongodb+srv:// URIs must not include a port number
Loading config from /opt/clearml/apiserver/config/default
Loading config from file /opt/clearml/apiserver/config/default/apiserver.conf
Loading config from file /opt/clearml/apiserver/config/default/hosts.conf
Loading config from file /opt/clearml/apiserver/config/default/logging.conf
Loading config from file /opt/clearml/apiserver/config/default/secure.conf
Loading config from file /opt/clearml/apiserver/config/default/services/organization.conf
Loading config from file /opt/clearml/apiserver/config/default/services/events.conf
Loading config from file /opt/clearml/apiserver/config/default/services/projects.conf
Loading config from file /opt/clearml/apiserver/config/default/services/auth.conf
Loading config from file /opt/clearml/apiserver/config/default/services/tasks.conf
Loading config from /opt/clearml/config

Please note that I'm not using any port in the config, just the two environment variables specified earlier.

(I also obfuscated the IP address resolved by ClearML)

[jkhenning]

Well, we use the standard pymongo driver, and it seems that in order to support SRV endpoints the dependency should change to pymongo[srv]==3.12. We'll fix this for the next release (where we'll probably also upgrade to MongoDB 4.4). Meanwhile, a possible fix is to try and set the command for the apiserver container to python3 -m pip install pymongo[srv]==3.12 && wrapper.sh apiserver and see if it helps 🙂

[Mokto]

I made it work! But changing the pymongo version was not enough.

I also commented those 2 lines : https://github.com/allegroai/clearml-server/blob/master/apiserver/database/__init__.py#L68-L69

And hardcoded the host here : https://github.com/allegroai/clearml-server/blob/master/apiserver/database/__init__.py#L50

Because ports & ip addresses are not supported with SRV.

[jkhenning]

Cool! If there's a way to do that in a configurable way in the server's code, we'll appreciate a PR 🙂

[Mokto]

Yes, sure! I'm still unsure when the domain is transformed into an IP though.

& I have a similar issue with Elasticsearch (impossible to use HTTPS) so I'm still working on it.

[Mokto]

Maybe you can point me in the right direction? (where is the domain resolved to an IP?) I'm not super experienced with Python.

[jkhenning]

Hi @Mokto,

Maybe you can point me in the right direction? (where is the domain resolved to an IP?)

I'm not sure I understand what you're referring to - are you taking about setting https access to ES?

[Mokto]

When using an hostname with CLEARMLHOSTSMONGOBACKENDHOST, https://github.com/allegroai/clearml-server/blob/master/apiserver/database/__init__.py#L50

The value here is not using an host but is translating that host into an IP address.

And I can't figure out where.

[jkhenning]

Well, if you're using a host name in the docker network, its resolved by the docker network. Otherwise, by some DNS - but that's not really interesting, it's a task for the network layer - why would you care about the translation? The only thing that matters is that you can use a host name or an IP address interchangeably...

[Mokto]

I care because at that exact moment override_hostname is containing and the mongo library is complaining that I have to use host and not IP addresses with SRV.

Which makes a lot of sense.

[jkhenning]

Well, if that's what you're asking, than the hostname/ip exactly as you provided (and not translated in any way) is passed to the driver code.

Did you make sure the mongodb[srv] package is installed when starting the container?

[Mokto]

I have no idea why it used to be transformed from dns to ip, maybe because I used to install dnspython & not pymongo[srv]

https://github.com/allegroai/clearml-server/pull/96

[jkhenning]

Merged, thanks!

[nielstenboom]

To everybody reading this thread trying to get the connection to AWS DocumentDB to work, it's not compatible.

https://github.com/allegroai/clearml-server/issues/101