Error: could not verify credentials with rerouted ports

[LouisSnz]

Hi everyone,

I am trying to set up ClearML on a new server with the 8080 port already used for an other system. I am rerouting the web_server on port 9080.

However, when I execute the clearml-init command, I get this result:

❯ clearml-init
ClearML SDK setup process

Please create new clearml credentials through the settings page in your `clearml-server` web app (e.g. http://localhost:8080//settings/workspace-configuration) 
Or create a free account at https://app.clear.ml/settings/workspace-configuration

In settings page, press "Create new credentials", then press "Copy to clipboard".

Paste copied configuration here:
api { 
    web_server: http://ip_adress:9080
    api_server: http://ip_adress:8008
    credentials {
        "access_key" = "WKMAKEWGGNAZJYB4BJE6"
        "secret_key"  = "xxxxxxxxxxxxxxxxxxxxxxxxxx"
    }
}
Detected credentials key="WKMAKEWGGNAZJYB4BJE6" secret="tZxx***"

ClearML Hosts configuration:
Web App: http://10.0.0.100:9080
API: http://10.0.0.100:9080
File Store: 

Verifying credentials ...
Error: could not verify credentials: key=WKMAKEWGGNAZJYB4BJE6 secret=xxxxxxxxxxxxxxxxxxxxxxxxxx
Enter user access key: WKMAKEWGGNAZJYB4BJE6
Enter user secret: xxxxxxxxxxxxxxxxxxxxxxxxxx
Verifying credentials ...
Error: could not verify credentials: key=WKMAKEWGGNAZJYB4BJE6 secret=xxxxxxxxxxxxxxxxxxxxxxxxxx
Exiting setup without creating configuration file

I don't understand why the ClearML Hosts configuration: displayed after pasting my configuration is wrong.

For rerouting my ports, I changed my docker-compose.yaml like after. Apparently, it seems to work. I am able to access my server http://ip_adress:9080 and everything seems fine.

docker-compose.yaml

```yml version: "3.6" services: apiserver: command: - apiserver container_name: clearml-apiserver image: allegroai/clearml:latest restart: unless-stopped volumes: - /opt/clearml/logs:/var/log/clearml - /opt/clearml/config:/opt/clearml/config - /opt/clearml/data/fileserver:/mnt/fileserver depends_on: - redis - mongo - elasticsearch - fileserver environment: CLEARML_ELASTIC_SERVICE_HOST: elasticsearch CLEARML_ELASTIC_SERVICE_PORT: 9200 CLEARML_ELASTIC_SERVICE_PASSWORD: ${ELASTIC_PASSWORD} CLEARML_MONGODB_SERVICE_HOST: mongo CLEARML_MONGODB_SERVICE_PORT: 27017 CLEARML_REDIS_SERVICE_HOST: redis CLEARML_REDIS_SERVICE_PORT: 6379 CLEARML_SERVER_DEPLOYMENT_TYPE: ${CLEARML_SERVER_DEPLOYMENT_TYPE:-linux} CLEARML__apiserver__pre_populate__enabled: "true" CLEARML__apiserver__pre_populate__zip_files: "/opt/clearml/db-pre-populate" CLEARML__apiserver__pre_populate__artifacts_path: "/mnt/fileserver" CLEARML__services__async_urls_delete__enabled: "true" ports: - "8008:8008" networks: - backend - frontend elasticsearch: networks: - backend container_name: clearml-elastic environment: ES_JAVA_OPTS: -Xms2g -Xmx2g -Dlog4j2.formatMsgNoLookups=true ELASTIC_PASSWORD: ${ELASTIC_PASSWORD} bootstrap.memory_lock: "true" cluster.name: clearml cluster.routing.allocation.node_initial_primaries_recoveries: "500" cluster.routing.allocation.disk.watermark.low: 500mb cluster.routing.allocation.disk.watermark.high: 500mb cluster.routing.allocation.disk.watermark.flood_stage: 500mb discovery.zen.minimum_master_nodes: "1" discovery.type: "single-node" http.compression_level: "7" node.ingest: "true" node.name: clearml reindex.remote.whitelist: '*.*' xpack.monitoring.enabled: "false" xpack.security.enabled: "false" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2 restart: unless-stopped volumes: - /opt/clearml/data/elastic_7:/usr/share/elasticsearch/data - /usr/share/elasticsearch/logs fileserver: networks: - backend - frontend command: - fileserver container_name: clearml-fileserver image: allegroai/clearml:latest environment: CLEARML__fileserver__delete__allow_batch: "true" restart: unless-stopped volumes: - /opt/clearml/logs:/var/log/clearml - /opt/clearml/data/fileserver:/mnt/fileserver - /opt/clearml/config:/opt/clearml/config ports: - "8081:8081" mongo: networks: - backend container_name: clearml-mongo image: mongo:4.4.9 restart: unless-stopped command: --setParameter internalQueryMaxBlockingSortMemoryUsageBytes=196100200 volumes: - /opt/clearml/data/mongo_4/db:/data/db - /opt/clearml/data/mongo_4/configdb:/data/configdb redis: networks: - backend container_name: clearml-redis image: redis:5.0 restart: unless-stopped volumes: - /opt/clearml/data/redis:/data webserver: command: - webserver container_name: clearml-webserver # environment: # CLEARML_SERVER_SUB_PATH : clearml-web # Allow Clearml to be served with a URL path prefix. image: allegroai/clearml:latest restart: unless-stopped depends_on: - apiserver ports: - "9080:80" networks: - backend - frontend async_delete: depends_on: - apiserver - redis - mongo - elasticsearch - fileserver container_name: async_delete image: allegroai/clearml:latest networks: - backend restart: unless-stopped environment: CLEARML_ELASTIC_SERVICE_HOST: elasticsearch CLEARML_ELASTIC_SERVICE_PORT: 9200 CLEARML_ELASTIC_SERVICE_PASSWORD: ${ELASTIC_PASSWORD} CLEARML_MONGODB_SERVICE_HOST: mongo CLEARML_MONGODB_SERVICE_PORT: 27017 CLEARML_REDIS_SERVICE_HOST: redis CLEARML_REDIS_SERVICE_PORT: 6379 PYTHONPATH: /opt/clearml/apiserver CLEARML__services__async_urls_delete__fileserver__url_prefixes: "[${CLEARML_FILES_HOST:-}]" entrypoint: - python3 - -m - jobs.async_urls_delete - --fileserver-host - http://fileserver:8081 volumes: - /opt/clearml/logs:/var/log/clearml agent-services: networks: - backend container_name: clearml-agent-services image: allegroai/clearml-agent-services:latest deploy: restart_policy: condition: on-failure privileged: true environment: CLEARML_HOST_IP: ${CLEARML_HOST_IP} CLEARML_WEB_HOST: ${CLEARML_WEB_HOST:-} CLEARML_API_HOST: http://apiserver:8008 CLEARML_FILES_HOST: ${CLEARML_FILES_HOST:-} CLEARML_API_ACCESS_KEY: ${CLEARML_API_ACCESS_KEY:-} CLEARML_API_SECRET_KEY: ${CLEARML_API_SECRET_KEY:-} CLEARML_AGENT_GIT_USER: ${CLEARML_AGENT_GIT_USER} CLEARML_AGENT_GIT_PASS: ${CLEARML_AGENT_GIT_PASS} CLEARML_AGENT_UPDATE_VERSION: ${CLEARML_AGENT_UPDATE_VERSION:->=0.17.0} CLEARML_AGENT_DEFAULT_BASE_DOCKER: "ubuntu:18.04" AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-} AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-} AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-} AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT:-} AZURE_STORAGE_KEY: ${AZURE_STORAGE_KEY:-} GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS:-} CLEARML_WORKER_ID: "clearml-services" CLEARML_AGENT_DOCKER_HOST_MOUNT: "/opt/clearml/agent:/root/.clearml" SHUTDOWN_IF_NO_ACCESS_KEY: 1 volumes: - /var/run/docker.sock:/var/run/docker.sock - /opt/clearml/agent:/root/.clearml depends_on: - apiserver entrypoint: > bash -c "curl --retry 10 --retry-delay 10 --retry-connrefused 'http://apiserver:8008/debug.ping' && /usr/agent/entrypoint.sh" networks: backend: driver: bridge frontend: driver: bridge ```

Any idea on how to solve that ? Thanks.

[Least1924]

same problem,someway to solve?

[naba89]

Same thing was happening to me, I just created the ~/clearml.conf file manually and pasted the copied configuration and it worked for me, was able to create tasks in code and check them on the web-ui properly. Not sure why clearml-init cannot seem to connect though 🤔

[NathanClearML]

@LouisSnz , what version of ClearML SDK and server are you using? What about @naba89's suggestion, does that work?

[korotaS]

I faced the same issue and found the problem with custom port. When you copy credentials from UI (create credentials dialogue window), the web server api is copied from your address (window.location.origin) with custom port:

https://github.com/allegroai/clearml-web/blob/2b6aa6043c3f36e3349c6fe7235b77a3fddd0a09/src/app/webapp-common/settings/admin/admin-dialog-template/admin-dialog-template.component.ts#L19

But when you run clearml-init and paste your configuration from UI, the parser gets the custom port (not 8080 and not 8008) and pastes it to api server (note lines 223-224).

https://github.com/allegroai/clearml/blob/b3e330ea9b00e081988c2effe3a3c62e3824e0a3/clearml/cli/config/__main__.py#L199-L224

So you get this:

ClearML Hosts configuration:
Web App: host:custom_port
API: host:custom_port  # (not host:8008, which is correct)

And then local clearml can't reach the api server to verify your credentials.

So there are 2 solutions:

1. Edit the thing which you copy from the UI and change custom port to 8080, then paste to clearml-init
2. Or @naba89 's solution - just create ~/clearml.conf manually and edit the ports to be correct.

[LouisSnz]

I managed to solve this problem by creating my ~/clearml.conf file on my own (same solution as @naba89 suggested) and it worked fine.

For those wondering, here is what my ~/clearml.conf file looks like :

api {
    # API server on port 8008
    api_server: "http://ip_address:8008"

    # web_server on port 9080
    web_server: "http://ip_address:9080"

    # file server on port 8081
    files_server: "http://ip_address:8081"

    # Credentials are generated using the webapp, http://ip_address:9080/profile
    credentials {"access_key": "X6QZ9EPB0T6QPYBBTDVN", "secret_key": "xxxxxxxxxxxxxxx"}

    # verify host ssl certificate, set to False only if you have a very good reason
    verify_certificate: True
}

@NathanClearML I am using the docker version of ClearML so I guess it is v1.9.2

@korotaS answer might be a good solution too, I will give it a try if I need to setup a new computer.

Thank you all for these answers!

[Least1924]

create ~/clearml.conf manually worked for me ,thx everybody

[pollfly]

Hey @LouisSnz ! Just letting you know that this issue has been resolved in the recently released clearml v1.10.0. Let us know if there are any issues :)