`clearml-agent init` fails with credentials error

Rusteam commented 3 months ago

Describe the bug

I've created a credentials config from the web app, then tried pasting it during the setup with clearml-agent init, however it throws a credentials validation error, even thought the credentials have been correctly parsed.

To reproduce

Create new credentials settings->workspaces->create new and copy them to clipboard.
Run clearml-agent init on the target machine and follow prompts.
Unable to validation the credentials.

Expected behaviour

A proper clearml.conf created.

Environment

Server type (self-hosted \ ~~app.clear.ml~~)
ClearML SDK Version: clearml-agent==1.7.0
ClearML Server Version (Only for self hosted): WebApp: 1.11.0-373 • Server: 1.11.0-373 • API: 2.25
Python Version: 3.10
OS (Windows \ Linux \ Macos): linux

ainoam commented 3 months ago

@Rusteam Can you provide the full error received?

BTW - any special reason you are not running a more current server version?

Rusteam commented 3 months ago

yes we're delaying the update because for now it's been running fine and it's a manual process.

here's the full log (I've changed the domain name and put stars intead of actual key/secret values):

~ % clearml-agent init
CLEARML-AGENT setup process

Please create new clearml credentials through the settings page in your `clearml-server` web app,
or create a free account at https://app.clear.ml/settings/webapp-configuration

In the settings > workspace  page, press "Create new credentials", then press "Copy to clipboard".

Paste copied configuration here:
api {
    web_server: https://app.internal.ai
    api_server: https://api.internal.ai
    files_server: https://files.internal.ai
    credentials {
        "access_key" = "HEX***"
        "secret_key"  = "YUZRf******"
    }
    verify_certificate = False
}
Detected credentials key="HEX***" secret="YUZR***"
WEB Host configured to: [https://app.internal.ai]
API Host configured to: [https://api.internal.ai]
File Store Host configured to: [https://files.internal.ai]

ClearML Hosts configuration:
Web App: https://app.internal.ai
API: https://api.internal.ai
File Store: https://files.internal.ai

Verifying credentials ...
Error: could not verify credentials: key=HEX***
secret=YUZRf*****
Enter user access key: HEX***
Enter user secret: YUZRf***
Verifying credentials ...
Error: could not verify credentials: key=HEX***
secret=YUZRf****
Exiting setup without creating configuration file

ainoam commented 3 months ago

@Rusteam This looks like some kind of networking issue. Is the API server reachable from where you are trying to run the agent? Is it indeed listening on the default HTTPS port?

Rusteam commented 3 months ago

It's reachable. I've created a clearml.conf manually and have added required agent configs and clearml-agent daemon is working like a magic.

ainoam commented 3 months ago

@Rusteam The UI used to have an issue with an unnecessary whitespace being copied and breaking the CLI. It would be best to validate with a more current version.

Rusteam commented 3 months ago

We thought it would require the update of clearml-server. We will try that

allegroai / clearml