allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
364 stars 132 forks source link

Http 401 in Reports section #218

Open mathrb opened 9 months ago

mathrb commented 9 months ago

Hello,

We have a selfhosted version of clearml. Everything works fine but the reports section, where widget integration (for example to display tasks plots/scalars) does not work. Looking at the request being made by the web browser shows a 401 Http error: POST https://app.clearml.<REDACTED>.com/api/v2.24/reports.get_task_data? Other requests made by the front to the API look like this and work: POST https://api.clearml.<REDACTED>.com/v2.24/projects.get_model_tags

It look likes widgets don't use the right hostname to perform api queries.

Any idea of what is misconfigured? The stack is deployed via docker-compose on a linux server

Update:

Could it be because of this code found in the clearml-web repository: https://github.com/allegroai/clearml-web/blob/aa038f4f8253f45558efe78cc43cf24045ff13d7/src/app/app.constants.ts#L82 As I understand it, it reroute to url + api. In my case, it should be https://api.clearml.<REDACTED>.com/ instead of https://app.clearml.<REDACTED>.com/api. Any possible workaround?

Thanks

jkhenning commented 9 months ago

Hi @mathrb, ClearML Server contains a reverse proxy that should allow calls to https://app.clearml.<REDACTED>.com/api. Can you check the browser developer tools and see the details of the ClearML Cookie? It's possible it's not being used by the browser for this request

mathrb commented 9 months ago

Hi @jkhenning , Indeed there's no cookie associated with the request.

jkhenning commented 9 months ago

And the cookie set on the main WebApp page, can you share its details?

mathrb commented 9 months ago

Hello @jkhenning

Here is the cookie when a request is made to https://api.clearml.<REDACTED>.com/v2.24/reports.get_tags:

_ga=GA1.2.1314995386.1670580637; clearml_token_basic=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2OTQxODgxNzQsImV4cCI6MTY5Njc4MDE3NCwiYXV0aF90eXBlIjoiQmVhcmVyIiwiaWRlbnRpdHkiOnsicm9sZSI6InVzZXIiLCJjb21wYW55X25hbWUiOiJjbGVhcm1sIiwiY29tcGFueSI6ImQxYmQ5MmEzYjAzOTQwMGNiYWZjNjBhN2E1YjFlNTJiIiwidXNlciI6IjdlN2ZhZTVhM2FkYzA5YzI1MGNhMjk2ZjUwYWNmNTQ4IiwidXNlcl9uYW1lIjoiTWF0aGlhcyBIZXJiYXV4In0sImVudiI6Ijx1bmtub3duPiIsImFwaV92ZXJzaW9uIjoiMi4yNCIsInNlcnZlcl92ZXJzaW9uIjoiMS4xMC4wIiwic2VydmVyX2J1aWxkIjoiMzU3IiwiZmVhdHVyZV9zZXQiOiJiYXNpYyJ9.g_<REDACTED>

For standard requests to https://app.clearml.<REDACTED>.com (https://app.clearml.<REDACTED>.com/datasets for example) , I only see a cookie without clearml token:

_ga=GA1.2.1314995386.1670580637
jkhenning commented 9 months ago

Can you share the cookie's properties? domain, secure, etc.?

mathrb commented 9 months ago

In the cookies section, I've got 2:

  1. https://app.clearml.<REDACTED>.com a. Name = _ga, Domain = .<REDACTED>.com, Path = /
  2. https://clearml.<REDACTED>.com a. Name = _ga, Domain = .<REDACTED>.com, Path = / b. Name = clearml_token_basic, Domain = clearml.<REDACTED>.com, Path = / I didn't shared the values since they have already been mentioned in my previous post. Are those the information you wanted?
jkhenning commented 9 months ago

These look OK...

mathrb commented 9 months ago

Is the issue my web browser (firefox/chrome) not forwarding the cookie to the app.clearml.<REDACTED>.com endpoint, whereas it is sent to the api.clearml.<REDACTED>.com endpoint ?

jkhenning commented 9 months ago

If that were the case than the WebApp would not work at all