analytics not accesible in https with public hostname

rho-sk commented 2 years ago

My actions before raising this issue

[x] Read/searched the docs
[x] Searched past issues

Expected Behaviour

As described in docu, when admin or business user clicks on "Analytics" in top toolbar, kibana should appear.

Current Behaviour

All cvat functionality works fine on https://v2-aerial.lab.ainalytics.online
Link seems to be generated fine, https://v2-aerial.lab.ainalytics.online/analytics/app/kibana
But after click on "Analytics" in top toolbar, there is just https://v2-aerial.lab.ainalytics.online/tasks?page=1

Possible Solution

Steps to Reproduce (for bugs)

server is on public cloud (gcloud), there is public IP allocated, cloud DNS record is pointed to this IP, fw rules are enabled

Install as usual version 2.0.0

if ! [ -d "$HOME/cvat" ]; then
git clone https://github.com/openvinotoolkit/cvat.git
git checkout v2.0.0
fi
sudo chown -Rf $_USER:$_USER ./cvat

Build

cd /home/${var.remote_user}/cvat && docker-compose -f docker-compose.yml -f docker-compose.dev.yml -f components/analytics/docker-compose.analytics.yml -f docker-compose-override.yml build

Run (i have small sh scipt for that)

cd /home/ubuntu/cvat
export CVAT_HOST=v2-aerial.lab.ainalytics.online
export ACME_EMAIL=my_secret_email :-)
docker-compose -f docker-compose.yml -f docker-compose.https.yml -f components/analytics/docker-compose.analytics.yml -f docker-compose-override.yml up -d

superuser account is created (i have small sh scipt for silent mode)

docker exec -t cvat bash -ic "export DJANGO_SUPERUSER_USERNAME=${local.superuser} && export DJANGO_SUPERUSER_PASSWORD=${local.superuser_pass} && python3 ~/manage.py createsuperuser --email=${var.admin_email} --noinput"

Context

I need analytics, to see how "labelers" are performing, because they are paid by their performance.

Your Environment

Git hash commit (git log -1): commit 6fd7a0e1aef17793d41a2362c0f8db53d9d83eac
Docker version docker version (e.g. Docker 17.0.05): 20.10.14
Are you using Docker Swarm or Kubernetes? NO
Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 18.04.6 LTS
Code example or link to GitHub repo or gist to reproduce problem:
Other diagnostic information / logs:
- Logs from all containers logs.tar.gz

Next steps

You may join our Gitter channel for community support.

rho-sk commented 2 years ago

My assumption is that, there is problem with rewrite rules in kibana config. Because i did test with bypassed direct connection to container:

exposed 5601 from kibana container
did ssh connect with -L localhost:5601:localhost:5601
access http://localhost:5601/api/status , all looks "green"
than any attempt http://localhost:5601/analytics/, http://localhost:5601/analytics/app/kibana - there's 404

azhavoro commented 2 years ago

I think there you issue is related to traefik route configuration in case of https

https://github.com/openvinotoolkit/cvat/blob/develop/components/analytics/kibana_conf.yml#L5

My assumption is that, there is problem with rewrite rules in kibana config. Because i did test with bypassed direct connection to container:

exposed 5601 from kibana container

did ssh connect with -L localhost:5601:localhost:5601

access http://localhost:5601/api/status , all looks "green"

than any attempt http://localhost:5601/analytics/, http://localhost:5601/analytics/app/kibana - there's 404

Hi, I think this issue is related to traefik route configuration in case of https

There is missed websecure entrypoint https://github.com/openvinotoolkit/cvat/blob/develop/components/analytics/kibana_conf.yml#L5 that added by docker-compose.https.yaml for frontend and backend services

rho-sk commented 2 years ago

Moving forward :-)

Make sense to change websecure, so kibana_conf.yml looks

http:
  routers:
    kibana:
      entryPoints:
      - websecure
      middlewares:
      - analytics-auth
      - strip-prefix
      service: kibana
      rule: Host(`{{ env "CVAT_HOST" }}`) && PathPrefix(`/analytics`)

  middlewares:
    analytics-auth:
      forwardauth:
        address: http://cvat:8080/analytics
        authRequestHeaders:
          - "Cookie"
          - "Authorization"

    strip-prefix:
      stripprefix:
        prefixes:
        - /analytics

  services:
    kibana:
      loadBalancer:
        servers:
        - url: http://{{ env "DJANGO_LOG_VIEWER_HOST" }}:{{ env "DJANGO_LOG_VIEWER_PORT" }}
        passHostHeader: false

There is also another problem, that this rule is not loaded by traefik, because "file.provider" is removed, via new "command" directive in docker-compose.https.yml Solution for this BUG is to add

     - "--providers.file.directory=/etc/traefik/rules"

to file (this can be permanent for everybody) https://github.com/openvinotoolkit/cvat/blob/develop/docker-compose.https.yml#L23

But problem still remains, i put traefik into debug mode. And here are the logs. There is request to /analytics, but its forwared internally to service at 80 port. So it seems there is problem with order of rules in traefik ?!

traefik.log.txt

rho-sk commented 2 years ago

Well, i found the problem, after i enabled traefik dashboard and saw that rule for kibana has no tls.

So my local fix when https is used together with analytics is:

https://github.com/openvinotoolkit/cvat/blob/develop/components/analytics/kibana_conf.yml

entryPoints must be websecure (as @azhavoro mentioned),
tls: {} must be declared there too

https://github.com/openvinotoolkit/cvat/blob/develop/docker-compose.https.yml

"--providers.file.directory=/etc/traefik/rules" after line 23

My working kibana_conf.yml:

http:
  routers:
    kibana:
      entryPoints:
      - websecure
      middlewares:
      - analytics-auth
      - strip-prefix
      service: kibana
      tls: {}
      rule: Host(`{{ env "CVAT_HOST" }}`) && PathPrefix(`/analytics`)

  middlewares:
    analytics-auth:
      forwardauth:
        address: http://cvat:8080/analytics
        authRequestHeaders:
          - "Cookie"
          - "Authorization"

    strip-prefix:
      stripprefix:
        prefixes:
        - /analytics

  services:
    kibana:
      loadBalancer:
        servers:
        - url: http://{{ env "DJANGO_LOG_VIEWER_HOST" }}:{{ env "DJANGO_LOG_VIEWER_PORT" }}
        passHostHeader: false

But this is not generic fix for github repo So, it is still valid bug, which remains unfixed in this repository, thats why i will not close this issue.

michellemoorre commented 2 years ago

We've faced the same problem. Is there more generic fix?

ConsciousML commented 2 years ago

Nice fix. It worked for me with the same configuration and on GCP too. Thanks for you're help !

AndrewDHill commented 2 years ago

Did you use letsencrypt? I am using a self signed cert and I think this may add another layer to this problem since your solution doesn't seem to fix this issue for me. I am still getting routed back to the task page

osinpl commented 1 year ago

thanks @rho-sk good job, your solutions works for me!

bsekachev commented 1 year ago

Analytics was signgificantly redesigned in new releases. I will close the issue as outdated.

cvat-ai / cvat