Closed iWoodsman closed 5 months ago
bsekachev
commented
5 months ago Hello,
{"level":"error","msg":"Bundle load failed: server replied with Internal Server Error","name":"cvat","plugin":"bundle","time":"2024-05-01T15:53:36Z"}
That may be a reason.
docker logs cvat_server?
iWoodsman
commented
5 months ago docker logs cvat_server mentions a missing file:
2024-05-02 02:54:05,807 DEBG 'uvicorn-1' stderr output:
[2024-05-02 02:54:05,806] ERROR django.request: Internal Server Error: /api/auth/rules
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 534, in thread_handler
raise exc_info[1]
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
response = await wrapped_callback(
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
return view_func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/home/django/cvat/apps/iam/views.py", line 127, in wrapper
return patched_viewset_method(wsgi_request, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/http.py", line 98, in inner
res_etag = etag_func(request, *args, **kwargs) if etag_func else None
File "/home/django/cvat/apps/iam/views.py", line 146, in <lambda>
@_etag(lambda _: RulesView._etag_func(RulesView._get_bundle_path()))
File "/home/django/cvat/apps/iam/views.py", line 143, in _etag_func
with open(file_path, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/django/static/opa/bundle.tar.gz'
2024-05-02 02:54:05,808 DEBG 'uvicorn-1' stdout output:
INFO: 172.20.0.2:0 - "GET /api/auth/rules HTTP/1.0" 500 Internal Server ErrorI enter the cvat_server container with docker exec -it cvat_server bash I discover that the path /home/django/static/opa/ exists but it is an empty directory, with no bundle.tar.gz there. What could cause that?
sigma-libra
commented
5 months ago Chiming in to add, we have the exact same issue, both with an upgrade from a previous version and installing a fresh version.
sigma-libra
commented
5 months ago Found a solution that works for me.
It appears this issue has happened before, but only to Mac users. The solution for that error, "Error 2", is here: https://github.com/cvat-ai/cvat/issues/6629#issuecomment-2080420315.
Note that 15 seconds was not enough for me, a good 60 second head start for opa did the trick for me, along with deleting the cvat_server image and pulling it again.
iWoodsman
commented
5 months ago I tried it but no luck. I was able to start cvat_opa by itself, and then the other containers, but the same errors were logged about a missing bundle.tar.gz, and the same stall and error when loading the login page. I'd be interested if you noticed that after your successful startup, a tar file did actually appear in /home/django/static/opa/. Is it written there and if so by what? Or is that directory a mount point from cvat_opa that isn't showing up because of a docker network problem underneath the whole mess?
iWoodsman
commented
5 months ago Whoa. I installed cvat on my Intel Mac and it worked immediately. I ducked inside the cvat_server container and checked the path to the missing tar file and the opa path doesn't even exist.
Linux:
django@c05f1e4faba8:~$ ll /home/django/static/
drwxr-xr-x 1 django django 86 May 3 00:48 ./
drwxr-x--- 1 django django 288 Mar 22 08:42 ../
drwxr-xr-x 1 django django 16 May 3 00:48 admin/
drwxr-xr-x 1 django django 0 May 3 00:48 opa/
drwxr-xr-x 1 django django 34 May 3 00:48 rest_framework/
drwxr-xr-x 1 django django 148 May 3 00:48 social_authentication/
Mac:
drwxr-xr-x 1 django django 4096 May 3 01:20 ./
drwxr-x--- 1 django django 4096 May 2 16:06 ../
drwxr-xr-x 5 django django 4096 May 3 01:20 admin/
drwxr-xr-x 7 django django 4096 May 3 01:20 rest_framework/
drwxr-xr-x 2 django django 4096 May 3 01:20 social_authentication/So the Mac seems untroubled by the absence of the path and file that Linux is insisting it needs. I am so confused.
bsekachev
commented
5 months ago @SpecLad
May it be related with recent changes in OPA rules?
SpecLad
commented
5 months ago I suspect the issue here is a mismatch between the version of the Compose file that was used and the Docker images. I recently made a change that updated the code and the Compose file in tandem. The symptoms here are what I would expect if you used the new Compose file with old images.
Basically, add --pull=always to your docker compose up command, and that should fix it. Or run from the master branch.
I discover that the path /home/django/static/opa/ exists but it is an empty directory, with no bundle.tar.gz there. What could cause that?
For the record, the new versions of CVAT no longer store the bundle on disk. So you should expect to see nothing there.
iWoodsman
commented
5 months ago After a docker compose pull, I was able to proceed and the site is now working as expected. Thank you!
boldyshev
commented
5 months ago Same for me, thanks!
Actions before raising this issue
Steps to Reproduce
The site is accessible from the Internet using Chrome at http://host.name.com:8080 but the page that loads shows a spinner before reporting
docker ps shows all containers running. Perhaps it is significant that cvat_opa does not expose a port?
docker logs cvat_opa gives
docker network inspect cvat_cvat gives
I can get inside cvat_server to check the IP address and ports exposed, but I can't docker exec inside cvat_opa which doesn't seem to offer bash or another shell, or ls or ps to poke around.
Expected Behavior
The cvat UI page I reach should present the opportunity to do something, like login as the superuser?
Possible Solution
I feel like there's something with the docker network that is behind the failure of opa to initialize.
Context
No response
Environment