Cannot get the application to run

dennisdebel commented 8 months ago

Dear! I am totally new with Docker and many of the technologies used here, sorry for that, trying to learn. I feel I got pretty far but still cannot run the image. What I did: adding to requirements.txt > 'networkx==3.1' (somehow the python version does not match with the latest networkx). Installing NVIDIA Container Toolkit, PostgreSQL (and making a database, user with password), making a new Firebase bucket and adding all credentials (and json) to the .env file (btw what is the SECRET_KEY, this is deprecated in firebase).

I am on Ubuntu 22.04.4 LTS.

What to actually put in the .env file?
Any other files to edit?
How to actually run the image after building?
I see firebase_admin gets installed, does this mean you will run a local firebase instance or is this just a interface to google firebase?

The frontend crashes, see:

translation-app-frontend  | yarn run v1.22.19
translation-app-frontend  | $ next dev
translation-app-frontend  | /bin/sh: 1: next: not found
translation-app-frontend  | error Command failed with exit code 127.
translation-app-frontend  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
translation-app-frontend exited with code 127

The backend complains about module 'postgresql' not found. If I add postgresql to requirements.txt the error message also happens but now complains 'example' is not found. Im at a loss here. See:

translation-app-backend   | ModuleNotFoundError: No module named 'postgresql'

Full log:

$ sudo docker compose up --build 
[+] Building 996.7s (23/23) FINISHED                                                                                                  docker:default
 => [backend internal] load build definition from Dockerfile.backend                                                                            0.0s
 => => transferring dockerfile: 482B                                                                                                            0.0s
 => [backend internal] load metadata for docker.io/nvidia/cuda:11.1.1-cudnn8-devel-ubuntu20.04                                                  1.9s
 => [backend internal] load .dockerignore                                                                                                       0.0s
 => => transferring context: 60B                                                                                                                0.0s
 => [backend 1/7] FROM docker.io/nvidia/cuda:11.1.1-cudnn8-devel-ubuntu20.04@sha256:83e4b2841034cdf45ea5b9a5b472eb2c07b1b23d4836d32666a881db29  0.0s
 => [backend internal] load build context                                                                                                       0.0s
 => => transferring context: 8.13kB                                                                                                             0.0s
 => CACHED [backend 2/7] RUN apt-get update && apt-get install -y python3 python3-pip cmake wget llvm                                           0.0s
 => CACHED [backend 3/7] RUN apt-get install -y libglib2.0-0 libsm6 ffmpeg libxext6 libxrender-dev                                              0.0s
 => CACHED [backend 4/7] WORKDIR /translation-app-backend/Backend/                                                                              0.0s
 => [backend 5/7] COPY ./Backend/requirements.txt requirements.txt                                                                              0.0s
 => [backend 6/7] RUN pip install -r requirements.txt                                                                                         950.5s
 => [backend 7/7] COPY . .                                                                                                                      2.1s 
 => [backend] exporting to image                                                                                                               39.6s 
 => => exporting layers                                                                                                                        39.6s 
 => => writing image sha256:a8e1ddae12623261584381448390dc61460491c5190e16f283b500efa77e84f8                                                    0.0s 
 => => naming to docker.io/library/translation_layoutrecovery-backend                                                                           0.0s 
 => [frontend internal] load build definition from Dockerfile.frontend                                                                          0.0s 
 => => transferring dockerfile: 663B                                                                                                            0.0s
 => [frontend internal] load metadata for docker.io/library/node:20.5.1                                                                         2.3s
 => [frontend internal] load .dockerignore                                                                                                      0.0s
 => => transferring context: 60B                                                                                                                0.0s
 => [frontend 1/6] FROM docker.io/library/node:20.5.1@sha256:8d9887b3b05d2e65598a18616c37cfc271346d12248dfcbeadd7b7bf4da1e827                   0.0s
 => [frontend internal] load build context                                                                                                      0.1s
 => => transferring context: 1.82kB                                                                                                             0.1s
 => CACHED [frontend 2/6] WORKDIR /translation-app-frontend/                                                                                    0.0s
 => CACHED [frontend 3/6] RUN apt-get -y update && apt-get install -y nmh                                                                       0.0s
 => CACHED [frontend 4/6] COPY ./Frontend/package.json ./Frontend/yarn.lock .                                                                   0.0s
 => CACHED [frontend 5/6] RUN yarn install && yarn start                                                                                        0.0s
 => CACHED [frontend 6/6] COPY ./Frontend/ .                                                                                                    0.0s
 => [frontend] exporting to image                                                                                                               0.0s
 => => exporting layers                                                                                                                         0.0s
 => => writing image sha256:ff78cbb7220b2d30633f8f6df5fe22a4a60e6aaad164242fbb95d1bb16c88865                                                    0.0s
 => => naming to docker.io/library/translation_layoutrecovery-frontend                                                                          0.0s
[+] Running 4/1
 ✔ Container translation-app-backend                                   Recreated                                                                0.1s 
 ! backend Published ports are discarded when using host network mode                                                                           0.0s 
 ✔ Container translation-app-frontend                                  Recreated                                                                0.0s 
 ! frontend Published ports are discarded when using host network mode                                                                          0.0s 
Attaching to translation-app-backend, translation-app-frontend
translation-app-backend   | 
translation-app-backend   | ==========
translation-app-backend   | == CUDA ==
translation-app-backend   | ==========
translation-app-backend   | 
translation-app-backend   | CUDA Version 11.1.1
translation-app-backend   | 
translation-app-backend   | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
translation-app-backend   | 
translation-app-backend   | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
translation-app-backend   | By pulling and using the container, you accept the terms and conditions of this license:
translation-app-backend   | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
translation-app-backend   | 
translation-app-backend   | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
translation-app-backend   | 
translation-app-frontend  | yarn run v1.22.19
translation-app-frontend  | $ next dev
translation-app-frontend  | /bin/sh: 1: next: not found
translation-app-frontend  | error Command failed with exit code 127.
translation-app-frontend  | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
translation-app-frontend exited with code 127
translation-app-backend   | /usr/local/lib/python3.8/dist-packages/gdown/__main__.py:132: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
translation-app-backend   |   warnings.warn(
translation-app-backend   | Downloading...
translation-app-backend   | From (original): https://drive.google.com/uc?id=1Jx2m_2I1d9PYzFRQ4gl82xQa-G7Vsnsl
translation-app-backend   | From (redirected): https://drive.google.com/uc?id=1Jx2m_2I1d9PYzFRQ4gl82xQa-G7Vsnsl&confirm=t&uuid=7fc30824-9127-47e2-82fc-5dc4acd98ec8
translation-app-backend   | To: /translation-app-backend/Backend/model_196000.pth
100%|██████████| 351M/351M [01:29<00:00, 3.91MB/s] 
translation-app-backend   | Obtaining file:///translation-app-backend
translation-app-backend   | Installing collected packages: translation-app
translation-app-backend   |   Running setup.py develop for translation-app
translation-app-backend   | Successfully installed translation-app
translation-app-backend   | Traceback (most recent call last):
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 113, in load_backend
translation-app-backend   |     return import_module("%s.base" % backend_name)
translation-app-backend   |   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
translation-app-backend   |     return _bootstrap._gcd_import(name[level:], package, level)
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
translation-app-backend   | ModuleNotFoundError: No module named 'postgresql'
translation-app-backend   | 
translation-app-backend   | The above exception was the direct cause of the following exception:
translation-app-backend   | 
translation-app-backend   | Traceback (most recent call last):
translation-app-backend   |   File "manage.py", line 22, in <module>
translation-app-backend   |     main()
translation-app-backend   |   File "manage.py", line 18, in main
translation-app-backend   |     execute_from_command_line(sys.argv)
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
translation-app-backend   |     utility.execute()
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/core/management/__init__.py", line 416, in execute
translation-app-backend   |     django.setup()
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/__init__.py", line 24, in setup
translation-app-backend   |     apps.populate(settings.INSTALLED_APPS)
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/apps/registry.py", line 116, in populate
translation-app-backend   |     app_config.import_models()
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/apps/config.py", line 269, in import_models
translation-app-backend   |     self.models_module = import_module(models_module_name)
translation-app-backend   |   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
translation-app-backend   |     return _bootstrap._gcd_import(name[level:], package, level)
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
translation-app-backend   |   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
translation-app-backend   |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/contrib/auth/models.py", line 3, in <module>
translation-app-backend   |     from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/contrib/auth/base_user.py", line 57, in <module>
translation-app-backend   |     class AbstractBaseUser(models.Model):
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py", line 143, in __new__
translation-app-backend   |     new_class.add_to_class("_meta", Options(meta, app_label))
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/models/base.py", line 371, in add_to_class
translation-app-backend   |     value.contribute_to_class(cls, name)
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/models/options.py", line 243, in contribute_to_class
translation-app-backend   |     self.db_table, connection.ops.max_name_length()
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/utils/connection.py", line 15, in __getattr__
translation-app-backend   |     return getattr(self._connections[self._alias], item)
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/utils/connection.py", line 62, in __getitem__
translation-app-backend   |     conn = self.create_connection(alias)
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 193, in create_connection
translation-app-backend   |     backend = load_backend(db["ENGINE"])
translation-app-backend   |   File "/usr/local/lib/python3.8/dist-packages/django/db/utils.py", line 126, in load_backend
translation-app-backend   |     raise ImproperlyConfigured(
translation-app-backend   | django.core.exceptions.ImproperlyConfigured: 'postgresql' isn't an available database backend or couldn't be imported. Check the above exception. To use one of the built-in backends, use 'django.db.backends.XXX', where XXX is one of:
translation-app-backend   |     'mysql', 'oracle', 'postgresql', 'sqlite3'
translation-app-backend exited with code 1

dennisdebel commented 8 months ago

Solved the 'postgresql' isn't an available database backend or couldn't be imported. error by hard-coding the database engine in `Backend/services/settings.py' like so:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": os.environ.get("DATABASE_NAME"),
        "USER": os.environ.get("DATABASE_USER"),
        "PASSWORD": os.environ.get("DATABASE_PASSWORD"),
        "HOST": os.environ.get("DATABASE_HOST"),
        "PORT": os.environ.get("DATABASE_PORT")
    }
}

Now I get the error that the database 'localhost' can not be found, which is correct. Somehow the settings in .env are not reaching this settings.py. Will hardcode the database name for now and do further testing.

nbtin commented 8 months ago

Hi @dennisdebel, Thanks for your feedback! Sorry for the late reply. Looks like you're having problems with the .env file, firebase storage, and the PostgreSQL database. Let me clarify a little bit for you.

If you got stuck in trying to run a localhost PostgreSQL server. You can try online hosting PostgreSQL database services. For example: Supabase. You can easily create an account and host a project. Then, go to the project setting and find all the related attributes needed for the .env file.
If you face any further problems with Docker, there's another way to run the project. Just install the Frontend (FE) and the Backend (BE) separately by following the instructions in the FE README and BE README files. (to run FE, I think you can try to install npm first, then remember to keep an eye on the HOST URL configuration at Frontend/app/components/config.ts).
Firebase: in this project, we use Firebase as an interface to Google Firebase cloud storage.

About the .env file, I will explain the first 3 attributes:

CREDENTIAL_JSON_FILE_NAME: contains the path to the credential JSON file that you get from configuring Firebase cloud storage for your purpose.
STORAGE_BUCKET_NAME: This attribute specifies the name of a storage bucket to store files on Firebase cloud storage (in this project, we use it to store user's avatars, and their uploaded pdf files). Of course, you can find it in the project's setting on Google Cloud Console. it might be something like “my-awesome-bucket.”
SECRET_KEY: a Django project secret key.

I hope this will help you explore our repo. If you have any concerns regarding our repository, feel free to ask, and we’ll do our utmost to assist you! Screenshot from 2024-02-26 14-49-00

dennisdebel commented 8 months ago

Thank you for the reply! I think I have the back-running after hard coding my .env variables in settings.py, all running locally (except firebase).

The frontend is still an issue. After manually installing nvm, node, npm, yarn, next, react, react-dom and fixing the dependency issues by adding the missing peer dependencies to package.json. The frontend installation does not work. Trying to build it manually in the dir reveals the possible error:

When running (as suggested to fix the BUILD_ID not found error below):

npm install
npm run build

I get the following error:

./app/history/page.tsx
73:33  Error: Missing "key" prop for element in iterator  react/jsx-key
112:37  Error: Missing "key" prop for element in iterator  react/jsx-key

/app/history/page.tsx code looks a bit weird and cant spot the error unfortunately

Console history from docker compose:

=> ERROR [frontend 5/6] RUN yarn install && yarn start                                                                                       435.4s
------                                                                                                                                               
 > [frontend 5/6] RUN yarn install && yarn start:                                                                                                    
0.320 yarn install v1.22.19                                                                                                                          
0.361 [1/4] Resolving packages...                                                                                                                    
9.376 [2/4] Fetching packages...                                                                                                                     
420.0 [3/4] Linking dependencies...                                                                                                                  
423.4 [4/4] Building fresh packages...
434.4 success Saved lockfile.
434.4 Done in 434.12s.
434.6 yarn run v1.22.19
434.6 $ next start
434.9 [Error: ENOENT: no such file or directory, open '/translation-app-frontend/.next/BUILD_ID'] {
434.9   errno: -2,
434.9   code: 'ENOENT',
434.9   syscall: 'open',
434.9   path: '/translation-app-frontend/.next/BUILD_ID'
434.9 }
434.9 error Command failed with exit code 1.
434.9 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
------
failed to solve: process "/bin/sh -c yarn install && yarn start" did not complete successfully: exit code: 1

dennisdebel commented 8 months ago

@nbtin Thank you so much for you instructions and explanation! I gave up using Docker and now running the frontend and backend seperately. The frontend seems to be working fine from the instructions in the Readme.md. Thank you so much for your time!

Two questions issues at the moment:

In the .env file, How to get/set the SECRET_KEY: a Django project secret key?
Getting the backend to work ends at this step python3 manage.py makemigrations account translation when I get the following errors:

(env) user@userpc:~/Documents/translation_layoutrecovery/Backend$ python3 manage.py makemigrations account translation
Traceback (most recent call last):
  File "/home/user/Documents/translation_layoutrecovery/Backend/manage.py", line 22, in <module>
    main()
  File "/home/user/Documents/translation_layoutrecovery/Backend/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/management/base.py", line 453, in execute
    self.check()
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/management/base.py", line 485, in check
    all_issues = checks.run_checks(
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    return check_method()
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/urls/resolvers.py", line 494, in check
    for pattern in self.url_patterns:
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/urls/resolvers.py", line 715, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/Documents/translation_layoutrecovery/Backend/services/urls.py", line 24, in <module>
    path("", include("account.urls")),  # Added account app
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/django/urls/conf.py", line 38, in include
    urlconf_module = import_module(urlconf_module)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/Documents/translation_layoutrecovery/Backend/account/urls.py", line 2, in <module>
    from .views import Register, Login, Logout, GetUserData, UpdateProfile
  File "/home/user/Documents/translation_layoutrecovery/Backend/account/views.py", line 28, in <module>
    cred = credentials.Certificate(credential_json)
  File "/home/user/Documents/translation_layoutrecovery/Backend/env/lib/python3.10/site-packages/firebase_admin/credentials.py", line 88, in __init__
    raise ValueError(
ValueError: Invalid certificate argument: "None". Certificate argument must be a file path, or a dict containing the parsed file contents.

workphone-002 commented 8 months ago

Thanks for your response before. As for the problems with next, the solution I indicated in error 2 #11 (modify docker-compose) seems to work. On the other hand, regarding your last question:

You can generate a key for django here https://djecrety.ir/
The error you have with the certificates seems to indicate that you have not configured the .env file correctly. You have to generate a private key in firebase (json file) and paste it in the backend folder, then in the .env file put the file name in the CREDENTIAL_JSON_FILE_NAME section.

dennisdebel commented 8 months ago

Thank you for the reply! Glad to see you got further! I cant get the json credentials to work. So strange, python can't find the file (checked name, path, permissions). Maybe I start with a fresh clone and work from there...

/EDIT

After a clean clone, the whole process was a lot smoother up until the same credentials error...

phkhanhtrinh23 / translation_layoutrecovery

Cannot get the application to run #10