Hit error when run local dev build "No such file or directory: '/opt/recipes/vue/webpack-stats.json'"

[srwareham]

Issue

I'd like to try writing a few features for Tandoor but cannot get a local build to run. In every instance, I hit a Server Error (500) on the frontend and FileNotFoundError: [Errno 2] No such file or directory: '/opt/recipes/vue/webpack-stats.json' in the command line backend.

I have tried two different approaches to setting up a test environment:

1. Following the Exact steps specified in the Django section of https://docs.tandoor.dev/contribute/
2. Modifying my existing (and happily working) tandoor docker-compose.yml to swap in a manually built image in place of vabene1111/recipes:latest (this modified version is attached below and I have cloned the repo to a recipes dir in the same folder as docker-compose.yml)

In both cases, I get the error saying there's no webpack-stats.json file. I have also tried both approaches while checking out the 1.5.3 tag of the git repo, but still hit the same error. In both cases, the error seems to originate during the python manage.py runserver part of boot.sh.

When I try approach 2, I can manually navigate to the settings page, but not any of the recipes or the default /search landing page. If I revert my docker-compose.yml to use vabene1111/recipes:1.5.3 everything works fine. But then I can't test new features.

Thank you!

Tandoor Version

1.5.3

OS Version

Arch Linux (Linux Arch 6.3.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 21 Jun 2023 20:46:20 +0000 x86_64 GNU/Linux)

Setup

Docker / Docker-Compose

Reverse Proxy

Others (please state below)

Other

nginx

Environment file

# only set this to true when testing/debugging
# when unset: 1 (true) - dont unset this, just for development
DEBUG=1
SQL_DEBUG=1

# HTTP port to bind to
# TANDOOR_PORT=8080

# hosts the application can run under e.g. recipes.mydomain.com,cooking.mydomain.com,...
ALLOWED_HOSTS=*

# random secret key, use for example `base64 /dev/urandom | head -c50` to generate one
# ---------------------------- REQUIRED -------------------------
SECRET_KEY=redacted
# ---------------------------------------------------------------

# your default timezone See https://timezonedb.com/time-zones for a list of timezones
TIMEZONE=America/New_York

# add only a database password if you want to run with the default postgres, otherwise change settings accordingly
DB_ENGINE=django.db.backends.postgresql
# DB_OPTIONS= {} # e.g. {"sslmode":"require"} to enable ssl
POSTGRES_HOST=db_recipes
POSTGRES_PORT=5432
POSTGRES_USER=djangouser
# ---------------------------- REQUIRED -------------------------
# TODO: replace with var/vault
POSTGRES_PASSWORD=redacted
# ---------------------------------------------------------------
POSTGRES_DB=djangodb

# database connection string, when used overrides other database settings.
# format might vary depending on backend
# DATABASE_URL = engine://username:password@host:port/dbname

# the default value for the user preference 'fractions' (enable/disable fraction support)
# default: disabled=0
FRACTION_PREF_DEFAULT=0

# the default value for the user preference 'comments' (enable/disable commenting system)
# default comments enabled=1
COMMENT_PREF_DEFAULT=1

# Users can set a amount of time after which the shopping list is refreshed when they are in viewing mode
# This is the minimum interval users can set. Setting this to low will allow users to refresh very frequently which
# might cause high load on the server. (Technically they can obviously refresh as often as they want with their own scripts)
SHOPPING_MIN_AUTOSYNC_INTERVAL=5

# Default for user setting sticky navbar
# STICKY_NAV_PREF_DEFAULT=1

# If base URL is something other than just / (you are serving a subfolder in your proxy for instance http://recipe_app/recipes/)
# Be sure to not have a trailing slash: e.g. '/recipes' instead of '/recipes/'
# SCRIPT_NAME=/recipes

# If staticfiles are stored at a different location uncomment and change accordingly, MUST END IN /
# this is not required if you are just using a subfolder
# This can either be a relative path from the applications base path or the url of an external host
# STATIC_URL=/static/

# If mediafiles are stored at a different location uncomment and change accordingly, MUST END IN /
# this is not required if you are just using a subfolder
# This can either be a relative path from the applications base path or the url of an external host
# MEDIA_URL=/media/

# Serve mediafiles directly using gunicorn. Basically everyone recommends not doing this. Please use any of the examples
# provided that include an additional nxginx container to handle media file serving.
# If you know what you are doing turn this back on (1) to serve media files using djangos serve() method.
# when unset: 1 (true) - this is temporary until an appropriate amount of time has passed for everyone to migrate
GUNICORN_MEDIA=1

# GUNICORN SERVER RELATED SETTINGS (see https://docs.gunicorn.org/en/stable/design.html#how-many-workers for recommended settings)
 GUNICORN_WORKERS=1
 GUNICORN_THREADS=1

# S3 Media settings: store mediafiles in s3 or any compatible storage backend (e.g. minio)
# as long as S3_ACCESS_KEY is not set S3 features are disabled
# S3_ACCESS_KEY=
# S3_SECRET_ACCESS_KEY=
# S3_BUCKET_NAME=
# S3_REGION_NAME= # default none, set your region might be required
# S3_QUERYSTRING_AUTH=1 # default true, set to 0 to serve media from a public bucket without signed urls
# S3_QUERYSTRING_EXPIRE=3600 # number of seconds querystring are valid for
# S3_ENDPOINT_URL= # when using a custom endpoint like minio
# S3_CUSTOM_DOMAIN= # when using a CDN/proxy to S3 (see https://github.com/TandoorRecipes/recipes/issues/1943)

# Email Settings, see https://docs.djangoproject.com/en/3.2/ref/settings/#email-host
# Required for email confirmation and password reset (automatically activates if host is set)
# EMAIL_HOST=
# EMAIL_PORT=
# EMAIL_HOST_USER=
# EMAIL_HOST_PASSWORD=
# EMAIL_USE_TLS=0
# EMAIL_USE_SSL=0
# email sender address (default 'webmaster@localhost')
# DEFAULT_FROM_EMAIL=
# prefix used for account related emails (default "[Tandoor Recipes] ")
# ACCOUNT_EMAIL_SUBJECT_PREFIX=

# allow authentication via reverse proxy (e.g. authelia), leave off if you dont know what you are doing
# see docs for more information https://vabene1111.github.io/recipes/features/authentication/
# when unset: 0 (false)
REVERSE_PROXY_AUTH=0

# Default settings for spaces, apply per space and can be changed in the admin view
# SPACE_DEFAULT_MAX_RECIPES=0 # 0=unlimited recipes
# SPACE_DEFAULT_MAX_USERS=0 # 0=unlimited users per space
# SPACE_DEFAULT_MAX_FILES=0 # Maximum file storage for space in MB. 0 for unlimited, -1 to disable file upload.
# SPACE_DEFAULT_ALLOW_SHARING=1 # Allow users to share recipes with public links

# allow people to create accounts on your application instance (without an invite link)
# when unset: 0 (false)
# ENABLE_SIGNUP=0

# If signup is enabled you might want to add a captcha to it to prevent spam
# HCAPTCHA_SITEKEY=
# HCAPTCHA_SECRET=

# if signup is enabled you might want to provide urls to data protection policies or terms and conditions
# TERMS_URL=
# PRIVACY_URL=
# IMPRINT_URL=

# enable serving of prometheus metrics under the /metrics path
# ATTENTION: view is not secured (as per the prometheus default way) so make sure to secure it
# trough your web server (or leave it open of you dont care if the stats are exposed)
# ENABLE_METRICS=0

# allows you to setup OAuth providers
# see docs for more information https://vabene1111.github.io/recipes/features/authentication/
# SOCIAL_PROVIDERS = allauth.socialaccount.providers.github, allauth.socialaccount.providers.nextcloud,

# Should a newly created user from a social provider get assigned to the default space and given permission by default ?
# ATTENTION: This feature might be deprecated in favor of a space join and public viewing system in the future
# default 0 (false), when 1 (true) users will be assigned space and group
# SOCIAL_DEFAULT_ACCESS = 1

# if SOCIAL_DEFAULT_ACCESS is used, which group should be added
# SOCIAL_DEFAULT_GROUP=guest

# Django session cookie settings. Can be changed to allow a single django application to authenticate several applications
# when running under the same database
# SESSION_COOKIE_DOMAIN=.example.com
# SESSION_COOKIE_NAME=sessionid # use this only to not interfere with non unified django applications under the same top level domain

# by default SORT_TREE_BY_NAME is disabled this will store all Keywords and Food in the order they are created
# enabling this setting makes saving new keywords and foods very slow, which doesn't matter in most usecases.
# however, when doing large imports of recipes that will create new objects, can increase total run time by 10-15x
# Keywords and Food can be manually sorted by name in Admin
# This value can also be temporarily changed in Admin, it will revert the next time the application is started
# This will be fixed/changed in the future by changing the implementation or finding a better workaround for sorting
# SORT_TREE_BY_NAME=0
# LDAP authentication
# default 0 (false), when 1 (true) list of allowed users will be fetched from LDAP server
#LDAP_AUTH=
#AUTH_LDAP_SERVER_URI=
#AUTH_LDAP_BIND_DN=
#AUTH_LDAP_BIND_PASSWORD=
#AUTH_LDAP_USER_SEARCH_BASE_DN=
#AUTH_LDAP_TLS_CACERTFILE=

# Enables exporting PDF (see export docs)
# Disabled by default, uncomment to enable
# ENABLE_PDF_EXPORT=1

# Recipe exports are cached for a certain time by default, adjust time if needed
# EXPORT_FILE_CACHE_DURATION=600

Docker-Compose file

version: "3"
name: tandoor
services:
  db_recipes:
    restart: unless-stopped
    image: postgres:11-alpine
    container_name: "tandoor_db"
    volumes:
      - "./postgresql:/var/lib/postgresql/data"
    env_file:
      - ./.env

  web_recipes:
    restart: unless-stopped
    build: ./recipes
    container_name: "tandoor_web"
    env_file:
      - ./.env
    ports:
      - "9106:8080"
    volumes:
      - "./postgres:/var/lib/postgresql/data"
      - "./staticfiles:/opt/recipes/staticfiles"
      - "./nginx_config:/opt/recipes/nginx/conf.d"
      - "./mediafiles:/opt/recipes/mediafiles"
    depends_on:
      - db_recipes

  nginx_recipes:
    image: nginx:mainline-alpine
    restart: unless-stopped
#    ports:
    env_file:
      - ./.env
    depends_on:
      - web_recipes
    volumes:
      - "./nginx_config:/etc/nginx/conf.d:ro"
      - "./staticfiles:/static:ro"
      - "./mediafiles:/media:ro"

Relevant logs

Database is ready
Migrating database
ERROR failed to initialize plugins
Operations to perform:
  Apply all migrations: account, admin, auth, authtoken, contenttypes, cookbook, oauth2_provider, sessions, sites, socialaccount
Running migrations:
  No migrations to apply.
  Your models in app(s): 'cookbook' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Generating static files
ERROR failed to initialize plugins

[gloriousDan]

I think the problem is, that you have to build the vue frontend first. That's why it complains about missing webpack files.

If you have a look at the build pipeline, there is an additional step using yarn to build the frontend. You would have to run something similar before building the docker file: https://github.com/TandoorRecipes/recipes/blob/develop/.github/workflows/build-docker.yml#L47-L58

# Build Vue frontend
- uses: actions/setup-node@v3
  with:
    node-version: '18'
    cache: yarn
    cache-dependency-path: vue/yarn.lock
- name: Install dependencies
  working-directory: ./vue
  run: yarn install --frozen-lockfile
- name: Build dependencies
  working-directory: ./vue
  run: yarn build

It would maybe be possible to use a multistage docker build but that's not possible with the official docker file right now.

This is not mentioned in the contributing guidelines because for local development you are supposed to run a dev server for the vue frontend with yarn serve. Is there a special reason why you want to build the docker image for development?

[srwareham]

Thank you!! That worked and taught me something new about github workflows/builds and vue. I wanted to build the docker image for development for 2 good reasons and one bad:

1. (bad) It's what I'm familiar with and knew how to do
2. It lets me easily use my existing infrastructure to test. Mostly ansible scripts that set up testing-specific reverse proxy subdomains and BTRFS snapshotting
3. It lets me actually use my changes before they are accepted upstream (which isn't always a guarantee)

Out of curiosity, why is it better to have the yarn steps outside of the Dockerfile? Wouldn't having them in the Dockerfile make it more accurately depict the contents of a running environment? No worries either way, I appreciate the time you've taken already!

[gloriousDan]

Great that it worked, these reasons are very valid and I also already stumbled over the problem that just executing the docker build doesn't work without extra steps. As to why it isn't done in the docker build: I don't know, probably historical reasons and a "simpler" Dockerfile. There shouldn't really be a reason why we can't change it.

[smilerz]

Great that it worked, these reasons are very valid and I also already stumbled over the problem that just executing the docker build doesn't work without extra steps. As to why it isn't done in the docker build: I don't know, probably historical reasons and a "simpler" Dockerfile. There shouldn't really be a reason why we can't change it.

Building in the container would require all of the source files and development tooling which would significantly bloat the size of the container.

[gloriousDan]

This can be avoided by using multi-stage builds. Essentially you do all the compiling and building steps in one stage, copy only the final wheels and other stuff which is needed to run the container into the final stage and then discard the build stage. So the yarn build process could be done in its own stage.

[smilerz]

This can be avoided by using multi-stage builds. Essentially you do all the compiling and building steps in one stage, copy only the final wheels and other stuff which is needed to run the container into the final stage and then discard the build stage. So the yarn build process could be done in its own stage.

Which in turn adds complexity to those running manual installs. I don’t see any compelling reason to switch how it’s currently working.

[gloriousDan]

I'm not sure how using multi-stage builds would add complexity. For people using the manual setup nothing changes because the changes would only affect the docker build. For people who want to build the docker image themselves it's arguably easier because they don't need to know/ remember the additional node build step before building the image. But I agree that it's not a very common use case. Issues like this one show though, that people stumble over this problem.

[srwareham]

@smilerz what complexity are you worried about for manual installs? I think a goal of "just build the dockerfile and it will run" is nice aspiration if not in conflict with core processes.