[Feature Request] Persistent Data

Hi, I am not sure if posting this here is right, as I am new to github. Please correct mistakes made.

My request

So there are several things:

When redeploying the container I always need to install/upgrade again, which is troublesome when there isn't actually an update. -> Updating only works when the installed version is newer then the one registered in the database. (May be an issue to Hesk itself).
I have to download my language pack into the container again.
I need to edit my presets e.g. /srv/language/<locale>/emails/<preset>.txt and /srv/hesk_settings.inc.php

This all takes some time so I am wondering if you may persist some of these settings. Maybe using environment variables or a conf.yml.

My configuration

I currently use your hesk image using docker-compose and mariadb. My docker-compose.yml looks like this:

version: '3.9'
services:
  hesk-mariadb:
    hostname: hesk-mariadb
    container_name: hesk-mariadb
    image: 'bitnami/mariadb:latest'
    environment:
      - MARIADB_ROOT_PASSWORD=<rpw>
      - MARIADB_USER=hesk
      - MARIADB_PASSWORD=<upw>
      - MARIADB_DATABASE=hesk
      - MARIADB_SKIP_TEST_DB=yes
    volumes:
      - '/opt/helpdesk/mariadb_data:/bitnami/mariadb'
    networks:
      - proxy
  hesk:
    image: ghcr.io/luketainton/hesk:latest
    hostname: hesk
    container_name: hesk
    depends_on:
      - hesk-mariadb
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.hesk-support.entrypoints=http"
      - "traefik.http.routers.hesk-support.rule=Host(`helpdesk.domain.tld`)"
      - "traefik.http.middlewares.hesk-support-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.hesk-support.middlewares=hesk-support-https-redirect"
      - "traefik.http.routers.hesk-support-secure.entrypoints=https"
      - "traefik.http.routers.hesk-support-secure.rule=Host(`helpdesk.domain.tld`)"
      - "traefik.http.routers.hesk-support-secure.tls=true"
      - "traefik.http.routers.hesk-support-secure.tls.certresolver=http"
      - "traefik.http.services.hesk-support.loadbalancer.server.port=80"
      - "traefik.docker.network=proxy"
    networks:
      - proxy

networks:
  proxy:
    external: true

Best regards, Kamino

Hi @I-am-Kamino!

Thanks for the feature request. For the following points:

I have to download my language pack into the container again.
_I need to edit my presets e.g. /srv/language//emails/.txt and /srv/hesksettings.inc.php

I'd suggest downloading the language pack and creating these preset files on your host, and volume-mounting them into the container. I'll see if there's a better way to do this in the meantime.

I'll look into why you keep getting asked to re-install it.

Hi @luketainton. I'd like to second @I-am-Kamino's request, and add some info. Hopefully this wall of text will be of some help to you both.

The issue

I'm experiencing similar issues to @I-am-Kamino. If, for some reason, I have to recreate the HESK container, it forces me to go through the install process again. Only it can't complete it, since the database already exists. Updating doesn't work either, because the script responsible for updating refers to the default HESK login values for the already-existing database. It then errors out with the following message if you've changed any of those values (which you should):

Fatal error: Uncaught mysqli_sql_exception: No such file or directory in /srv/install/install_functions.inc.php:105 Stack trace: #0 /srv/install/install_functions.inc.php(105): mysqli_connect('localhost', 'test', 'test', 'hesk') #1 /srv/install/update.php(87): hesk_iTestDatabaseConnection(true) #2 {main} thrown in /srv/install/install_functions.inc.php on line 105

What didn't work

The /srv directory within the docker container seems to be the one where all the HESK files live. This is, incidentally, also where in the install directory lives that needs to be removed after installation. There's also a themes folder, which is where you'd add any new HESK themes you'd want to use. In short, i'd like to be able to mount the whole /srv/ directory as a persistent volume in order to make life a little easier. I've tried adding the following to my docker-compose.yml file under the HESK container section in an effort to persist data:

    volumes:
      - ./config:/srv

However, this creates an empty directory without any of the necessary files after spinning up the containers.

What worked

Eventually, I got it to work by going through the following steps:

Clone this repo and cd into the directory.
Edit the docker-compose.example.yml file so that it looks like the one at the bottom of this post, plus any other things you'd like to add. Save it as docker-compose.yml.
Create the directory you're going to use as persistent volume with mkdir (in my case: config in the local directory).
Copy the contents of the hesk directory in the repo to the directory created in the previous step.
Give global access (chmod 777) to the directory from step 3, and all files within it. If you don't do this, HESK will complain that PHP cannot write to certain directories. I didn't have the time to try and ferret out all affected directories in order to selectively set permissions, so I opted for the stupid and easy option.
Execute docker-compose up -d
Go through the installation steps as you normally would. At the end of installation, remove the install directory from the directory created in step 3.
HESK should work fine now, and persist through recreated containers without reinstalling. You can use the directory from step 3 to add themes, edit files, install language packs, or do any other things you'd like to do.

My `docker-compose.yml` file

version: "3"
services:
  db:
    image: mariadb:10.7
    container_name: hesk_db
    volumes:
      - ./db_data:/var/lib/mysql
    restart: unless-stopped
    environment:
      MARIADB_RANDOM_ROOT_PASSWORD: yes
      MARIADB_DATABASE: <DB_NAME>
      MARIADB_USER: <HESK_USER>
      # Please change the password before deploying.
      MARIADB_PASSWORD: <PASSWORD>

  hesk:
    depends_on:
      - db
    image: ghcr.io/luketainton/hesk:latest
    container_name: hesk
    volumes:
      - ./config:/srv
    ports:
      - "80:80"
    restart: unless-stopped

@Gitoffomalawn

Right, that would work because it's editing the HESK config file directly on the host. The issue comes when you update HESK to a new version, although I'd like to assume they provide guides on how to do this.

The easiest option I can think of right now is to keep the config file and other persistent files on the host (or in a Docker volume) and mount them into the container.

It might take some time for me to identify all relevant files (I have a day job too 😄) so bear with, unless you know of them?

Right, that would work because it's editing the HESK config file directly on the host. The issue comes when you update HESK to a new version, although I'd like to assume they provide guides on how to do this.

Yeah, it's definitely not working as expected. Normally, manual editing on the host shouldn't be necessary. But as a workaround it works.

Updating HESK shouldn't be an issue. You usually only need to replace a few files, and you're done. A diff of two .zip files with different versions should show you which. You could do this manually, of course (not ideal), or possibly by deploying a new image that references the new files.

It might take some time for me to identify all relevant files (I have a day job too 😄) so bear with, unless you know of them?

I'll see if I can dig around my notes. I think I may have already looked into it once.

As to the permissions settings I talked about, I've not had time to look into those (which is why I opted for the global 777 approach). Same problems with a day job here too.

I'll post an update if I find anything. Might take a while though.

@luketainton

Here's a list of files that are different between HESK v3.2.5 and v3.3.1:

Files hesk325/admin/admin_reply_ticket.php and hesk331/admin/admin_reply_ticket.php differ
Files hesk325/admin/admin_settings_email.php and hesk331/admin/admin_settings_email.php differ
Files hesk325/admin/admin_settings_general.php and hesk331/admin/admin_settings_general.php differ
Files hesk325/admin/admin_settings_help_desk.php and hesk331/admin/admin_settings_help_desk.php differ
Files hesk325/admin/admin_settings_misc.php and hesk331/admin/admin_settings_misc.php differ
Files hesk325/admin/admin_settings_save.php and hesk331/admin/admin_settings_save.php differ
Files hesk325/admin/admin_settings_ticket_list.php and hesk331/admin/admin_settings_ticket_list.php differ
Files hesk325/admin/admin_submit_ticket.php and hesk331/admin/admin_submit_ticket.php differ
Files hesk325/admin/admin_ticket.php and hesk331/admin/admin_ticket.php differ
Files hesk325/admin/assign_owner.php and hesk331/admin/assign_owner.php differ
Files hesk325/admin/banned_emails.php and hesk331/admin/banned_emails.php differ
Files hesk325/admin/banned_ips.php and hesk331/admin/banned_ips.php differ
Files hesk325/admin/custom_fields.php and hesk331/admin/custom_fields.php differ
Files hesk325/admin/delete_tickets.php and hesk331/admin/delete_tickets.php differ
Files hesk325/admin/edit_post.php and hesk331/admin/edit_post.php differ
Only in hesk331/admin: elevator.php
Files hesk325/admin/email_templates.php and hesk331/admin/email_templates.php differ
Files hesk325/admin/export.php and hesk331/admin/export.php differ
Files hesk325/admin/find_tickets.php and hesk331/admin/find_tickets.php differ
Files hesk325/admin/index.php and hesk331/admin/index.php differ
Files hesk325/admin/mail.php and hesk331/admin/mail.php differ
Files hesk325/admin/manage_canned.php and hesk331/admin/manage_canned.php differ
Files hesk325/admin/manage_categories.php and hesk331/admin/manage_categories.php differ
Only in hesk331/admin: manage_category.php
Files hesk325/admin/manage_knowledgebase.php and hesk331/admin/manage_knowledgebase.php differ
Only in hesk331/admin: manage_mfa.php
Files hesk325/admin/manage_ticket_templates.php and hesk331/admin/manage_ticket_templates.php differ
Files hesk325/admin/manage_users.php and hesk331/admin/manage_users.php differ
Only in hesk331/admin: module_satisfaction.php
Files hesk325/admin/move_category.php and hesk331/admin/move_category.php differ
Files hesk325/admin/new_ticket.php and hesk331/admin/new_ticket.php differ
Files hesk325/admin/password.php and hesk331/admin/password.php differ
Files hesk325/admin/profile.php and hesk331/admin/profile.php differ
Files hesk325/admin/reports.php and hesk331/admin/reports.php differ
Files hesk325/admin/resend_notification.php and hesk331/admin/resend_notification.php differ
Files hesk325/admin/service_messages.php and hesk331/admin/service_messages.php differ
Files hesk325/admin/test_connection.php and hesk331/admin/test_connection.php differ
Files hesk325/cron/email_overdue_tickets.php and hesk331/cron/email_overdue_tickets.php differ
Files hesk325/css/app.css and hesk331/css/app.css differ
Files hesk325/css/app.min.css and hesk331/css/app.min.css differ
Only in hesk331/css: dropzone.min.css
Only in hesk331/css: reduce_table_padding.css
Files hesk325/docs/changelog.html and hesk331/docs/changelog.html differ
Files hesk325/docs/index.html and hesk331/docs/index.html differ
Files hesk325/docs/quick-guide.html and hesk331/docs/quick-guide.html differ
Files hesk325/docs/step-by-step-guide.html and hesk331/docs/step-by-step-guide.html differ
Files hesk325/hesk_settings.inc.php and hesk331/hesk_settings.inc.php differ
Files hesk325/img/label-low.svg and hesk331/img/label-low.svg differ
Only in hesk331/img: satisfaction.jpg
Files hesk325/inc/admin_functions.inc.php and hesk331/inc/admin_functions.inc.php differ
Files hesk325/inc/admin_settings_status.inc.php and hesk331/inc/admin_settings_status.inc.php differ
Files hesk325/inc/attachments.inc.php and hesk331/inc/attachments.inc.php differ
Files hesk325/inc/common.inc.php and hesk331/inc/common.inc.php differ
Files hesk325/inc/database.inc.php and hesk331/inc/database.inc.php differ
Files hesk325/inc/database_mysqli.inc.php and hesk331/inc/database_mysqli.inc.php differ
Files hesk325/inc/email_functions.inc.php and hesk331/inc/email_functions.inc.php differ
Files hesk325/inc/export_functions.inc.php and hesk331/inc/export_functions.inc.php differ
Files hesk325/inc/footer.inc.php and hesk331/inc/footer.inc.php differ
Files hesk325/inc/header.inc.php and hesk331/inc/header.inc.php differ
Files hesk325/inc/htmlpurifier/HeskHTMLPurifier.php and hesk331/inc/htmlpurifier/HeskHTMLPurifier.php differ
Files hesk325/inc/mail/email_parser.php and hesk331/inc/mail/email_parser.php differ
Files hesk325/inc/mail/hesk_imap.php and hesk331/inc/mail/hesk_imap.php differ
Files hesk325/inc/mail/hesk_pipe.php and hesk331/inc/mail/hesk_pipe.php differ
Files hesk325/inc/mail/hesk_pop3.php and hesk331/inc/mail/hesk_pop3.php differ
Files hesk325/inc/mail/mime_parser.php and hesk331/inc/mail/mime_parser.php differ
Only in hesk325/inc/mail: smtp.php
Only in hesk331/inc: mfa_functions.inc.php
Files hesk325/inc/pipe_functions.inc.php and hesk331/inc/pipe_functions.inc.php differ
Files hesk325/inc/posting_functions.inc.php and hesk331/inc/posting_functions.inc.php differ
Files hesk325/inc/print_tickets.inc.php and hesk331/inc/print_tickets.inc.php differ
Files hesk325/inc/profile_functions.inc.php and hesk331/inc/profile_functions.inc.php differ
Files hesk325/inc/setup_functions.inc.php and hesk331/inc/setup_functions.inc.php differ
Files hesk325/inc/show_admin_nav.inc.php and hesk331/inc/show_admin_nav.inc.php differ
Files hesk325/inc/show_search_form.inc.php and hesk331/inc/show_search_form.inc.php differ
Files hesk325/inc/statuses.inc.php and hesk331/inc/statuses.inc.php differ
Files hesk325/inc/ticket_list.inc.php and hesk331/inc/ticket_list.inc.php differ
Only in hesk325/inc/tiny_mce: 5.10.2
Only in hesk331/inc/tiny_mce: 5.10.5
Files hesk325/inc/tiny_mce/tinymce.inc.php and hesk331/inc/tiny_mce/tinymce.inc.php differ
Files hesk325/index.php and hesk331/index.php differ
Files hesk325/install/install_functions.inc.php and hesk331/install/install_functions.inc.php differ
Files hesk325/install/install.php and hesk331/install/install.php differ
Files hesk325/install/update.php and hesk331/install/update.php differ
Files hesk325/js/app.js and hesk331/js/app.js differ
Files hesk325/js/app.min.js and hesk331/js/app.min.js differ
Only in hesk331/js: dropzone.min.js
Files hesk325/js/hesk_javascript.js and hesk331/js/hesk_javascript.js differ
Files hesk325/js/hesk_javascript.min.js and hesk331/js/hesk_javascript.min.js differ
Only in hesk331/language/en/emails: mfa_verification.txt
Only in hesk331/language/en/emails: new_ticket_by_staff.txt
Only in hesk331/language/en/emails: survey.txt
Files hesk325/language/en/help_files/email.html and hesk331/language/en/help_files/email.html differ
Files hesk325/language/en/help_files/helpdesk.html and hesk331/language/en/help_files/helpdesk.html differ
Files hesk325/language/en/help_files/misc.html and hesk331/language/en/help_files/misc.html differ
Files hesk325/language/en/help_files/ticket_list.html and hesk331/language/en/help_files/ticket_list.html differ
Only in hesk331/language/en: html_emails
Files hesk325/language/en/text.php and hesk331/language/en/text.php differ
Files hesk325/reply_ticket.php and hesk331/reply_ticket.php differ
Files hesk325/submit_ticket.php and hesk331/submit_ticket.php differ
Files hesk325/theme/hesk3/customer/create-ticket/category-select.php and hesk331/theme/hesk3/customer/create-ticket/category-select.php differ
Files hesk325/theme/hesk3/customer/create-ticket/create-ticket-confirmation.php and hesk331/theme/hesk3/customer/create-ticket/create-ticket-confirmation.php differ
Files hesk325/theme/hesk3/customer/create-ticket/create-ticket.php and hesk331/theme/hesk3/customer/create-ticket/create-ticket.php differ
Files hesk325/theme/hesk3/customer/css/app.css and hesk331/theme/hesk3/customer/css/app.css differ
Files hesk325/theme/hesk3/customer/css/app.min.css and hesk331/theme/hesk3/customer/css/app.min.css differ
Only in hesk331/theme/hesk3/customer/css: dropzone.min.css
Files hesk325/theme/hesk3/customer/error.php and hesk331/theme/hesk3/customer/error.php differ
Files hesk325/theme/hesk3/customer/img/label-low.svg and hesk331/theme/hesk3/customer/img/label-low.svg differ
Files hesk325/theme/hesk3/customer/index.php and hesk331/theme/hesk3/customer/index.php differ
Only in hesk331/theme/hesk3/customer/js: dropzone.min.js
Files hesk325/theme/hesk3/customer/js/hesk_functions.js and hesk331/theme/hesk3/customer/js/hesk_functions.js differ
Files hesk325/theme/hesk3/customer/knowledgebase/search-results.php and hesk331/theme/hesk3/customer/knowledgebase/search-results.php differ
Files hesk325/theme/hesk3/customer/knowledgebase/view-article.php and hesk331/theme/hesk3/customer/knowledgebase/view-article.php differ
Files hesk325/theme/hesk3/customer/knowledgebase/view-category.php and hesk331/theme/hesk3/customer/knowledgebase/view-category.php differ
Files hesk325/theme/hesk3/customer/maintenance.php and hesk331/theme/hesk3/customer/maintenance.php differ
Only in hesk331/theme/hesk3/customer/util: attachments.php
Files hesk325/theme/hesk3/customer/util/custom-fields.php and hesk331/theme/hesk3/customer/util/custom-fields.php differ
Files hesk325/theme/hesk3/customer/view-ticket/form.php and hesk331/theme/hesk3/customer/view-ticket/form.php differ
Files hesk325/theme/hesk3/customer/view-ticket/partial/add-reply.php and hesk331/theme/hesk3/customer/view-ticket/partial/add-reply.php differ
Files hesk325/theme/hesk3/customer/view-ticket/view-ticket.php and hesk331/theme/hesk3/customer/view-ticket/view-ticket.php differ
Only in hesk331/: upload_attachment.php
Only in hesk331/: vendor

Looks like this list could be different between each version, though. The upgrade guide provided by HESK (look under "Upgrade from old HESK version") will tell you all the files need to be replaced when upgrading, but you can probably get away with replacing only the files that are different and removing those that are unique to the old version (diff -qr dir1/ dir2/ should tell you which files those are).

As for the file permissions, looks like you only need to give read/write permissions to three locations (according to HESK UI):

/hesk_settings.inc.php
/attachments
/cache

Hope this helps.

Hi @luketainton. I'd like to second @I-am-Kamino's request, and add some info. Hopefully this wall of text will be of some help to you both.

The issue

I'm experiencing similar issues to @I-am-Kamino. If, for some reason, I have to recreate the HESK container, it forces me to go through the install process again. Only it can't complete it, since the database already exists. Updating doesn't work either, because the script responsible for updating refers to the default HESK login values for the already-existing database. It then errors out with the following message if you've changed any of those values (which you should):
Fatal error: Uncaught mysqli_sql_exception: No such file or directory in /srv/install/install_functions.inc.php:105 Stack trace: #0 /srv/install/install_functions.inc.php(105): mysqli_connect('localhost', 'test', 'test', 'hesk') #1 /srv/install/update.php(87): hesk_iTestDatabaseConnection(true) #2 {main} thrown in /srv/install/install_functions.inc.php on line 105

@Gitoffomalawn I still catch the "Fatal error: " exception with your docker compose file. [UPDATE ] I guess I had to connect to the internal ip of mariadb container. Default "localhost" wouldn't do it. [/UPDATE] and also this: Deprecated: DateTime::__construct(): Passing null to parameter #1 ($datetime) of type string is deprecated in /srv/inc/setup_functions.inc.php on line 86

Any ideas?

@Gitoffomalawn I still catch the "Fatal error: " exception with your docker compose file. [UPDATE ] I guess I had to connect to the internal ip of mariadb container. Default "localhost" wouldn't do it. [/UPDATE]

Well, yes. The HESK app and the database are two separate containers. This means that inputting localhost as the database location in the HESK app will attempt to find that database within the HESK app container, which isn't where it is. You need to point it to the other container for it to work.

However, you don't necessarily need to use the IP address to point to the database. You can input db (or whatever you chose as service name for the database in docker-compose.yml), which would use Docker's built-in DNS to resolve to the correct container. This prevents your installation from getting borked should those internal IP address assignments ever change for whatever reason.

and also this: Deprecated: DateTime::__construct(): Passing null to parameter #1 ($datetime) of type string is deprecated in /srv/inc/setup_functions.inc.php on line 86

Any ideas?

Yes, I addressed this issue in #11. According to @luketainton, this is an issue with HESK itself which he cannot resolve as he doesn't maintain the application. In any case, this error doesn't appear to affect the installation process or how HESK functions in any way that I've encountered so far. Seems to me you can safely ignore this error.

@Gitoffomalawn thank you. Makes sense.

Hey all, apologies it's taken me so long. I have an idea that could work here:

Deploy temporary container
Run docker cp hesk:/srv/hesk_settings.inc.php hesk_settings.inc.php
Destroy temporary container
Update docker-compose.yml to volume mount hesk_settings.inc.php on the host to /srv/hesk_settings.inc.php
Ensure the container has write permissions to this file
Perform the installation

Hey all, apologies it's taken me so long. I have an idea that could work here:

1. Deploy temporary container

2. Run `docker cp hesk:/srv/hesk_settings.inc.php hesk_settings.inc.php`

3. Destroy temporary container

4. Update `docker-compose.yml` to volume mount `hesk_settings.inc.php` on the host to `/srv/hesk_settings.inc.php`

5. Ensure the container has write permissions to this file

6. Perform the installation

I would think that one would also want to do this with the /srv/attachments and /srv/cache folder to prevent loosing attachments on update

luketainton / hesk-docker