Slow performance due to persistence

ericp96 commented 5 years ago

Persisting the root www folder leads to slow performance. This is in part due to the persistence of cache and logs under the www folder. It would be ideal if we could specify the directories where the caching and logging are stored as environment variables and/or if a preferred mountpoint other than the root www folder is provided.

--- Want to back this issue? **[Post a bounty on it!](https://app.bountysource.com/issues/73085340-slow-performance-due-to-persistence?utm_campaign=plugin&utm_content=tracker%2F20392502&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://app.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F20392502&utm_medium=issues&utm_source=github).

luizeof commented 5 years ago

You can configure Mautic cache and logs path on Mautic Settings

download (1)

ericp96 commented 5 years ago

I did that, however performance is still not as great as when I just mounted the config directory. It seems like it would be worthwhile to note the directories that should be persisted and which ones would be fine to omit?

TheBags commented 5 years ago

I agree with @ericp96. It would be better if I can choose path for cache and logs during first deploy

aspiers commented 5 years ago

@ericp96 Please could you clarify why persisting cache and logs under /var/www/html makes things slow? Are you saying that persisting somewhere else would be faster? Or that they should be put on tmpfs or some other RAM-backed filesystem? Or something else? Sorry if these are stupid questions but I'm not understanding this issue fully yet.

ericp96 commented 5 years ago

Yup, persisting the html folder to a standard disk caused the entire site to be slow. I'm guessing if I choose to persist cache to a ssd it would be quicker. While I realize there may be a few locations, my personal suggestion would be to identify the folders that contain information that should be persisted instead of the entire mautic install. That would force people to be explicit in how they choose to backup different portions of the install. It would also allow for multiple installs to access the same copy of the cache in kubernetes.

luizeof commented 5 years ago

@ericp96 putting www folder on SDD is slow? I Didn't find the point ....

Modifying Mautic following the tips you gave seems a good idea, but there is a problem and I wanted to see the solution with you: How would I execute the migrations and update the schema? would it be in the entrypoint?

ericp96 commented 5 years ago

@luizeof Creating a persistent volume claim on a standard disk for the www folder caused a substantial degradation in performance. Also, there are some issues in mounting the www folder:

Injecting custom code requires you download a full install of Mautic and then replacing only the files you are concerned with (e.g. config files or theme files)
Switching between versions of Mautic while persisting your configuration is currently impossible. This has been a headache for me because I've encountered subtle differences in compatibility between different InnoDB database engines and specific versions of Mautic
The image version does not actually reflect the installed version when you persist the www folder. As a result, the Mautic image is more representative of the Mautic runtime environment with whatever code you've previously persisted.

As far as it goes with updating schema and migrations: I would advocate for adherence to the Single Responsibility Prinicple. Upgrading should probably NOT be included in the entrypoint.d script, but instead should be a different entrypoint entirely. In docker world it could be run once from the command line. From Kubernetes world it could be included as an init container in a deployment. I'm a little uncertain how Mautic modifies configuration files, however it would be nice if we could make this work with Kubernetes configmaps where appropriate.

aspiers commented 5 years ago

@ericp96 Sorry, I'm still a little confused what this issue is about. A person deploying Mautic via Docker has multiple choices to make:

whether the Mautic container is on standard disk or SSD
whether the cache goes inside the container (i.e. no separate volume), or in a volume on standard disk, or in a volume on SSD
whether the logs goes inside the container (i.e. no separate volume), or in a volume on standard disk, or in a volume on SSD

And of course we all know that standard disk is slower than SSD, so the more that gets put on standard disk, the slower things will go. But that is a choice for the person deploying Mautic, and AFAICS nothing to do with this repository.

So I'm confused why you wrote "Persisting the root www folder leads to slow performance"? If Mautic was deployed in a container on SSD with no volume, then why would redeploying it with some files split out into a separate volume on the same SSD make it significantly faster or slower?

TheBags commented 5 years ago

@aspiers I can made a personal example. We're deploing on Amazon's K8s. The persistem volume /var/www/html is on an "efs" filesystem because cost reduction. This filesystem is really slow, so we had to change the cache from /var/www/html/app/cache to /tmp/cache so to have the cache on the pod/container instead of onto the persistent volume. This operation can be done "only" after the initial configuration process. I think (please @ericp96 let me know if I understand correctly) an env variable like MAUTIC_CACHE it should be inserted so that the parameter can be valued when creating the pod / container. My 5 cents. TheBags

aspiers commented 5 years ago

Yes thanks, this is a good example of a scenario where the wrong backing storage can harm performance. But it's nothing to do with persistence per se. It would also be slow if you deployed the container on efs without any volume. So it does not prove that persistence in general causes slow performance, so I don't understand why this issue's title is claiming that.

ericp96 commented 5 years ago

@aspiers As a user of this image:

I don't care if my cache or the mautic install persists I do care that my configuration persists

Given my use case, I would like to not pay for a premium ssd. Since I didn't understand the mautic architecture included a built in cache

aspiers commented 5 years ago

@ericp96 I agree 100% with that use case, and I also agree that this repo needs to handle volumes better for sure, to allow users flexibility in their choice of backing storage for code vs. config vs. cache vs. logs vs. anything else.

But I still think it's incorrect for this issue's title to say "slow performance due to persistence". AFAICS there's no inherent poor performance when using volumes.

MikeDaniel18 commented 5 years ago

There is an issue on Mac with docker where mounted volumes take a massive performance hit. Unfortunately it's not going anywhere anytime soon. I think one issue is that these Docker containers are not stateless in the slightest. Configs I can understand using a volume for, but having the entire app in a volume seems like bad practice. I think this repo should be striving to have the code pre-compiled and ready to run, within the container. That said, I appreciate that I don't know the limitations of the application and why it may need a compilation stage.

aspiers commented 5 years ago

There is an issue on Mac with docker where mounted volumes take a massive performance hit.

Any chance of a URL? However, that still wouldn't really justify the title given to this issue, because the term "persistence" generally refers to any non-volatile storage, not just docker volumes.

I agree that the application code should be in a prebuilt container image. IIUC the challenge is that currently Mautic puts not only the application code under /var/www/html, but also the log files, cache, config, etc. This is discussed extensively e.g. in #2 and #90 but I see you have already found the latter at least. I think that this situation won't improve until these elements are correctly separated out.

BTW Mautic is PHP which is an interpreted language, not a compiled one. So AFAICS it doesn't really make sense to talk about a compilation stage for Mautic, only the container build stage.

ivanmonteiro commented 2 years ago

I also experienced very slow performance due to a slow disk on a Google Cloud free tier instance (e2-micro, 1gb ram). The free tier offers only conventional hard disks with very limited performance. I ended up mounting the cache folder with a tmpfs volume and it did improve performance a lot. The response times greatly reduced, and navigating the application seems much quicker. Navigating between contacts/emails on Mautic dashboard took 3-5 seconds, now it loads under 0,5 second. But on other VM instance with better cpu and disk It didn't make any differece. I think that if disk is a bottleneck, mounting the cache folder as tmpfs did help improving performance.

Here is the docke-compose.yml file I used, if someone is in the same situation:

version: '3.1'
services:
  mautic:
    container_name: mautic
    image: mautic/mautic:v4-fpm
    restart: unless-stopped
    env_file: ./mautic.env
    volumes:
      - ./public/:/var/www/html/
    tmpfs:
      - /var/www/html/var/cache/prod:uid=33,gid=33
    ports:
      - 9001:9000

mautic / docker-mautic

Slow performance due to persistence #100