Proposals (devops, installation, development, etc.)

[melck]

Hi,

I'm testing your pim product and wanted to share with you some ideas on your workflow (like Devops, CI / CD, good practices, etc). I find getting your product into production way too complicated. I think the development workflow could be simpler as well, but I focused on setting up production.

I think it would be a lot easier if javascript dependencies weren't needed in production. I can help you put that in place. I started working on a PR, but it makes too much of a change in how you operate. So I am coming here to discuss this with you first.

We could start with minimal changes that would simplify the documentation for installation and ensure more stability for each release. The objectives of these changes would be to not have to install nodejs in production to compile the assets. It will also significantly reduce the size of your Docker images.

List of minimal changes

- Before tagging a new version : - Make commands : - `bin/console pim:installer:assets --symlink` - `yarn run --cwd vendor/akeneo/pim-community-dev/akeneo-design-system install --frozen-lock` - `yarn run --cwd vendor/akeneo/pim-community-dev/akeneo-design-system lib:build` - `yarn run install --frozen-lock` - `yarn run webpack` - `yarn run less` - `yarn run update-extensions` - Add compiled files : - git add -f public/bundles public/css public/dist public/js - git commit -m "Publish assets" - git push origin ${version}

List of ideas to go futher (no sure if possible)

- Repo akeneo-design-system : - Don't extract from akeneo/pim-community-dev - Optionnal : Ensure strong convention on commit (ex: [Conventionnal commit](https://www.conventionalcommits.org/en/v1.0.0/)) - Change CI / CD when deliver a new relase : - Compile sources - Increment version - Optionnal : Generate CHANGELOG from commit message (ex: [commitizen changelog generator](https://github.com/commitizen/cz-cli)) - Git Add force files to bypass gitinore - Git remove uncessary files (like from ci / cd) - Git commit for release - Tag && push - npm publish - Repo pim-community-standard : - I don't understand why you need a dev and standard repository ? It would much simpler if you delete this repo and integrate project generator directly in pim-community-dev - Repo pim-community-dev : - If you integreate project generator (cf. question repo pim-community-standard ), rename it to pim-ce (for community edition, i found that more logic with your enterprise edition) - I think usage of make (and makefile) make your project to complex and add a unnecessary dependancies : - Remove all makefile - Create shell scripts to install in production (one per OS) in a scripts folder - Force only containers for development : - Move base dockerfile to docker folder for pim - Change you `docker-compose.yml` file to integrate only ready production containers - Create a file `docker-compose.override.yml.exemple` with development containers, add ignore file `docker-compose.override.yml` and document in contributing section. - Make entrypoints shell script to integrate command from makefile : - for php image : - start script by waiting avaibility of mysql, elasticsearch, etc. - capture command if provided ex for commands : - `migrate` : For database schema creation or upgrade - `start` : To launch production apache - `dev` : To launch dev server - `sh` : Start a shell to debug containers - commands for testing - Change php dockerfile production image into [multistage build](https://docs.docker.com/develop/develop-images/multistage-build/) : - Using node to compile assets as build-js - Using debian to install php deps as php - Using php to copy compiled files by build-js - Using php to install appliation dependancies, prepare assets - Change CI / CD when deliver a new relase : - Compile sources - Git Add force files to bypass gitinore - Git remove uncessary files (like from ci / cd) - Git commit for release - Tag && push - Managing all synfony assets at once (So you will have only one way to manage them and avoid complexity with several steps): - Study migration for usage of [Symfony Encore](https://github.com/symfony/webpack-encore) - Integrate all your assets : images, less compilation, js and ts compilation. - Documentation : - Move all development reference of product installation to contributing section - Rework production installation to be much simpler (without building js dependencies) and links scripts to install for each OS supported. - Create release cycle with when maintenance end for each versions - Want to do some Kubernetes in future for more visibility ? List of my advices : - Change your base image to bitnami and apply their philosophy - Add a separate repository or folder to create helm charts to install you product in production ready Kubernetes - Contact Bitnami to publish docker image / helm charts (don't know real process for that)

Exemple of my work to build a production ready Akeneo PIM CE

Alpine OS and Nginx UNIT have been used because i like them. Dockerfile (multi stage) ```dockerfile FROM alpine:3.14 as php ARG PHP_MEMORY_LIMIT="1G" ARG PHP_DATE_TIMEZONE="Europe/Paris" RUN set -ex \ && apk add --no-cache --virtual .run-deps \ bash \ gettext \ netcat-openbsd \ curl \ unit \ unit-php7 \ composer \ php7-bcmath \ php7-curl \ php7-fileinfo \ php7-fpm \ php7-gd \ php7-intl \ php7-pdo_mysql \ php7-xml \ php7-xmlwriter \ php7-xmlreader \ php7-simplexml \ php7-zip \ php7-exif \ php7-opcache \ php7-tokenizer \ php7-zip \ php7-mbstring \ php7-pecl-apcu \ php7-pecl-imagick \ php7-pecl-memcached \ aspell \ aspell-en \ aspell-fr \ && sed -i -E \ -e "s/^;?(memory_limit =).*/\1 ${PHP_MEMORY_LIMIT}/" \ -e "s|^;?(date.timezone =).*|\1 ${PHP_DATE_TIMEZONE}|" \ /etc/php7/php.ini ARG AKENEO_PIM_VERSION="5.0.39" ENV AKENEO_PIM_VERSION="${AKENEO_PIM_VERSION}" RUN COMPOSER_MEMORY_LIMIT=-1 composer create-project --no-cache akeneo/pim-community-standard /app "${AKENEO_PIM_VERSION}" WORKDIR /app RUN set -ex \ && bin/console pim:installer:assets --symlink --clean \ && bin/console oro:translation:dump en \ && bin/console oro:translation:dump fr FROM node:lts-alpine AS build-js-deps ENV NODE_ENVIRONMENT=production ARG AKENEO_DESIGN_SYSTEM_PATH="vendor/akeneo/pim-community-dev/akeneo-design-system" COPY --from=php "/app/${AKENEO_DESIGN_SYSTEM_PATH}" /app WORKDIR "/app" RUN set -ex \ && yarn --frozen-lockfile \ && yarn run lib:build FROM node:lts-alpine AS build-js ARG AKENEO_DESIGN_SYSTEM_PATH="vendor/akeneo/pim-community-dev/akeneo-design-system" COPY --from=php "/app" "/app" COPY --from=build-js-deps "/app/lib" "/app/${AKENEO_DESIGN_SYSTEM_PATH}/lib" WORKDIR "/app" RUN set -ex \ && yarn install --frozen-lockfile \ && yarn run webpack \ && yarn run update-extensions \ && yarn run less FROM php COPY --from=build-js /app/public/dist /app/public/dist COPY --from=build-js /app/public/js/extensions.json /app/public/js/extensions.json COPY --from=build-js /app/public/css/pim.css /app/public/css/pim.css ARG AKENEO_PIM_UID="2000" ARG AKENEO_PIM_GID="2000" RUN set -ex \ && mkdir -p public/uploads \ && addgroup \ -g "${AKENEO_PIM_GID}" \ akeneo-pim \ && adduser \ -H \ -h /app/var \ -u "${AKENEO_PIM_UID}" \ -G akeneo-pim \ -s /bin/bash \ -D akeneo-pim \ && chown -R "${AKENEO_PIM_UID}:${AKENEO_PIM_GID}" /app/public/uploads /app/var VOLUME [ "/app/var/log", "/app/var/uploads", "/app/public/uploads"] COPY docker-entrypoint.sh /usr/local/bin/entrypoint RUN set -ex \ && chmod 0755 /usr/local/bin/entrypoint COPY unit.json /var/lib/unit/conf.json EXPOSE 9200 ENTRYPOINT [ "entrypoint" ] CMD [ "start" ] ``` Unit.json ```json { "listeners": { "*:9200": { "pass": "routes" } }, "applications": { "akeneo-pim": { "type": "php", "processes": 20, "user": "akeneo-pim", "group": "akeneo-pim", "root": "/app/public", "script": "index.php", "options": { "file": "/etc/php.ini", "user": { "display_errors": "0" } } } }, "routes": [ { "action": { "share": "/app/public", "fallback": { "pass": "applications/akeneo-pim" } } } ] } ``` Entrypoint script ```shell #!/usr/bin/env bash set -e DB_HOST=${DB_HOST:-db} DB_PORT=${DB_PORT:-3306} ES_FIRST_HOST=$(echo "${APP_INDEX_HOSTS}" | cut -f1 -d ',' | cut -f1 -d ':') ES_FIRST_PORT=$(echo "${APP_INDEX_HOSTS}" | cut -f1 -d ',' | cut -f2 -d ':') echoerr() { echo "$@" 1>&2; } ACTION=$1 shift 1 wait_host_port() { local \ WAIT_NAME=$1 \ WAIT_HOST=$2 \ WAIT_PORT=$3 echoerr "wait-for-${WAIT_NAME}: waiting for ${WAIT_HOST}:${WAIT_PORT}" timeout -s TERM 15 bash </dev/null 2>&1; do sleep 1; done; EOT RESULT=$? if [ $RESULT -eq 0 ]; then # sleep to avaid service is starting sleep 5 echoerr "wait-for-${WAIT_NAME}: done" else echoerr "wait-for-${WAIT_NAME}: timeout out after 15 seconds waiting for ${DB_HOST}:${DB_PORT}" fi } wait_host_port "db" "${DB_HOST}" "${DB_PORT}" wait_host_port "elasticsearch" "${ES_FIRST_HOST}" "${ES_FIRST_PORT}" case "${ACTION}" in install-db) exec bin/console pim:installer:db --catalog vendor/akeneo/pim-community-dev/src/Akeneo/Platform/Bundle/InstallerBundle/Resources/fixtures/minimal ;; job) exec bin/console akeneo:batch:job-queue-consumer-daemon ;; messenger) exec bin/console messenger:consume webhook ;; # migrate) # ;; start) exec unitd --no-daemon --log /dev/stdout "$@" ;; *) exec "${ACTION}" "$@" ;; esac exit 1 ```

PS: I am a computer enthusiast. I work as devops for a hosting company since 5 years for internal products and customers all over the world. I am also a certified Kubernetes administrator. I have a lot of esteem for Akeneo (I live in Nantes). In the past, I have also enjoyed working with two of your employees (Hi @mmetayer and @jmleroux).

[BitOne]

Hello Melck,

Thanks for your message!

First of all, JavaScript build dependencies are not needed in production. For example, in our Enterprise Edition Serenity production environment, we don't have NodeJS.

JS assets (as well as PHP assets) are compiled during our built process. And we use a multistage Docker alongside the builder pattern in order to have only the resulting assets in the final production docker image, which we run on our Kubernetes environment.

So I'm not sure what you would like to do exactly.

Is it about providing documentation for using the Community Edition in production?

And indeed right now our documentation is focused only on the development environment. And the reason behind that is that there are too many ways to use Akeneo CE in production.

From traditional server-based hosting to serverless ones, as well as running it in a Kubernetes cluster. And those are only a few examples, and I'm not even digging into the variation in MySQL and Elasticsearch installation (replication or not, Es cluster definition, way to assign roles to ES clusters, etc...).

When you think about it, it's usually much simpler with dev environments, and that's why we provide documentation for non-Docker-based and Docker-based development environments. And we are obviously not exhaustive with that either.

If you would like to write documentation or a tutorial on how to run Akeneo PIM CE in a production Kubernetes environment, we could definitely link it from our official documentation. And I think that would be very beneficial for a lot of people.

But for the reason I explained above, we would not add production documentation inside our official documentation. And we could maybe explain a bit what I said above in our documentation, as it could help make things clearer.

Maybe I'm misinterpreting what you want to do, so please let me know.

But for sure, some stuff you considered as necessary, like building the assets in production, are clearly not. And are not our recommendation either.

[melck]

Thank you for your answers. That explains a lot.

I wanted to test your product. Before going into a enterprise edition, I wanted to see if the opensource version was sufficient to start.

When I started looking to do an install it turned out to take a lot longer than I expected. I wanted to give my feedback. At first I was just thinking of proposing changes in the documentation, but I realized that it would be easier to modify the behavior of opensource releases.

Currently opensource releases do not contain compiled assets. We have to build them before we deploy. This adds considerable time to understanding and deployment. The documentation does not have a section on a production image, only how to build it.

The rest of my proposals were to go in this direction. To facilitate the understanding of the project and to be able to participate in it more quickly, but indeed everything depends on your objectives with this opensource project.

[BitOne]

Okay, thanks for the clarification.

We don't provide release builds for our PIM Community Edition, we only provide source code.

Providing release packages would mean putting in place a specific build infrastructure, with a non-negligible maintenance cost.

And there's a high chance that the provided built packages won't work with the targeted environment. For example, on the dev side, you will need a specific built of JavaScript in order to debug it. And this package cannot be the same for production too.

Moreover, even if we provide production built of the JS side, we know a lot of people customize the code, so they will need to rebuild the JavaScript itself anyway.

So providing source-code-only releases is certainly a good compromise in terms of flexibility and cost.

By the way, I re-did some tests, and going from scratch to a working PIM in production mode on my laptop took exactly 13 minutes to have a full built PIM (3 minutes for composer and 10 minutes to build the rest of the application, including around 5 minutes of JS building). While not exactly fast, this is not crazy long, especially as it needs to be done only once. So the ratio cost vs gain of providing already built JS packages is not really worth it.

On the broader subject of providing some documentation or tutorial on running PIM Community Edition on Kubernetes, building a production image for it, I do still think this could very useful for the community. And as I said, we could put a link into our documentation to this content if you want to put it together. But as I explained above, we won't provide a production image because the one size fits all approach would cost a lot and provide a very small number of people.

[melck]

I understand your reasons better, although in my case I would rather follow the release of an open source product version, not customize it, and participate in the patch, features, etc. directly on github. I also think that it is better to do one version upstream than several downstream versions.

I didn't take the cost of construction and maintenance on purpose, as it's really a question of how it's done, what is used, etc. and this part is not really visible.

I am really surprised by the user customizing js code. From my point of view, it is to be avoided and rather encouraged to participate in creating settings to customize. Do you have any examples of what users are customizing?

For the installation time, I wasn't thinking of the installation itself, but the time to understand the documentation, the time to understand what is require to compile the assets, the time to understand what Makefile does (I prefer to avoid using a makefile a system without understanding what it is doing). In my opinion, it takes too much skills to figure all this out, but it could be much clearer. Maybe this is what you want.

For the Kubernetes installation, I will think about it, it will depend on the use of the product.

[BitOne]

I am really surprised by the user customizing js code. From my point of view, it is to be avoided and rather encouraged to participate in creating settings to customize. Do you have any examples of what users are customizing?

It happens a lot. People are adding specific features for their needs, with new screens, customizing existing features, etc...

For the installation time, I wasn't thinking of the installation itself, but the time to understand the documentation, the time to understand what is require to compile the assets, the time to understand what Makefile does (I prefer to avoid using a makefile a system without understanding what it is doing). In my opinion, it takes too much skills to figure all this out, but it could be much clearer. Maybe this is what you want.

Using a pre-built package would be akin to using something without understanding how it has been built. So the complexity to understand all of that is still the same ;)

For the Kubernetes installation, I will think about it, it will depend on the use of the product.

Okay, let us know if you need anything or any clarification on how and why things are done the way they are.

[melck]

Thanks for you answers, I don't necessarily agree, but I understand your reasons and it's very instructive.

Using a pre-built package would be akin to using something without understanding how it has been built. So the complexity to understand all of that is still the same ;)

I remain convinced that it's a brake on the adoption of the product. From my perspective, people will appreciate a product more if the learning curve is steeper and participate more easily in its evolution.

For example where I work, the start of the product test was very successful, we started with your ova. However, we have fairly strict standards that require us to install and host the products. We cannot take the enterprise edition directly, because to justify the cost we need to show what the product brings. So we start with the community edition and when we need to go further, it will be easier to validate transition to EE. When we started looking at how to install and update it, there was a deadlock with the documentation.

Regarding the documentation, I have the following questions on maintenance of product:

• How to make the changes? The documentation explains the raptriement of the new code but not of the evolutions of the diagram of the base for ex. Is it enough to apply classic symfony commands ?
• Is it possible to migrate from CE to EE? I did not find any mention of this in the documentation.
• Do you want to keep documentation like this, or can we do some PR to clarify these points ?

[Thijzer]

I would honestly love some documentation for a kubernetes cluster running a docker or containerd image. I have been doing akeneo builds with Jenkins since akeneo 2.0 and it would be immensely helpful to have a better insight on this. Especially what your choices would be around the file storage options and a managed MYSQL or not. Pros / cons.