Docker image with node 18.x core dumped

gray380 commented 5 months ago

Hello,

Docker service with 3.1 image works:

[info] Node-RED version: v3.1.3
[info] Node.js  version: v16.20.2
[info] Linux 4.18.0-425.19.2.el8_7.x86_64 x64 LE

Docker service with 3.1.3-18 (3.0.2-18 as well) core dumped:

./entrypoint.sh: line 14:     7 Illegal instruction     (core dumped) /usr/local/bin/node $NODE_OPTIONS node_modules/node-red/red.js --userDir /data $FLOWS "${@}"

What are the steps to reproduce?

docker stack deploy --resolve-image=always --with-registry-auth --compose-file docker-compose.yml stackname

What happens?

core dumped

What do you expect to happen?

service up and running

Please tell us about your environment:

[ ] Platform/OS:

cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

[ ] Browser:

If you are a Portainer user, please attach a screenshot of your container details.

Containers -> Click on the node-red container -> Scroll to Container Details and take a screenshot

If you are a command line user please execute the following command and attach the log file.
```
docker inspect <container_id> > container.log
```

hardillb commented 5 months ago

We are going to need some more details here, to start with:

Can we get more of the log running up to the failure
What nodes do you have installed
What version of docker are you using

hardillb commented 5 months ago

Also what is the output of the following command:

sudo scmp_sys_resolver -a arm clock_gettime64

gray380 commented 5 months ago

I've changed the package.json in the /data from:

{
    "name": "node-red-project",
    "description": "A Node-RED Project",
    "version": "0.0.1",
    "private": true
}

to the:

{
    "name": "node-red-docker",
    "version": "3.1.3",
    "description": "Low-code programming for event-driven applications",
    "homepage": "http://nodered.org",
    "license": "Apache-2.0",
    "repository": {
        "type": "git",
        "url": "https://github.com/node-red/node-red-docker.git"
    },
    "main": "node_modules/node-red/red/red.js",
    "scripts": {
        "start": "node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS",
        "debug": "node --inspect=0.0.0.0:9229 $NODE_OPTIONS node_modules/node-red/red.js $FLOWS",
        "debug_brk": "node --inspect=0.0.0.0:9229 --inspect-brk $NODE_OPTIONS node_modules/node-red/red.js $FLOWS"
    },
    "dependencies": {
        "node-red": "3.1.3",
        "node-red-contrib-mssql-plus": "~0.10.2",
        "node-red-contrib-redis": "~1.3.9",
        "node-red-contrib-uuid": "~0.0.4"
    },
    "engines": {
        "node": ">=14"
    }
}

and service has started.

To be honest I do not recall where package.json comes from.

hardillb commented 5 months ago

Just changing the package.json should not have caused node-red to fail to start.

Also node-red should not be listed in the dependencies for a package.json in /data

gray380 commented 5 months ago

Also what is the output of the following command:
sudo scmp_sys_resolver -a arm clock_gettime64

no such command

gray380 commented 5 months ago

Just changing the package.json should not have caused node-red to fail to start.

miracle ... That is the only change I've made.

Also node-red should not be listed in the dependencies for a package.json in /data

I used this example.

hardillb commented 5 months ago

That file is for building the container, not for the userDir (/data)

Please supply the information I asked for re the versions and the docker-compose file you used

gray380 commented 5 months ago

We are going to need some more details here, to start with:
* Can we get more of the log running up to the failure
docker service logs shows the only string: ./entrypoint.sh: line 14: 7 Illegal instruction (core dumped) /usr/local/bin/node $NODE_OPTIONS node_modules/node-red/red.js --userDir /data $FLOWS "${@}"

What nodes do you have installed

What nodes are you asking about? Docker swarm's or something else?
* What version of docker are you using
23.0.5 and 24.0.6

docker-compose.yml

version: '3.7'
services:
  nodered_prod_formnext:
    image: registry.api.lan:8083/common/common.nodered:3.1.3-18
    user: 988:988
    hostname: nodered_prod_formnext
    networks:
      docker.dc:
        aliases:
          - nodered_prod_formnext.docker.dc
      traefik.dc:
        aliases:
          - nodered_prod_formnext.traefik.dc
    dns_search:
      - traefik.dc
      - docker.dc
    environment:
      - TZ=Europe/Kyiv
      - LANG=uk_UA.UTF-8
      - LC_ALL=uk_UA.UTF-8
      - LANGUAGE=uk_UA:uk
      - ASPNETCORE_ENVIRONMENT=Prod
      - BC=11
    volumes:
      - ./volumes/data:/data
    healthcheck:
      test: ["CMD-SHELL", "node /healthcheck.js"]
      interval: 30s
      start_period: 10s
      timeout: 10s
      retries: 3
    deploy:
      labels:
      - traefik.enable=true
      - traefik.docker.network=traefik
      - traefik.http.services.nodered_prod_formnext.loadbalancer.server.port=1880
      - traefik.http.routers.nodered_prod_formnext.entrypoints=websecure
      - traefik.http.routers.nodered_prod_formnext.tls=true
      - traefik.http.routers.nodered_prod_formnext.rule=Host(`formnext.cdc.lan`)
      - traefik.http.routers.nodered_prod_formnext.middlewares=redirect-regex-main
      - "traefik.http.middlewares.redirect-regex-main.redirectregex.regex=^https:\\/\\/([^\\/]+)\\/?$$"
      - "traefik.http.middlewares.redirect-regex-main.redirectregex.replacement=https://$$1/main"
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.worker==true
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
    logging:
      driver: "gelf"
      options:
        gelf-address: "udp://graylog.lan:12201"
        tag: "common:nodered_prod_formnext:prod"
networks:
  docker.dc:
    external: true
    name: microservices
  traefik.dc:
    external: true
    name: traefik

Dockerfile:

ARG VERSION
FROM nodered/node-red:${VERSION}

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
ENV TZ=Europe/Kyiv

WORKDIR /usr/src/node-red

USER root

ADD CA/*.crt /usr/local/share/ca-certificates/
RUN apk update && apk upgrade && apk add ca-certificates && rm -rf /var/cache/apk/* \
    && update-ca-certificates \
    && npm install --unsafe-perm --no-update-notifier --no-fund --only=production \
    && npm install crypto -g

USER node-red

gray380 commented 5 months ago

That file is for building the container, not for the userDir (/data)

so, what is the correct way to set dependencies?

Please supply the information I asked for re the versions and the docker-compose file you used

hardillb commented 5 months ago

OK, so first you are not using the default image you are using a custom extended container you have built. So it is not the Node-RED images which are crashing but your extended container images, this would have been helpful to know upfront (this is why we ask for the docker inspect output, so we can see what container you are actually using).

The illegal instruction error implies that the container was built for a different target architecture than the one you are running on.

Did you do a multi arch manifest build and push that to your local container repository?
If you use the default Node-RED shipped image does it start?
If you build the container locally on the machine it will run on does it start?

As for including dependencies:

so, what is the correct way to set dependencies?

The file you pointed at lives in /usr/src/node-red this is where the code node-red app is installed in the container. If you edit that package.json to include dependencies then you need to rebuild the whole container. (which you could then use as a base for your extended container)

The other option is to add dependencies to the package.json in the /data volume and run npm install in that directory as part of you extended container. These will be copied to any empty volume mounted on /data when you start up the container for the first time.

At this point I'm not convinced there is a problem with Node-RED shipped containers and the problem is with your build/deployment environment.

gray380 commented 5 months ago

This is alittle bit weird. My image is based on the official image with added certificates and crypto npm package.

And yes, your images starts successfully:

docker run -it -v myNodeREDdata:/data --name mynodered nodered/node-red:3.1.3-18
30 Jan 16:55:44 - [info]

Welcome to Node-RED
===================

30 Jan 16:55:44 - [info] Node-RED version: v3.1.3
30 Jan 16:55:44 - [info] Node.js  version: v18.19.0
30 Jan 16:55:44 - [info] Linux 4.18.0-477.15.1.el8_8.x86_64 x64 LE
30 Jan 16:55:46 - [info] Loading palette nodes

dceejay commented 5 months ago

I think crypto indeed has a binary component so needs to be rebuilt for the correct architecture.

node-red / node-red-docker