Closed aklyachkin closed 3 years ago
Thanks for raising this issue. I would think that complication should be relatively straightforward. I'm not aware of any contributors with access to equipment, so testing, support and debugging would be a challenge.
Here are some of the component parts:
https://docs.openfaas.com/contributing/get-started/#openfaas
What kind of involvement or support are you looking for from the project? I noticed a few PRs and new Dockerfiles.
Are you hoping to upstream support to each component and then test everything together, or would it also work to prove it out in your pipeline and then report back on the issue?
Perhaps any issues or PRs could be linked back to this issue for tracking?
As for now I would like just to push necessary changes to support the platform a "clean" build as far as I can and as near to the original as it possible. That's why I made first PRs to some components used in OpenFaaS:
In my own repositories I have OpenFaaS and Faas-Netes for ppc64le:
I also compiled several docker images for ppc64le today which are available at https://hub.docker.com/u/powerlinux
The next steps:
If it works, it means for me that OpenFaaS can work on ppc64le.
The most difficult question is testing/support/debugging afterwards. I can do it only in my free time and as long as the project is interesting to me.
I think there are potentially up to 20 +/- projects (excluding templates) that you may need to rebuild for your architecture. See a complete list at: https://docs.openfaas.com/contributing/get-started/#openfaas
What the difference between s390x and ppc64le? I've had one of your colleagues send a PR to add a binary for s390x to the license-check component. Is this the same thing in other words, or are there two IBM platforms which we need to support upstream in each component?
The most difficult question is testing/support/debugging afterwards. I can do it only in my free time and as long as the project is interesting to me.
Are you with IBM? I see that there are patches / questions about these two IBM platforms coming in from around 3-4 people at the moment. I'm finding it confusing to track all these efforts and would like to know if they are co-ordinated, or if OpenFaaS is just gaining random traction inside a new community.
Durgadas Kamath @imdurgadas has also requested both of these IBM-specific architectures for license-check
.
I would suggest that you may also need faas-cli / watchdog / of-watchdog in addition to the server-side components.
How far do you want to go with the experiment before reviewing whether to submit upstream patches to every component? Is there a way or place where the OpenFaaS community can test the patches or test new, breaking changes against the ppc64le platform?
@aklyachkin see the binary release here that you requested: https://github.com/teamserverless/license-check/releases/tag/0.3.2
I'm not from IBM and I don't know anybody from IBM interesting in this project.
s390x and ppc64le are two different platforms. s390x is IBM Z mainframe hardware and ppc64le is little-endian Linux running on IBM Power Systems. I don't have access to s390x, so it is out of scope for me. I recently learned some people who are running Kubernetes on s390x, but they didn't show interest on OpenFaaS.
As for me I'm interested in bringing OpenFaaS server side on ppc64le. I understand I have to port a little bit more thing as I did before, but as for now it works:
[root@k8s yaml_ppc64le]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
172.16.200.139 Ready etcd,management,master,proxy,worker 4d11h v1.13.5+icp 172.16.200.139 <none> Red Hat Enterprise Linux 3.10.0-957.21.3.el7.ppc64le docker://18.6.2
[root@k8s yaml_ppc64le]# kubectl get pod -n openfaas
NAME READY STATUS RESTARTS AGE
alertmanager-554f5fdc78-hh2l5 1/1 Running 0 19m
basic-auth-plugin-6f7df46d79-tg5rl 1/1 Running 0 20m
faas-idler-858f9cb89b-fn2bq 1/1 Running 0 4m40s
gateway-d95cb88f7-fl2nv 2/2 Running 0 5m32s
nats-bbfdfbcc8-hzfjh 1/1 Running 0 20m
prometheus-7d4cfd954f-lj455 1/1 Running 0 11m
queue-worker-7df45bfb8d-95crg 1/1 Running 0 13m
All patches I did to get it working I will commit back to the project. It is not so much work.
faas-cli template store list
Congratulations on getting it working!
Once your patches are contributed back, who will be fixing, updating and maintaining support for ppc64le? And for how long?
I believe Rishi from the IBM enablement team was trying to get [OpenFaaS Cloud](https://github.com/openfaas/openfaas-cloud working) - which would of course also include OpenFaaS (this issue)
Pull request https://github.com/openfaas/faas/pull/1258
As you can see Dockerfile for gateway is absolute identical for amd64 and ppc64le.
I checked also faas-cli. There is no single change needed to compile it on ppc64le.
At least the most easiest function works:
[root@k8s faas-ppc64le]# faas-cli new hello-ppc64le --lang python-ppc64le
Folder: hello-ppc64le created.
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: hello-ppc64le
Stack file written: hello-ppc64le.yml
[root@k8s faas-ppc64le]# faas-cli build -f ./hello-ppc64le.yml
[0] > Building hello-ppc64le.
Clearing temporary build folder: ./build/hello-ppc64le/
Preparing ./hello-ppc64le/ ./build/hello-ppc64le//function
Building: k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest with python-ppc64le template. Please wait..
Sending build context to Docker daemon 8.192kB
Step 1/29 : FROM powerlinux/classic-watchdog:latest-dev-ppc64le as watchdog
---> b66ef1b27615
Step 2/29 : FROM ppc64le/python:2.7-alpine
---> 4e90e5cc7f1b
Step 3/29 : ARG ADDITIONAL_PACKAGE
---> Using cache
---> 64cd296363f3
Step 4/29 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
---> Using cache
---> ec95dc31b623
Step 5/29 : RUN chmod +x /usr/bin/fwatchdog
---> Using cache
---> 9ccc8d988ce1
Step 6/29 : RUN apk --no-cache add ca-certificates ${ADDITIONAL_PACKAGE}
---> Using cache
---> 6a40fb26f47e
Step 7/29 : RUN addgroup -S app && adduser app -S -G app
---> Using cache
---> c1c5a926ffc3
Step 8/29 : WORKDIR /home/app/
---> Using cache
---> 74823a8a0a07
Step 9/29 : COPY index.py .
---> Using cache
---> bf89a5a2faca
Step 10/29 : COPY requirements.txt .
---> Using cache
---> 9d767e38817b
Step 11/29 : RUN chown -R app /home/app && mkdir -p /home/app/python && chown -R app /home/app
---> Using cache
---> 2708aefcf526
Step 12/29 : USER app
---> Using cache
---> 37d2733ebe95
Step 13/29 : ENV PATH=$PATH:/home/app/.local/bin:/home/app/python/bin/
---> Using cache
---> 3255279c1e7c
Step 14/29 : ENV PYTHONPATH=$PYTHONPATH:/home/app/python
---> Using cache
---> e9d6901b8e1c
Step 15/29 : RUN pip install -r requirements.txt --target=/home/app/python
---> Using cache
---> 098aa3bc4736
Step 16/29 : RUN mkdir -p function
---> Using cache
---> 43ced72ed227
Step 17/29 : RUN touch ./function/__init__.py
---> Using cache
---> 4715f9604dd9
Step 18/29 : WORKDIR /home/app/function/
---> Using cache
---> ca004600b735
Step 19/29 : COPY function/requirements.txt .
---> Using cache
---> f485f5a48d77
Step 20/29 : RUN pip install -r requirements.txt --target=/home/app/python
---> Using cache
---> 74b4e29d13ed
Step 21/29 : WORKDIR /home/app/
---> Using cache
---> 3484604d606a
Step 22/29 : USER root
---> Using cache
---> f64bef9e3593
Step 23/29 : COPY function function
---> Using cache
---> 409e2055e1bc
Step 24/29 : RUN chown -R app:app ./ && chmod -R 777 /home/app/python
---> Using cache
---> d11b57289573
Step 25/29 : USER app
---> Using cache
---> 358dfb0099b7
Step 26/29 : ENV fprocess="python index.py"
---> Using cache
---> 6ae3f1fadb62
Step 27/29 : EXPOSE 8080
---> Using cache
---> 2d1fecd36bb1
Step 28/29 : HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1
---> Using cache
---> 10791fbccc6d
Step 29/29 : CMD ["fwatchdog"]
---> Using cache
---> 5ff3a8f97d4c
Successfully built 5ff3a8f97d4c
Successfully tagged k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest
Image: k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest built.
[0] < Building hello-ppc64le done.
[0] worker done.
[root@k8s faas-ppc64le]# docker push k8s.enfence.net:8500/openfaas-fn/hello-ppc64le:latest
The push refers to repository [k8s.enfence.net:8500/openfaas-fn/hello-ppc64le]
d6e3a21c5437: Pushed
4258813f7d04: Pushed
aa10c958c3c7: Pushed
9eabaf692b12: Pushed
614c5e3ae3ea: Pushed
6f0f83f8a964: Pushed
addbc69d004c: Pushed
9a518ec71d66: Pushed
fdd6c21b969c: Pushed
8851091ea059: Pushed
1b6665f8f6f8: Pushed
82f2ffdb0adc: Pushed
58cfb351121c: Pushed
b2f2b401a975: Pushed
5e2687363386: Pushed
a131a19cf682: Pushed
3533830e9fa9: Pushed
latest: digest: sha256:670429ea822bd578615f0d261a1f897154b5eae1cfb3fb3c8d3b50708861dca8 size: 4074
[root@k8s faas-ppc64le]# faas-cli deploy -f ./hello-ppc64le.yml
Deploying: hello-ppc64le.
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Deployed. 202 Accepted.
URL: http://k8s.enfence.net:31112/function/hello-ppc64le
[root@k8s faas-ppc64le]# faas-cli list -g $OPENFAAS_URL
Function Invocations Replicas
hello-ppc64le 0 1
[root@k8s faas-ppc64le]# echo hi | faas-cli invoke hello-ppc64le -g $OPENFAAS_URL
Hello from ppc64le! You said: hi
hi
Somehow I can only see up to Step 25/29 : USER app
on mobile
https://github.com/openfaas/faas/issues/1257#issuecomment-512201195
Any need for a dockerfile then?
Not really. I needed it first because of license-check. Now I think I can drop it away and it will compile anyway.
yes, it compiles:
[root@linux ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
openfaas/basic-auth-plugin latest-ppc64le-dev 0a9c6a69a479 5 minutes ago 14.9MB
<none> <none> 9ec411b011d4 5 minutes ago 284MB
openfaas/classic-watchdog latest-dev-x86_64 081c163f089d 6 minutes ago 6.59MB
openfaas/classic-watchdog latest-dev-ppc64le e85795b350f0 6 minutes ago 6.59MB
openfaas/classic-watchdog latest-dev-windows ed6864c24942 6 minutes ago 6.83MB
openfaas/classic-watchdog latest-dev-arm64 e5591cc7ba2a 6 minutes ago 6.33MB
openfaas/classic-watchdog latest-dev-armhf 87263af1c16d 6 minutes ago 5.77MB
openfaas/watchdog build 8608232f8397 6 minutes ago 838MB
openfaas/gateway latest-ppc64le-dev e007f38f9147 7 minutes ago 22.9MB
I change the PR (if I can...)
I have no clue about NodeJS, but new watchdog and https://github.com/openfaas-incubator/node10-express-template/ work like a charm on ppc64le with small corrections.
[root@k8s faas-ppc64le]# faas-cli new hello-node-ppc64le --lang node10-express-ppc64le
Folder: hello-node-ppc64le created.
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: hello-node-ppc64le
Stack file written: hello-node-ppc64le.yml
[root@k8s faas-ppc64le]# vi hello-node-ppc64le.yml
[root@k8s faas-ppc64le]# faas-cli build -f ./hello-node-ppc64le.yml
[0] > Building hello-node-ppc64le.
Clearing temporary build folder: ./build/hello-node-ppc64le/
Preparing ./hello-node-ppc64le/ ./build/hello-node-ppc64le//function
Building: k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le:latest with node10-express-ppc64le template. Please wait..
Sending build context to Docker daemon 10.75kB
Step 1/28 : FROM powerlinux/of-watchdog:latest-dev-ppc64le as watchdog
latest-dev-ppc64le: Pulling from powerlinux/of-watchdog
2a91e252b9fe: Pull complete
Digest: sha256:b8da6e9aaffd06300ce00f6ef5a811db967e0d2a265b93170f2839be7856ad8a
Status: Downloaded newer image for powerlinux/of-watchdog:latest-dev-ppc64le
---> 948daa732beb
Step 2/28 : FROM ppc64le/node:10.12.0-alpine as ship
10.12.0-alpine: Pulling from ppc64le/node
d6201b52ea9b: Pull complete
7c5cedde51de: Pull complete
641568bb3a3a: Pull complete
045a66b3718c: Pull complete
Digest: sha256:dde115c13a25c41dc8f5bb5f4ea30b08a8af20717dd3401909f0e56824d2de95
Status: Downloaded newer image for ppc64le/node:10.12.0-alpine
---> 768c9dfa2e55
Step 3/28 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
---> 7d1e0144c902
Step 4/28 : RUN chmod +x /usr/bin/fwatchdog
---> Running in fe3e2834462e
Removing intermediate container fe3e2834462e
---> c9b34277c428
Step 5/28 : RUN apk --no-cache add curl ca-certificates && addgroup -S app && adduser -S -g app app
---> Running in 06139de230e5
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/ppc64le/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/ppc64le/APKINDEX.tar.gz
(1/5) Installing ca-certificates (20190108-r0)
(2/5) Installing nghttp2-libs (1.32.0-r0)
(3/5) Installing libssh2 (1.8.2-r0)
(4/5) Installing libcurl (7.61.1-r2)
(5/5) Installing curl (7.61.1-r2)
Executing busybox-1.28.4-r1.trigger
Executing ca-certificates-20190108-r0.trigger
OK: 8 MiB in 20 packages
Removing intermediate container 06139de230e5
---> 910fe21f7d9e
Step 6/28 : WORKDIR /root/
---> Running in d897226c255f
Removing intermediate container d897226c255f
---> 3dd89317dc89
Step 7/28 : ENV NPM_CONFIG_LOGLEVEL warn
---> Running in 96e3bec81c33
Removing intermediate container 96e3bec81c33
---> 5e53653768ad
Step 8/28 : RUN mkdir -p /home/app
---> Running in 07e28f2bf446
Removing intermediate container 07e28f2bf446
---> 6f73c0ce4194
Step 9/28 : WORKDIR /home/app
---> Running in 556392809e5b
Removing intermediate container 556392809e5b
---> 36ea185c0982
Step 10/28 : COPY package.json ./
---> 0c401531ff53
Step 11/28 : RUN npm i
---> Running in 39c6b8fa363f
npm WARN node10-express@1.0.0 No description
npm WARN node10-express@1.0.0 No repository field.
added 50 packages from 37 contributors and audited 158 packages in 2.37s
found 0 vulnerabilities
Removing intermediate container 39c6b8fa363f
---> e19e233b9ab2
Step 12/28 : COPY index.js ./
---> f4afa50d0096
Step 13/28 : WORKDIR /home/app/function
---> Running in fcb8c5d0807d
Removing intermediate container fcb8c5d0807d
---> da033fd4e43b
Step 14/28 : COPY function/*.json ./
---> 0b05e4bfd37a
Step 15/28 : RUN npm i || :
---> Running in f4fa7b43527b
npm WARN function@1.0.0 No description
npm WARN function@1.0.0 No repository field.
up to date in 0.707s
found 0 vulnerabilities
Removing intermediate container f4fa7b43527b
---> 8dc1725dc9ef
Step 16/28 : COPY function/ ./
---> 53ffecff3719
Step 17/28 : WORKDIR /home/app/
---> Running in 7b8b5868205d
Removing intermediate container 7b8b5868205d
---> f5f22bdb738c
Step 18/28 : RUN chown app:app -R /home/app && chmod 777 /tmp
---> Running in b8afc93ae064
Removing intermediate container b8afc93ae064
---> 9f626d0dddb6
Step 19/28 : USER app
---> Running in ae4b8f39447e
Removing intermediate container ae4b8f39447e
---> 9b2609dd2ea8
Step 20/28 : ENV cgi_headers="true"
---> Running in 97ad77384bd3
Removing intermediate container 97ad77384bd3
---> 1f94c263ec20
Step 21/28 : ENV fprocess="node index.js"
---> Running in 62aa3683220f
Removing intermediate container 62aa3683220f
---> a281a8b16a46
Step 22/28 : ENV mode="http"
---> Running in 4940b07cba7c
Removing intermediate container 4940b07cba7c
---> 6d33de8fc283
Step 23/28 : ENV upstream_url="http://127.0.0.1:3000"
---> Running in d3ac908084b5
Removing intermediate container d3ac908084b5
---> fb3875dc3c15
Step 24/28 : ENV exec_timeout="10s"
---> Running in e2e9f6af0fee
Removing intermediate container e2e9f6af0fee
---> 9166920bbdfb
Step 25/28 : ENV write_timeout="15s"
---> Running in 92250ef5061d
Removing intermediate container 92250ef5061d
---> 46b44ea7719b
Step 26/28 : ENV read_timeout="15s"
---> Running in 2644b2ff4251
Removing intermediate container 2644b2ff4251
---> 9176bbc177cf
Step 27/28 : HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1
---> Running in e6af509f832f
Removing intermediate container e6af509f832f
---> d3f05bb7bf47
Step 28/28 : CMD ["fwatchdog"]
---> Running in ad3d58c09e51
Removing intermediate container ad3d58c09e51
---> a109b7a7d4ab
Successfully built a109b7a7d4ab
Successfully tagged k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le:latest
Image: k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le:latest built.
[0] < Building hello-node-ppc64le done.
[0] worker done.
[root@k8s faas-ppc64le]# docker images | grep hello-node
k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le latest a109b7a7d4ab 35 seconds ago 95MB
[root@k8s faas-ppc64le]# docker push k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le:latest
The push refers to repository [k8s.enfence.net:8500/openfaas-fn/hello-node-ppc64le]
321f51c90230: Pushed
c0855c0b2b91: Pushed
ee3888e2df21: Pushed
68680c84b798: Pushed
018097ac38e1: Pushed
979316d3c516: Pushed
15d8eba5c710: Pushed
29807fdc3c49: Pushed
4d200f875e35: Pushed
f26693e74866: Pushed
75bba5b16d90: Pushed
8c9e0897b857: Pushed
a4946bd55684: Pushed
a90a6890b39a: Pushed
latest: digest: sha256:e81ec5a4a4b5850c4417b5f81834c9c1ee7573f05bdc11cf91d1b2f358abacae size: 3455
[root@k8s faas-ppc64le]# faas-cli list
Function Invocations Replicas
hello-ppc64le 2 1
[root@k8s faas-ppc64le]# faas-cli deploy -f ./hello-node-ppc64le.yml
Deploying: hello-node-ppc64le.
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Deployed. 202 Accepted.
URL: http://k8s.enfence.net:31112/function/hello-node-ppc64le
[root@k8s faas-ppc64le]# faas-cli list
Function Invocations Replicas
hello-node-ppc64le 0 1
hello-ppc64le 2 1
[root@k8s faas-ppc64le]# echo "hi" | faas-cli invoke hello-node-ppc64le
{"status":"You said: \"hi\\n\""}[root@k8s faas-ppc64le]#
Pull request for of-watchdog https://github.com/openfaas-incubator/of-watchdog/pull/69
wow. it works too - https://github.com/openfaas/workshop/blob/master/lab9.md
[root@k8s faas-ppc64le]# faas-cli list
Function Invocations Replicas
go-echo 10000 5
hello-node-ppc64le 2 1
hello-ppc64le 4 1
Is there a way project contributors can access Power environments free of cost?
2 open alternatives I know:
Thank you for your great demo in the community meeting today. once a recording is ready I will post a link here for anybody else who is interested in power and OpenFaaS
/lock: inactivity
I would like to see support for Linux/ppc64le architecture in OpenFaaS.
What I saw till now it should be quite easy task. I didn't test it, but compilation of almost everything went without any problems. I plan to try out what I compiled till now this or next week.
Expected Behaviour
OpenFaaS works on Linux/ppc64le
Current Behaviour
OpenFaaS doesn't work on Linux/ppc64le
Your Environment
My test environments are IBM Cloud Private 3.2.0 Community Edition on one OpenPOWER server and IBM Cloud Private 3.2.0 Enterprise Edition on IBM Power.
Docker version
docker version
(e.g. Docker 17.0.05 ): 18.06Are you using Docker Swarm or Kubernetes (FaaS-netes)? Kubernetes
Operating System and version (e.g. Linux, Windows, MacOS): RHEL 7/ppc64le
Next steps
PR