Closed viktor-lt closed 3 months ago
@vikomte: This issue is currently awaiting triage.
If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted
label and provide further guidance.
The triage/accepted
label can be added by org members by writing /triage accepted
in a comment.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
No one has been able to clear this out ?
There is no data in this issue that someone can analyse. There is some documentation here https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/ .
But practically if someone is really going to create a new Kubernetes cluster and then write new php based apps on it, then they will find it more practical to combine the php-code, the fpm software and the nginx web server in one single docker image and spawn one or more pod(s) from that image. Because its simpler to have a 12 factor microservice app in a single self-contained rather than splitting php-code, fpm software and nginx web server into 3 different pods.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/reopen
/remove-lifecycle rotten
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
@k8s-triage-robot: Closing this issue.
There is no data in this issue that someone can analyse. There is some documentation here https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/ .
But practically if someone is really going to create a new Kubernetes cluster and then write new php based apps on it, then they will find it more practical to combine the php-code, the fpm software and the nginx web server in one single docker image and spawn one or more pod(s) from that image. Because its simpler to have a 12 factor microservice app in a single self-contained rather than splitting php-code, fpm software and nginx web server into 3 different pods.
Or... the one pod with the php files and php-fpm in one pod, and using the already-existing "ingress nginx" pods and ecosystem the cluster already has. Spinning up more nginx pods is redundant and wasteful for resources when a configuration should be able to serve this.
Any Updates ?
Were having exactly this issue - the page https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/ is what we followed and no joy, I can see my liviness/rediness probes working (using the php-fpm-healthcheck we get /status hits in the php-fpm log) but all I seem to get is 504.
Using php-fpm latest (8.3) - if I put my own NGINX in my deployment and proxy then this works, but since the ingress is NGINX and I don't care about static serving using the built in ingress to talk FCGI seems the right route to go.
apiVersion: v1
kind: Service
metadata:
name: web
labels:
app: web
spec:
selector:
app: web
ports:
- port: 9000
targetPort: 9000
name: fastcgi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: php-fpm-web-default
data:
SCRIPT_FILENAME: "/var/www/public/index.php"
HTTP_PROXY: ""
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# docs say you just need this -
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
# https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/
nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
# above triggers include /etc/nginx/fastcgi_params;
nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
nginx.ingress.kubernetes.io/fastcgi-params-configmap: "php-fpm-web-default"
name: web
spec:
ingressClassName: nginx
rules:
- host: another.domain
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
name: fastcgi
The workaround for us was to add another container to our deployment for NGINX and handle fast-cgi there so we can proxy from the ingress controller via the HTTP protocol, however it seems like this should be possible without.
Are there any updates on this issue?
well this issue is closed so maybe we need to open a new one - bit of a joke really, asking google it seems this is a common issue but the docs do state this sort of thing should work - would be great if I'd just missed something, but I'm certainly struggling - if I get a chance I'll get a full basic test case together and raise a fresh as our robot enemy @k8s-ci-robot has closed this issue - which means its unlikely to be being worked on.
- its too far from being a joke for the maintainers because there was feedback from survey that there were no users, so the feature was deprecated, and then later someone said they are using so the deprecation was reverted. all done in volunteering time hence far from being a joke.
- There is this https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/ and the issue template asks questions to help a reader do some analysis. But there is not enough info provided here to do any analysis. in fact if you see the issue description, its one yaml post and nothing else
- If there is enough data posted here for readers to analyse and reproduce, I guess it will be easier to make comment based on that data (if posted here)
that page was the guide I followed - there is more info on the github pages - but this documentation does not work for me - pretty sure the yamls provided should be proof enough, I'd be happy to pull together something that can be used as a unit test as a proof this issue still presents.
ok - i'll do my best - but we currently have several large projects on the go and while this would be awesome to get sorted (as it removes yet another layer of abstraction) the unfortunate thing is the work around while daft/wasteful it does work... maybe I missed something but I have already lost several days trying to make this work so would be good to get it written up properly for you guys.
in that case ;
I have tried this ;
k create deploy phpfpm --image bitnami/php-fpm --port 9000
k expose deploy phpfpm --port 9000
kubectl apply -f - << EOF\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: phpfpm\ndata:\n SCRIPT_FILENAME: "/app/index.php"\nEOF
k create ing phpfpm --class nginx --rule phpfpm.dev.enjoydevops.com/"*"=phpfpm:9000
Then I edited the ingress and added the required annotations like thus ;
% k describe ing phpfpm
Name: phpfpm
Labels: <none>
Namespace: default
Address: 192.168.122.193
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
phpfpm.dev.enjoydevops.com
/ phpfpm:9000 (10.244.0.18:9000)
Annotations: nginx.ingress.kubernetes.io/backend-protocol: FCGI
nginx.ingress.kubernetes.io/fastcgi-index: index.php
nginx.ingress.kubernetes.io/fastcgi-params-configmap: phpfpm
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 5m20s (x3 over 23m) nginx-ingress-controller Scheduled for sync
So you can do the same and point your app's serverblock to use it like ;
server {
listen 0.0.0.0:80;
server_name myapp.com;
root /app;
location / {
try_files $uri $uri/index.php;
}
location ~ \.php$ {
fastcgi_pass phpfpm:9000;
fastcgi_index index.php;
include fastcgi.conf;
}
}
as described here https://hub.docker.com/r/bitnami/php-fpm
but i feel my comments or the info in them and your comments need to go into another issue that has the appropriate issue description
These annotations are for the ingress controller to determine how the traffic should be routed to the container -
nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
# above triggers include /etc/nginx/fastcgi_params;
nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
nginx.ingress.kubernetes.io/fastcgi-params-configmap: "php-fpm-web-default"
So I was expecting the HTTP request to go direct to the fastcgi container as we've told the ingress controller what the protocol is.
This is a bit confusing since it gives the impression to the reader that you could route traffic to an API endpoint from external HTTP requests without the need of an additional NGINX container within your deployment, else whats the purpose here? Since NGINX server block above is stating its a fastcgi_pass etc, it would not use the annotations - the pod exposes ports - your block would be for an nginx instance within the deployment no? What I expected (righty or wrongly) was that the annotations would instruct the ingress controller to proxy to php for you since we've told it about the protocol.
Please move to another issue because this issue describes the problem as "I'm still unable to run a simple website:"
- from what little I understand this project's controller is providing a feature of exposing a fastcgi backend via a ingress
I think i get this now - so you can expose the FGCI to outside world, rather than implementing the proxy within the ingress.
- from why I understand this is offered, is the feature is an alternative for multiple apps to use one fastcgi server instead of each app deploying its own fastcgi server
Dont think this is the case since the FCGI server is the destination of said ingress.
- other specs are details for the above goal
- so I did not expect that fastci feature implies routing to a webserver in addition to a fastcgi server
- so not proxy to php but proxy to a fastcgi server, which in my example is fpm for php
- so the webserver (nginx/apache/blah) hosts the app code in a different pod but that webserver uses proxy_pass to this fastcgi server
Please move to another issue because this issue describes the problem as
"I'm still unable to run a simple website:"
So basically what we are saying is that the ingress routes traffic to an endpoint that must support HTTP if you want to route web traffic to it, and that HTTP->FCGI is something you have to handle within your deployment since nginx-ingress wont help you with this, however you can also expose FCGI to external proxy if you require which is what this feature does.
Thanks for your help in clarifying this - in fact it is more obvious now re-reading the docs -
The ingress-nginx ingress controller can be used to directly expose FastCGI servers. Enabling FastCGI in your Ingress only requires setting the backend-protocol annotation to FCGI, and with a couple more annotations you can customize the way ingress-nginx handles the communication with your FastCGI server.
directly expose means literally the raw service, not proxied as an http endpoint that a browser can visit. The mistake made is in assuming ingress would can handle this for you - its a little more simple than that, a point I'd missed.
Thanks @longwuyuan for your patience - I get this now, in our current context we route into this cluster via a proxy so this feature will still be useful since the HTTP server can proxy to the FPM direct instead.
@BloodyIron @fabpico I see you 2 as successful users of the FCGI feature, in https://github.com/kubernetes/ingress-nginx/issues/10462. Requesting you to comment on the last few comments above. If I am wrong, I apologise but your comments could help clarify for @web-engineer . Thanks
So basically what we are saying is that the ingress routes traffic to an endpoint that must support HTTP if you want to route web traffic to it, and that HTTP->FCGI is something you have to handle within your deployment since nginx-ingress wont help you with this, however you can also expose FCGI to external proxy if you require which is what this feature does.
directly expose means literally the raw service, not proxied as an http endpoint that a browser can visit. The mistake made is in assuming ingress would can handle this for you - its a little more simple than that, a point I'd missed.
I don't think this understanding is correct.
I just configured my Ingress definition as per the documentation (https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/) and it looks like it started to correctly proxy requests to FCGI backend. Frontend protocol exposed by ingress controller is still HTTP(s).
Even the annotation name - nginx.ingress.kubernetes.io/backend-protocol
suggests you are only changing the protocol which nginx uses to talk to your backend.
@kamilzzz if what you are reporting is true (at least my interpretation) then I think I have made a huge huge mistake.
My request is that you help me understand this data. The port name in the ingress here https://kubernetes.github.io/ingress-nginx/user-guide/fcgi-services/ is fastcgi
the port number is 9000
And the port name in the ingress resource is fastcgi
the name fastcgi in the ingress resource hints the port number behind name fastcgi is port number 9000
is this assumption wrong ?
If its wrong, kindly help understand your description
Frontend protocol exposed by ingress controller is still HTTP(s).
Where is the HTTPS port 443 in that example.
If youhave a testbed, can you post all the kubectl describe
output for pod, service, ingress involved and also a curl request exactly as you used and the logs of the controller and the response to curl with -v etc etc
This is very similar setup to mine.
Now, you should be able to access app.example.com
hostname via your ingress controller IP (cluster or external, depending on the setup), via HTTP/HTTPS protocol (http can be disabled via Helm chart value when installing ingress controller).
This is the same like your app would expose something on port 12345
. You wouldn't see any explicit HTTP/HTTPS ports defined at the ingress level, but the ingress controller itself is listening for HTTP/HTTPS traffic on ports defined when installing controller.
The only difference is that by default ingress controller talks to backend via HTTP, while in FCGI sample this is changed to FCGI protocol by annotation.
I wish we could have the kubectl describe
outputs for all the resources related and the curl request sent to that ingress along with the controller logs for that request.
To begin with I am confused about this
% k explain ingress.spec.rules.http.paths.backend.service.port
GROUP: networking.k8s.io
KIND: Ingress
VERSION: v1
FIELD: port <ServiceBackendPort>
DESCRIPTION:
port of the referenced service. A port name or port number is required for a
IngressServiceBackend.
ServiceBackendPort is the service port being referenced.
FIELDS:
name <string>
name is the name of the port on the Service. This is a mutually exclusive
setting with "Number".
number <integer>
number is the numerical port number (e.g. 80) on the Service. This is a
mutually exclusive setting with "Name".
So if port number is fcgi=9000 then how was the request received on port 443 ? Or was the request sent to port 9000 and the controller accepted it because the controller is listening on 9000 in addition to 80 & 443 ?
I will try to test but I am confused if the example-app is a fpm service or a php application
@kamilzzz can you post the kubectl describe
of your example-app pod + service + ingress
I need some help here to reproduce what you posted.
php:fpm-alpine
to create the pod. Pod created
% k -n fcgi describe po example-app
Name: example-app
Namespace: fcgi
Priority: 0
Service Account: default
Node: kind-control-plane/172.19.0.2
Start Time: Tue, 11 Jun 2024 22:47:58 +0530
Labels: app=example-app
Annotations: <none>
Status: Running
IP: 10.244.0.17
IPs:
IP: 10.244.0.17
Containers:
example-app:
Container ID: containerd://6fa9c879fcb1d3b6cc652d8561c4acebb0abca6254df11d2d2aaf109c63f5522
Image: php:fpm-alpine
Image ID: docker.io/library/php@sha256:b10c555b338ad8e9b4f4de24819c8c15bf442bf8c3a48cd54ff2e7ea65d2f8d2
Port: 9000/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 11 Jun 2024 22:47:58 +0530
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4ksh6 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-4ksh6:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m6s default-scheduler Successfully assigned fcgi/example-app to kind-control-plane
Normal Pulled 4m6s kubelet Container image "php:fpm-alpine" already present on machine
Normal Created 4m6s kubelet Created container example-app
Normal Started 4m6s kubelet Started container example-app
Service created
% k -n fcgi describe svc example-service
Name: example-service
Namespace: fcgi
Labels: <none>
Annotations: <none>
Selector: app=example-app
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.96.42.173
IPs: 10.96.42.173
Port: fastcgi 9000/TCP
TargetPort: 9000/TCP
Endpoints: 10.244.0.17:9000
Session Affinity: None
Events: <none>
configMap created
% k -n fcgi describe cm example-cm
Name: example-cm
Namespace: fcgi
Labels: <none>
Annotations: <none>
Data
====
SCRIPT_FILENAME:
----
/example/index.php
BinaryData
====
Events: <none>
Ingress Created
% k -n fcgi describe ing example-app
Name: example-app
Labels: <none>
Namespace: fcgi
Address: 172.19.0.2
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
app.example.com
/ example-service:fastcgi (10.244.0.17:9000)
Annotations: nginx.ingress.kubernetes.io/backend-protocol: FCGI
nginx.ingress.kubernetes.io/fastcgi-index: index.php
nginx.ingress.kubernetes.io/fastcgi-params-configmap: example-cm
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 5m45s (x2 over 6m17s) nginx-ingress-controller Scheduled for sync
Log Messages in controller after creating Ingress
I0611 17:20:00.497617 12 admission.go:149] processed ingress via admission controller {testedIngressLength:2 testedIngressTime:0.021s renderingIngressLength:2 renderingIngressTime:0.001s admissionTime:0.022s testedConfigurationSize:22.0kB}
I0611 17:20:00.497659 12 main.go:107] "successfully validated configuration, accepting" ingress="fcgi/example-app"
I0611 17:20:00.505761 12 store.go:440] "Found valid IngressClass" ingress="fcgi/example-app" ingressclass="nginx"
I0611 17:20:00.505992 12 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"fcgi", Name:"example-app", UID:"81289ab7-f986-4d53-b225-9be15b993fc4", APIVersion:"networking.k8s.io/v1", ResourceVersion:"28951", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0611 17:20:00.506648 12 controller.go:190] "Configuration changes detected, backend reload required"
I0611 17:20:00.549716 12 controller.go:210] "Backend successfully reloaded"
I0611 17:20:00.549876 12 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7b5589785f-nq5rx", UID:"94c764e6-b8bf-47d5-92c6-b3a58e85a904", APIVersion:"v1", ResourceVersion:"18413", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0611 17:20:32.465633 12 status.go:304] "updating Ingress status" namespace="fcgi" ingress="example-app" currentValue=null newValue=[{"ip":"172.19.0.2"}]
I0611 17:20:32.472198 12 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"fcgi", Name:"example-app", UID:"81289ab7-f986-4d53-b225-9be15b993fc4", APIVersion:"networking.k8s.io/v1", ResourceVersion:"29001", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
@kamilzzz , can you please guide me on what should my curl command be ? Or how should I send a request to this ingress ?
I had assumed that a nginx webserver would use this service like this example https://gist.github.com/md5/d9206eacb5a0ff5d6be0#file-wordpress-fpm-conf-L25
The pod is listening on port 9000 ;
% k -n fcgi exec -ti pods/example-app -- sh
/var/www/html # netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 1/php-fpm.conf)
netstat: /proc/net/tcp6: No such file or directory
/var/www/html #
And the example-app service is available to clients outside the cluster, via the ingress, for connections from outside the cluster to the example-app pod running fpm ;
% k -n fcgi get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
example-app nginx app.example.com 172.19.0.2 80 24m
Should copy/paste the fair warning officially provided by the developers of the PHP interpreter here https://hub.docker.com/_/php
Ingress controller installation:
helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-fpm
labels:
app: php-fpm
spec:
replicas: 1
selector:
matchLabels:
app: php-fpm
template:
metadata:
labels:
app: php-fpm
spec:
containers:
- name: php-fpm
image: php:8.2-fpm
ports:
- containerPort: 9000
Service:
apiVersion: v1
kind: Service
metadata:
name: php-fpm-service
spec:
selector:
app: php-fpm
ports:
- protocol: TCP
port: 9000
targetPort: 9000
Config map (for ingress):
apiVersion: v1
kind: ConfigMap
metadata:
name: example-cm
data:
SCRIPT_FILENAME: "/var/www/html/index.php"
Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: php-fpm-ingress
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "FCGI"
nginx.ingress.kubernetes.io/fastcgi-index: "index.php"
nginx.ingress.kubernetes.io/fastcgi-params-configmap: "example-cm"
spec:
ingressClassName: nginx
rules:
- host: fcgi.kubernetes.int
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: php-fpm-service
port:
number: 9000
Deployment order: deployment -> service -> config map -> ingress (config map should be deployed before ingress definition as described in the ingress-controller docs).
By default php:8:2-fpm
image is empty. Exec into it and create index.php
file with any valid PHP code under /var/www/html
. I used sample one:
<!DOCTYPE html>
<html>
<head>
<title>PHP Test</title>
</head>
<body>
<?php echo '<p>Hello World</p>'; ?>
</body>
</html>
Testing with curl:
kubectl run --rm -it --image=ubuntu ubuntu -- bin/bash
apt-get update && apt-get install -y curl
curl -vik <ingress-controller-svc-cluster-ip> -H "Host: fcgi.kubernetes.int"
Response:
So the flow is like that:
Client ---HTTP/HTTPS (80/443)---> Ingress Controller ---FCGI (9000)---> Backend pod
@kamilzzz, you are right. Thank you very very much for the help. If you want to improve the docs with a PR, it will be a very useful contribution for users of FastCGI.
@web-engineer sorry. No need to proxy from elsewhere.
These tests did not include all the directives supported by the controller so that will be a task to-do.
Reopening this so that we can improve the docs on FastCGI, if posible.
/reopen /assign
@longwuyuan: Reopened this issue.
@vikomte latest messagesare data showing successful php rendering html
@kamilzzz I just locust tested 20K users on laptop kind cluster to php:fpm-alpine. Are you using php-fpm images as-is in production and not breaking a sweat !
If you are, does your CI just keep rolling out new images that are php-code, plus php plus fpm .
@kamilzzz Can you look at this https://deploy-preview-11454--kubernetes-ingress-nginx.netlify.app/user-guide/fcgi-services/ and comment
@BloodyIron @fabpico I see you 2 as successful users of the FCGI feature, in #10462. Requesting you to comment on the last few comments above. If I am wrong, I apologise but your comments could help clarify for @web-engineer . Thanks
Sorry probably going to be a while before I can say anything useful on this topic, life stuff, hence the delay in my response, sorry about that! :((
Have you managed to do it ? I'm still unable to run a simple website:
php:7.4-fpm -> EXPOSE 4243 -> /var/www/html/index.php -> <?php phpinfo(); ?>
ConfigMap -> SCRIPT_FILENAME: "/var/www/html/index.php"
Originally posted by @vikomte in https://github.com/kubernetes/ingress-nginx/issues/6602#issuecomment-1025126932