ory / k8s

Kubernetes Helm Charts for the ORY ecosystem.
https://k8s.ory.sh/helm
Apache License 2.0
334 stars 261 forks source link

OathKeeper Default Helm Chart Issue | Pod throwing 503. #595

Open sabinayakc opened 1 year ago

sabinayakc commented 1 year ago

Preflight checklist

Describe the bug

Installed helm chart using helm install oathkeeper ory/oathkeeper.

oathkeeper pod has an error with 503 code.

image

Reproducing the bug

Relevant log output

time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for api, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4456 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=TLS has not been configured for proxy, skipping audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:53Z level=info msg=Listening on http://:4455 audience=application service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:55Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33658 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:56Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:33662 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2
time=2023-04-21T17:39:57Z level=error msg=An error occurred while handling a request audience=application error=map[message:The requested resource could not be found] http_request=map[headers:map[accept:*/* connection:close user-agent:kube-probe/1.23+] host:127.0.0.1 method:GET path:/health/ready query:<nil> remote:172.26.6.14:42292 scheme:http] http_response=map[status_code:503] service_name=ORY Oathkeeper service_version=v0.40.2

Relevant configuration

- Default Helm Values

Version

Helm Version: 0.31.0

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

Demonsthere commented 1 year ago

Hello there, This behaviour is due to the k8s readiness probe failing and restarting the pod. The probe is failing because no rules are present in the system, that is why the demo mode works, as it deploys sample data. As Oathkeeper is not k8s native, it expects the rules to be present on start, and treats an an empty rule array as an error state.

sabinayakc commented 1 year ago

How can I instantiate it with a basic rule? Do I have to provide it a basic rule via Helm Values always?

The default helm has the following access rule which might be blank.

  config:
    access_rules:
      repositories:
        - file:///etc/rules/access-rules.json

I also created a rule using the CRD to see if it picks it up.

apiVersion: oathkeeper.ory.sh/v1alpha1
kind: Rule
metadata:
  name: test-rule
  namespace: default
spec:
  match:
    url: http://http-bin.example/echo
    methods:
    - GET
  authenticators:
  - handler: anonymous
  authorizer:
    handler: allow
  mutators:
  - handler: noop
Demonsthere commented 1 year ago

To clear up some confusion :)

TartanLeGrand commented 1 year ago

I see your version is 0.40.2 try to downgrade in 0.40.1

Demonsthere commented 1 year ago

Closing as this is an user error, please reopen if you need more guidance :)

TartanLeGrand commented 1 year ago

I think the bug is not fixed in the actual version (0.40.6)

👋 @Demonsthere

Jorgagu commented 1 year ago

This is a problem created by PR https://github.com/ory/oathkeeper/pull/1061 and so Release v0.40.2, we should have a flag or option to enable the check performed on PR 1061.

@zepatrik @hperl In our case, if we use Oathkeeper Maester, we get a 503 error when launching Oathkeeper Readiness Probe because there are no rules, although we create them later from the controller.

👋 @Demonsthere can you please reopen the issue ?

Demonsthere commented 1 year ago

I see, so this is a upstream issue from oathkeeper itself 🤔 I will talk with the devs, maybe adding a --allow-empty-rules flag could be added to disable that check, which would be the default option for maester enabled charts

zepatrik commented 1 year ago

Would be fine by me.

ognyvrac commented 11 months ago

Is there any progress on this issue?

jeffmccune commented 9 months ago

This issue is unrelated to rules, the pod starts fine with no rules.

Please disregard, I was checking /health/alive instead of /health/ready

MohammadBnei commented 2 months ago

Any news on this issue ? I'm facing the exact same challenge, and it's quite frustrating.

If i put a dummy access rule via file, then use the CRD for more control, will there be any issues ?

Demonsthere commented 2 weeks ago

yes, using a dummy placeholder rule is a valid workaround, as it's a design in oathkeeper that it should start with the access rules supplied. We need to open a issue in oathkeeper to allow for such situation