search

nolar / kopf

A Python framework to write Kubernetes operators in just a few lines of code
https://kopf.readthedocs.io/
MIT License
2.13k stars 162 forks source link

Unordered and missed resources #958

Closed davidp1404 closed 2 years ago

davidp1404 commented 2 years ago

Long story short

Hello, I've created an dns operator to manage dns instances and records and it works well when create/update/delete resources one by one, but when I create multiple resources using a yaml file only some of them are processed by kopf completely and they are executed in a order different than created (in my use case the dnsrecord is created before the dnsserver is fully processed so it fails).

I've read in issue 729 this:

Kopf has some protection measures against quickly arriving events — the "batch window" (settings.batching.batch_window, defaults to 0.1 seconds). All events that arrived within this time window are reduced only to the latest event, which means the latest object state. All intermediate events/states are ignored.

I don't know if that could be related.

Kopf version

1.35.6

Kubernetes version

1.22

Python version

No response

Code

https://github.com/davidp1404/dns-operator)

Logs

$ kopf run --priority=10 --all-namespaces src/dns-operator.py
[2022-09-09 18:08:48,282] kopf.activities.star [INFO    ] Activity 'configure' succeeded.
[2022-09-09 18:08:48,283] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2022-09-09 18:08:48,323] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2022-09-09 18:08:48,324] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2022-09-09 18:09:07,450] kopf.objects         [INFO    ] [default/record1] Adding Record "record1 IN A 10.0.0.2
[2022-09-09 18:09:07,485] kopf.objects         [INFO    ] [default/record3] Adding Record "3 IN PTR record3.sample.org.
[2022-09-09 18:09:07,488] kopf.objects         [INFO    ] [default/record4] Adding Record "4 IN PTR record4.sample.org.
[2022-09-09 18:09:07,489] kopf.objects         [INFO    ] [default/record5] Adding Record "5 IN PTR record5.sample.org.
[2022-09-09 18:09:07,505] kopf.objects         [INFO    ] [default/record2] Adding Record "record2 IN A 10.0.0.1
[2022-09-09 18:09:07,502] kopf.objects         [INFO    ] [default/record6] Adding Record "record6 IN CNAME record1.sample.org.
[2022-09-09 18:09:07,539] kopf.objects         [INFO    ] [default/record2] Adding Record "record2 IN A 10.0.0.3
[2022-09-09 18:09:17,655] kopf.objects         [INFO    ] [default/record1] Rollout deployment finish without errors
[2022-09-09 18:09:17,660] kopf.objects         [INFO    ] [default/record1] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,662] kopf.objects         [INFO    ] [default/record1] Creation is processed: 1 succeeded; 0 failed.
[2022-09-09 18:09:17,697] kopf.objects         [INFO    ] [default/record3] Rollout deployment finish without errors
[2022-09-09 18:09:17,702] kopf.objects         [INFO    ] [default/record3] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,703] kopf.objects         [INFO    ] [default/record3] Creation is processed: 1 succeeded; 0 failed.
[2022-09-09 18:09:17,770] kopf.objects         [INFO    ] [default/record5] Rollout deployment finish without errors
[2022-09-09 18:09:17,773] kopf.objects         [INFO    ] [default/record5] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,774] kopf.objects         [INFO    ] [default/record5] Creation is processed: 1 succeeded; 0 failed.
[2022-09-09 18:09:17,865] kopf.objects         [INFO    ] [default/record2] Rollout deployment finish without errors
[2022-09-09 18:09:17,869] kopf.objects         [INFO    ] [default/record2] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,870] kopf.objects         [INFO    ] [default/record2] Creation is processed: 1 succeeded; 0 failed.
[2022-09-09 18:09:17,886] kopf.objects         [INFO    ] [default/record6] Rollout deployment finish without errors
[2022-09-09 18:09:17,895] kopf.objects         [INFO    ] [default/record6] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,895] kopf.objects         [INFO    ] [default/record6] Creation is processed: 1 succeeded; 0 failed.
[2022-09-09 18:09:17,941] kopf.objects         [INFO    ] [default/record4] Rollout deployment finish without errors
[2022-09-09 18:09:17,944] kopf.objects         [INFO    ] [default/record4] Handler 'create_dnsrecords' succeeded.
[2022-09-09 18:09:17,945] kopf.objects         [INFO    ] [default/record4] Creation is processed: 1 succeeded; 0 failed.

Additional information

In previous logs, all the records seem to be processed but the changes were not shown in the configmap, something that works fine if I create/delete/update records one by one. In this example record 3 an 4 are not in configmap and kopf doesn't show any error

`$ k get cm dns-operator-sample-org -o yaml apiVersion: v1 data: Corefile: | .:1053 { root /etc/coredns errors health ready loadbalance round_robin prometheus :9153 reload 2s 1s log file db.sample.org sample.org file db.0.0.10.in-addr.arpa 0.0.10.in-addr.arpa file db.example.org example.org file db.0.1.10.in-addr.arpa 0.1.10.in-addr.arpa } db.0.0.10.in-addr.arpa: | $ORIGIN 0.0.10.in-addr.arpa. @ 3600 IN SOA sns.dns.icann.org. noc.dns.icann.org. ( 2017042761 ; serial 7200 ; refresh (2 hours) 3600 ; retry (1 hour) 1209600 ; expire (2 weeks) 3600 ; minimum (1 hour) ) 3600 IN NS a.iana-servers.net. 3600 IN NS b.iana-servers.net. 5 IN PTR record5.sample.org. db.0.1.10.in-addr.arpa: | $ORIGIN 0.1.10.in-addr.arpa. @ 3600 IN SOA sns.dns.icann.org. noc.dns.icann.org. ( 2017042761 ; serial 7200 ; refresh (2 hours) 3600 ; retry (1 hour) 1209600 ; expire (2 weeks) 3600 ; minimum (1 hour) ) 3600 IN NS a.iana-servers.net. 3600 IN NS b.iana-servers.net. db.example.org: | $ORIGIN example.org. @ 3600 IN SOA sns.dns.icann.org. noc.dns.icann.org. ( 2017042761 ; serial 7200 ; refresh (2 hours) 3600 ; retry (1 hour) 1209600 ; expire (2 weeks) 3600 ; minimum (1 hour) ) 3600 IN NS a.iana-servers.net. 3600 IN NS b.iana-servers.net. db.sample.org: | $ORIGIN sample.org. @ 3600 IN SOA sns.dns.icann.org. noc.dns.icann.org. ( 2017042761 ; serial 7200 ; refresh (2 hours) 3600 ; retry (1 hour) 1209600 ; expire (2 weeks) 3600 ; minimum (1 hour) ) 3600 IN NS a.iana-servers.net. 3600 IN NS b.iana-servers.net. record1 IN A 10.0.0.2 record2 IN A 10.0.0.1 record2 IN A 10.0.0.3 kind: ConfigMap metadata: creationTimestamp: "2022-09-09T15:39:59Z" labels: app: dns-operator-sample-org app.kubernetes.io/managed-by: dns-operator name: dns-operator-sample-org namespace: default ownerReferences:

davidp1404 commented 2 years ago

Hi. my problems were solved after applying async io best practices, basically: