Open cyrilgdn opened 6 years ago
ping @saltstack/team-core any ideas on how to solve the issue where hte engine and events start sending before the reactor is up
We would need to add the ability to order the starting of engines. To make sure the reactor engine is up before the sqs engine.
hi team, any update on this issue ? i am facing the same problem. root@salt:~# salt --versions-report Salt Version: Salt: 2018.3.3
Dependency Versions: cffi: 1.11.5 cherrypy: unknown dateutil: 2.7.5 docker-py: Not Installed gitdb: 2.0.3 gitpython: 2.1.8 ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: 1.0.7 msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: 2.19 pycrypto: 2.6.1 pycryptodome: 3.7.2 pygit2: Not Installed Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15) python-gnupg: 0.4.1 PyYAML: 3.12 PyZMQ: 16.0.2 RAET: Not Installed smmap: 2.0.3 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.5
System Versions: dist: Ubuntu 18.04 bionic locale: UTF-8 machine: x86_64 release: 4.15.0-1021-aws system: Linux version: Ubuntu 18.04 bionic
afaik there is no work currently being done allowing ordering of engine starts.
But you could try directly specifying the reactor in the list of engines.
log_level: debug
engines:
- reactor: {}
- sqs_events:
queue: salt-queue
sqs:
message_format: json
sqs.region: eu-central-1
reactor:
- salt/engine/sqs:
- salt://test_sqs_events.sls
And see if explicitly putting the reactor engine before the sqs events engine causes the reactor to be started first.
The way that the reactor is started now is if you have a reactor
key in the master config, it adds the reactor engine to the list of engines, and starts them in order.
@gtmanfred ordering the engine starts doesn't seem to help. is there any other solution?
thanks Srinivas
nope, it would require a change to allow for ordering of the engines to start, and something to signify that an engine has finished starting before the next one is started.
and that would be a feature request if @saltstack/team-triage will tag it.
thanks
thanks, Daniel,
it would be great if @saltstack/team-triage https://github.com/orgs/saltstack/teams/team-triage can tag it and provide a solution/fix at the earliest.
Since this is asking for new functionality, the ability to specify some dependencies in what order engines load, which doesn't currently exist I'll go ahead and label it as a feature request and approve it for a future feature addition.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
Still valid
Thank you for updating this issue. It is no longer marked as stale.
Description of Issue/Question
See also: https://groups.google.com/forum/#!topic/salt-users/tAm-c-Cs5ss
We are using the sqs_events engine but we are losing messages. When Salt is starting, sqs_events engine is ready and starts to send events before the reactor engines is properly started.
I also test to sleep a bit at the beginning of the
start
function of the sqs_events engine. With this, messages are not lost at startup anymore but we are still losing messages when restarting so it seems that when Salt is stopping, events may be sent when reactors are already stopped.Test setup
For testing purpose, I've just created a reactor which creates empty files with filenames retrieved for the SQS queue and I'm sending messages in the queue manually.
Master configuration (relevant part):
states/test_sqs_events.sls
Steps to Reproduce Issue
The first test is to stop salt-master, then to run
Then to start salt-master
No messages will be processed by reactors:
Logs (relevant part)
Restart
If we test to restart salt-master during message sending:
Then just after messages started to be sent:
We can see that some messages have been lost:
Versions Report
(also tested on Salt 2016.11.X)