Thorium reactor not working

[azelezni]

Description of Issue/Question

Thorium reactor doesn't seem to work at all. Followed the following article https://hub.packtpub.com/thorium-and-salt-api/, and tried some custom events but nothing seems to work.

I have a reactor setup for the same event tag and the reactor is working fine, but with the same event tags thorium just does nothing.

Even with thorium_test_no_event.sls (this is supposed to write to a file every 0.5 seconds, based on thorium_interval (an option that is no where to be found in the docs)), but again, nothing seems to be happening.

No error in the logs, actually no logs what so ever from Thorium, I had to add my own logs to try and figure out what is happening and it seems like there is an issue with the top file, is the class "ThorState" the get_chunks() functions just returns an empty list, meaning no top file matches were found (if I understand correctly).

Setup

master config:

engines:
  - thorium: {}

thorium_roots:
  base:
    - /srv/thorium

thorium top.sls

base:
  '*':
    - thorium_test
    - thorium_test_no_event

thorium_test.sls:

checker:
  check.event:
    - value: trigger
    - name: salt/thorium/*/test

shell_test:
  local.cmd:
    - tgt: saltmaster.example.com
    - func: cmd.run
    - args:
      - cmd: 'echo "thorium event success" >> /tmp/thorium.txt'
    - require:
      - check: checker

thorium_test_no_event.sls

shell_test:
  local.cmd:
    - tgt: saltmaster.example.com
    - func: cmd.run
    - args:
      - cmd: 'echo "thorium success" >> /tmp/thorium.txt'

Versions Report

Salt Version: Salt: 2018.3.0

Dependency Versions: cffi: 1.10.0 cherrypy: 3.2.3 dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.9.6 libgit2: 0.25.1 libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: 2.18 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: 0.25.0 Python: 2.7.14 (default, Dec 14 2017, 15:51:29) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 16.0.2 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.5.2 ZMQ: 4.2.2

System Versions: dist:
locale: UTF-8 machine: x86_64 release: 4.4.41-36.55.amzn1.x86_64 system: Linux version: Not Installed

[Ch3LL]

and what command are you running to set off the thorium reactor? and how are you verifing that it works?

[azelezni]

what command are you running to set off the thorium reactor

salt-call event.fire_master '{"checker":"trigger"}' 'salt/thorium/myminion/test'

and how are you verifing that it works?

I don't understand the question, if I could verify Thorium worked I wouldn't be here :) I can verify that the event reached the master, I have a reactor listening to the same tag and writing to a file whenever that tag is matched. reactor.conf:

reactor:
  - salt/thorium/myminion/test:
    - salt://reactor/test.sls

test.sls:

test:
  local.cmd.run:
    - tgt: saltmaster
    - args:
      - cmd: 'echo "It worked!" >> /tmp/test.txt'

Anyway, after playing around the master configurations I made some progress, it seems thorium doesn't play nice with GitFS. I white-listed only the base env from GitFS, and then thorium started showing up in the logs, and apperently it (Thorium) uses the top.sls file that is used for states and highstate and ignored the thorium_roots option.

After I changed that top.sls file to match what was in /srv/thorium/top.sls it started working, but this is not a solution since that top file is used for highstate.

Here is my environment.conf (with GitFS):

### fileserver ####

thorium_roots:
  base:
    - /srv/thorium

top_file_merging_strategy: merge
#state_top_saltenv: base
default_top: base

fileserver_backend:
  - git

#env_order:
#  - base
#  - dev
#  - ci
#  - qa

gitfs_provider: pygit2

gitfs_env_whitelist:
  - base

gitfs_remotes:
  - git@github.com:example.com/salt-backend.git:
    - name: my-salt-backend
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa
    - root: salt
    - base: master
    - ref_types:
      - branch

[Ch3LL]

apologies I should have clarified my request. I was trying to make sure the way you were attempting to verifying if it was working or not was correct. and that you weren't checking for something else. Anyways, i wonder since you are using gitfs if you were to add the following if it would help?:

fileserver_backend:
  - roots
  - git

[azelezni]

That did that trick, after adding roots thorium started using the top file located at /srv/thorium/top.sls.

This gave me an idea, since thorium doesn't directly support gitfs, it's based on the state system that does support it, with some tweaking I managed to get thorium to use gitfs. Using this config:

fileserver_backend:
  - git
gitfs_provider: pygit2
gitfs_env_whitelist:
  - base
  - thorium
gitfs_remotes:
  - git@github.com:example.com/salt-backend.git:
    - name: salt-backend
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa
    - root: salt
    - base: master
    - ref_types:
      - branch
  - git@github.com:example.com/salt-backend.git:
    - name: thorium-backend
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa
    - root: thorium
    - saltenv:
      - thorium:
        - ref: master

top.sls atthorium/top.sls in repo:

thorium:
  '*':
    - thorium_test_git

Also had to force thorium to use a specific saltenv opts['saltenv'] = 'thorium' in the thorium module, might be nice to allow setting custom opts for thorium engine in the config file.

[Ch3LL]

You could also implement using a mountpoint with gitfs to use gitfs with thorium. Seems you have a grasp around a fix to allow thorium to use a different environment. Mind submitting a PR for that?

[azelezni]

Sure no problem.

[Ch3LL]

Thanks for that! Do you view this issue as closed?

[azelezni]

Yeah, working great!