refactor(all): align to template; fix bugs and ci

noelmcloughlin commented 4 years ago

BREAKING CHANGE: The data dictionary is simplified and expanded. Retest your states and update pillar data accordingly. For developer convenience, connectors and gui states are introduced. See pillar.example, defaults.yaml, and docs/README.

PR progress checklist (to be filled in by reviewers)

[x] Changes to documentation are appropriate (or tick if not required)
[x] Changes to tests are appropriate (or tick if not required)
[ ] Reviews completed

What type of PR is this?

Primary type

[ ] [build] Changes related to the build system
[ ] [chore] Changes to the build process or auxiliary tools and libraries such as documentation generation
[x] [ci] Changes to the continuous integration configuration
[ ] [feat] A new feature
[x] [fix] A bug fix
[ ] [perf] A code change that improves performance
[x] [refactor] A code change that neither fixes a bug nor adds a feature
[ ] [revert] A change used to revert a previous commit
[ ] [style] Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.)

Secondary type

[x] [docs] Documentation changes
[x] [test] Adding missing or correcting existing tests

Does this PR introduce a `BREAKING CHANGE`?

No.

Related issues and/or pull requests

Addresses. #56, #70, #74, #80, #81, #82

Describe the changes you're proposing

This is a rewrite of the formula to modernize and address various issues.

Align to template formula
Allow granular installations choices (#56 and #60)
Fix Travis CI

By default, the formula installs the Database only

Pillar / config required to test the proposed changes

Debug log showing how the proposed changes work

Documentation checklist

[x] Updated the README (e.g. Available states).
[x] Updated pillar.example.

Testing checklist

[x] Included in Kitchen (i.e. under state_top).
[x] Covered by new/existing tests (e.g. InSpec, Serverspec, etc.).
[x] Updated the relevant test pillar.

Additional context

pull-assistant[bot] commented 4 years ago

Best reviewed: commit by commit

Optimal code review plan (1 warning)

refactor(all): align to template; fix bugs and ci

> `.travis.yml` 75% changes removed in chore(semantic-relea... > `.yamllint` 67% changes removed in chore(semantic-relea...

chore(semantic-release): use standardised structure

Powered by Pull Assistant. Last update fc1ff28 ... 4616edc. Read the comment docs.

noelmcloughlin commented 4 years ago

MacOS trace. Launchd services need attention but mostly working now.

          ID: mongodb-config-usergroup-org.mongo.mongodb.mongod-install-usergroup-present
    Function: group.present
        Name: mongodb
      Result: True
     Comment: Group mongodb is present and up to date
     Started: 03:19:01.825077
    Duration: 25.939 ms
     Changes:   
----------
          ID: mongodb-config-usergroup-org.mongo.mongodb.mongod-install-usergroup-present
    Function: user.present
        Name: mongodb
      Result: True
     Comment: unless condition is true
     Started: 03:19:01.852372
    Duration: 6846.352 ms
     Changes:   
----------
          ID: mongodb-config-usergroup-org.mongo.mongodb.mongos-install-usergroup-present
    Function: group.present
        Name: mongos
      Result: True
     Comment: Group mongos is present and up to date
     Started: 03:19:08.699009
    Duration: 18.783 ms
     Changes:   
----------
          ID: mongodb-config-usergroup-org.mongo.mongodb.mongos-install-usergroup-present
    Function: user.present
        Name: mongos
      Result: True
     Comment: unless condition is true
     Started: 03:19:08.718535
    Duration: 38.969 ms
     Changes:   
----------
          ID: mongodb-install-prerequisites
    Function: pkg.installed
        Name: curl
      Result: True
     Comment: All specified packages are already installed
     Started: 03:19:08.769643
    Duration: 8274.975 ms
     Changes:   
----------
          ID: mongodb-install-prerequisites
    Function: pip.installed
        Name: pymongo
      Result: True
     Comment: Python package pymongo was already installed
              All specified packages are already installed
     Started: 03:19:17.045358
    Duration: 938.725 ms
     Changes:   
----------
          ID: mongodb-install-prerequisites
    Function: file.directory
        Name: /var/lib/mongodb
      Result: True
     Comment: The directory /var/lib/mongodb is in the correct state
     Started: 03:19:17.984824
    Duration: 2.217 ms
     Changes:   
----------
          ID: mongodb-install-prerequisites
    Function: file.directory
        Name: /tmp/downloads
      Result: True
     Comment: The directory /tmp/downloads is in the correct state
     Started: 03:19:17.987570
    Duration: 1.122 ms
     Changes:   
----------
          ID: mongodb-database-mongod-archive-install
    Function: file.directory
        Name: /usr/local/mongodb/mongod-4.2.6
      Result: True
     Comment: The directory /usr/local/mongodb/mongod-4.2.6 is in the correct state
     Started: 03:19:17.989031
    Duration: 4.838 ms
     Changes:   
----------
          ID: mongodb-database-mongod-archive-install
    Function: archive.extracted
        Name: /usr/local/mongodb/mongod-4.2.6
      Result: True
     Comment: All files in archive are already present
     Started: 03:19:17.994727
    Duration: 2343.415 ms
     Changes:   
----------
          ID: mongodb-database-org.mongo.mongodb.mongod-install-service-directory
    Function: file.directory
        Name: /var/lib/mongodb/mongod
      Result: True
     Comment: The directory /var/lib/mongodb/mongod is in the correct state
     Started: 03:19:20.338597
    Duration: 1.341 ms
     Changes:   
----------
          ID: mongodb-database-org.mongo.mongodb.mongod-install-service-launched
    Function: file.managed
        Name: /Library/LaunchAgents/org.mongo.mongodb.mongod.plist
      Result: True
     Comment: File /Library/LaunchAgents/org.mongo.mongodb.mongod.plist is in the correct state
     Started: 03:19:20.340549
    Duration: 9.637 ms
     Changes:   
----------
          ID: mongodb-database-org.mongo.mongodb.mongos-install-service-directory
    Function: file.directory
        Name: /var/lib/mongodb/mongos
      Result: True
     Comment: The directory /var/lib/mongodb/mongos is in the correct state
     Started: 03:19:20.350625
    Duration: 1.35 ms
     Changes:   
----------
          ID: mongodb-database-org.mongo.mongodb.mongos-install-service-launched
    Function: file.managed
        Name: /Library/LaunchAgents/org.mongo.mongodb.mongos.plist
      Result: True
     Comment: File /Library/LaunchAgents/org.mongo.mongodb.mongos.plist is in the correct state
     Started: 03:19:20.352578
    Duration: 6.791 ms
     Changes:   
----------
          ID: mongodb-gui-compass-macapp-download
    Function: cmd.run
        Name: curl -Lo  /tmp/downloads/guicompass1.21.2 https://downloads.mongodb.com/compass/mongodb-compass-community-1.21.2-darwin-x64.dmg
      Result: True
     Comment: unless condition is true
     Started: 03:19:20.359754
    Duration: 8.587 ms
     Changes:   
----------
          ID: mongodb-gui-compass-macapp-download
    Function: module.run
        Name: file.check_hash
      Result: True
     Comment: Module function file.check_hash executed
     Started: 03:19:20.368769
    Duration: 327.743 ms
     Changes:   
              ----------
              ret:
                  True
----------
          ID: mongodb-gui-compass-macapp-download
    Function: file.absent
        Name: /tmp/downloads/guicompass1.21.2
      Result: True
     Comment: State was not run because onfail req did not change
     Started: 03:19:20.697256
    Duration: 0.003 ms
     Changes:   
----------
          ID: mongodb-gui-compass-macapp-install
    Function: macpackage.installed
        Name: /tmp/downloads/guicompass1.21.2
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 03:19:20.697623
    Duration: 0.003 ms
     Changes:   
----------
          ID: mongodb-gui-compass-macapp-install
    Function: file.managed
        Name: /tmp/mac_shortcut.sh
      Result: True
     Comment: File /tmp/mac_shortcut.sh updated
     Started: 03:19:20.698035
    Duration: 2488.322 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,6 +1,6 @@
                   #!/usr/bin/env bash

                  -Source="/Applications/Robo 3T.app"
                  +Source="/Applications/MongoDB Compass Community.app"
                   Destination="/Users/bobby/Desktop"
                   diskutil resetUserPermissions / bobby
                   /usr/bin/osascript -e "tell application \"Finder\" to make alias file to POSIX file \"$Source\" at POSIX file \"$Destination\""
----------
          ID: mongodb-gui-compass-macapp-install
    Function: cmd.run
        Name: /tmp/mac_shortcut.sh
      Result: True
     Comment: Command "/tmp/mac_shortcut.sh" run
     Started: 03:19:23.186820
    Duration: 1725.333 ms
     Changes:   
              ----------
              pid:
                  36397
              retcode:
                  0
              stderr:
              stdout:
                  Ready to reset user permissions on disk1s5 Macintosh HD
                  Started verify/repair permissions on disk1s5 Macintosh HD
                  User permissions have been reset
                  Finished verify/repair permissions on disk1s5 Macintosh HD
                  alias file MongoDB Compass Community alias 26 of folder Desktop of folder bobby of folder Users of startup disk
----------
          ID: mongodb-gui-robo3t-macapp-download
    Function: cmd.run
        Name: curl -Lo  /tmp/downloads/guirobo3t1.3.1 https://download-test.robomongo.org/mac/robo3t-1.3.1-darwin-x86_64-7419c40.dmg
      Result: True
     Comment: unless condition is true
     Started: 03:19:24.912750
    Duration: 9.713 ms
     Changes:   
----------
          ID: mongodb-gui-robo3t-macapp-install
    Function: macpackage.installed
        Name: /tmp/downloads/guirobo3t1.3.1
      Result: True
     Comment: State was not run because none of the onchanges reqs changed
     Started: 03:19:24.923532
    Duration: 0.004 ms
     Changes:   
----------
          ID: mongodb-gui-robo3t-macapp-install
    Function: file.managed
        Name: /tmp/mac_shortcut.sh
      Result: True
     Comment: File /tmp/mac_shortcut.sh updated
     Started: 03:19:24.923947
    Duration: 9.006 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,6 +1,6 @@
                   #!/usr/bin/env bash

                  -Source="/Applications/MongoDB Compass Community.app"
                  +Source="/Applications/Robo 3T.app"
                   Destination="/Users/bobby/Desktop"
                   diskutil resetUserPermissions / bobby
                   /usr/bin/osascript -e "tell application \"Finder\" to make alias file to POSIX file \"$Source\" at POSIX file \"$Destination\""
----------
          ID: mongodb-gui-robo3t-macapp-install
    Function: cmd.run
        Name: /tmp/mac_shortcut.sh
      Result: True
     Comment: Command "/tmp/mac_shortcut.sh" run
     Started: 03:19:24.933271
    Duration: 1434.328 ms
     Changes:   
              ----------
              pid:
                  36416
              retcode:
                  0
              stderr:
              stdout:
                  Ready to reset user permissions on disk1s5 Macintosh HD
                  Started verify/repair permissions on disk1s5 Macintosh HD
                  User permissions have been reset
                  Finished verify/repair permissions on disk1s5 Macintosh HD
                  alias file Robo 3T alias 23 of folder Desktop of folder bobby of folder Users of startup disk
----------
          ID: mongodb-gui-robo3t-macapp-install-symlink-robo3t
    Function: file.symlink
        Name: /usr/local/bin/robo3t
      Result: True
     Comment: onlyif condition is false
     Started: 03:19:26.367868
    Duration: 16.594 ms
     Changes:   
----------
          ID: mongodb-connectors-kafka-archive-install
    Function: file.directory
        Name: /usr/local/mongodb/kafka-1.1.0
      Result: True
     Comment: The directory /usr/local/mongodb/kafka-1.1.0 is in the correct state
     Started: 03:19:26.385064
    Duration: 4.512 ms
     Changes:   
----------
          ID: mongodb-connectors-kafka-archive-install
    Function: archive.extracted
        Name: /usr/local/mongodb/kafka-1.1.0
      Result: True
     Comment: All files in archive are already present
     Started: 03:19:26.390356
    Duration: 12907.595 ms
     Changes:   
----------
          ID: mongodb-config-file-etc-file-directory
    Function: file.directory
        Name: /etc/mongodb
      Result: True
     Comment: The directory /etc/mongodb is in the correct state
     Started: 03:19:39.298425
    Duration: 1.002 ms
     Changes:   
----------
          ID: mongodb-config-file-org.mongo.mongodb.mongod-file-managed
    Function: file.managed
        Name: /etc/mongodb/mongod.yml
      Result: True
     Comment: File /etc/mongodb/mongod.yml is in the correct state
     Started: 03:19:39.300179
    Duration: 32.667 ms
     Changes:   
----------
          ID: mongodb-config-file-org.mongo.mongodb.mongos-file-managed
    Function: file.managed
        Name: /etc/mongodb/mongos.yml
      Result: True
     Comment: File /etc/mongodb/mongos.yml is in the correct state
     Started: 03:19:39.333647
    Duration: 10.677 ms
     Changes:   
----------
          ID: mongodb-config-install-mongod-environ_file
    Function: file.managed
        Name: /etc/default/mongod.sh
      Result: True
     Comment: File /etc/default/mongod.sh is in the correct state
     Started: 03:19:39.344763
    Duration: 36.145 ms
     Changes:   
----------
          ID: mongodb-service-running-prerequisites
    Function: file.managed
        Name: /etc/init.d/disable-transparent-hugepages
      Result: True
     Comment: onlyif execution failed, bad type passed
              unless condition is true
     Started: 03:19:39.381392
    Duration: 25.664 ms
     Changes:   
----------
          ID: mongodb-service-running-prerequisites
    Function: cmd.run
        Name: echo never >/sys/kernel/mm/transparent_hugepage/enabled
      Result: True
     Comment: onlyif condition is false
     Started: 03:19:39.409581
    Duration: 1.319 ms
     Changes:   
----------
          ID: mongodb-service-running-database-org.mongo.mongodb.mongod-install-datapath
    Function: file.directory
        Name: /var/lib/mongodb/mongod
      Result: True
     Comment: The directory /var/lib/mongodb/mongod is in the correct state
     Started: 03:19:39.411293
    Duration: 1.696 ms
     Changes:   
----------
          ID: mongodb-service-running-database-org.mongo.mongodb.mongod-unmasked
    Function: service.unmasked
        Name: org.mongo.mongodb.mongod
      Result: True
     Comment: onlyif condition is false
     Started: 03:19:39.413529
    Duration: 28.867 ms
     Changes:   
----------
          ID: mongodb-service-running-database-org.mongo.mongodb.mongod
    Function: service.running
        Name: org.mongo.mongodb.mongod
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/state.py", line 1981, in call
                  **cdata['kwargs'])
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/loader.py", line 1977, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/states/service.py", line 431, in running
                  before_toggle_status = __salt__['service.status'](name, sig, **status_kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 507, in status
                  output = list_(runas=runas)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 336, in list_
                  runas=runas)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 295, in launchctl
                  return __utils__['mac_utils.launchctl'](sub_cmd, *args, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/utils/mac_utils.py", line 303, in launchctl
                  ret = __salt__['cmd.run_all'](cmd, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 932, in _run_all_quiet
                  success_retcodes=success_retcodes)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 422, in _run
                  user_shell = __salt__['user.info'](runas)['shell']
              NameError: name '__salt__' is not defined
     Started: 03:19:39.445774
    Duration: 5.641 ms
     Changes:   
----------
          ID: mongodb-service-running-database-org.mongo.mongodb.mongos-unmasked
    Function: service.unmasked
        Name: org.mongo.mongodb.mongos
      Result: True
     Comment: onlyif condition is false
     Started: 03:19:39.452805
    Duration: 31.718 ms
     Changes:   
----------
          ID: mongodb-service-running-database-org.mongo.mongodb.mongos
    Function: service.running
        Name: org.mongo.mongodb.mongos
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/state.py", line 1981, in call
                  **cdata['kwargs'])
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/loader.py", line 1977, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/states/service.py", line 431, in running
                  before_toggle_status = __salt__['service.status'](name, sig, **status_kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 507, in status
                  output = list_(runas=runas)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 336, in list_
                  runas=runas)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/mac_service.py", line 295, in launchctl
                  return __utils__['mac_utils.launchctl'](sub_cmd, *args, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/utils/mac_utils.py", line 303, in launchctl
                  ret = __salt__['cmd.run_all'](cmd, **kwargs)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 932, in _run_all_quiet
                  success_retcodes=success_retcodes)
                File "/usr/local/Cellar/salt/3000.2/libexec/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 422, in _run
                  user_shell = __salt__['user.info'](runas)['shell']
              NameError: name '__salt__' is not defined
     Started: 03:19:39.487077
    Duration: 13.471 ms
     Changes:   

Summary for local
-------------
Succeeded: 36 (changed=5)
Failed:     2

noelmcloughlin commented 4 years ago

MacOS issues could be investigated as a separate issue.

bobby@work mongodb % ls /usr/local/mongodb/mongod-4.2.6/bin/mongod
/usr/local/mongodb/mongod-4.2.6/bin/mongod

bobby@work mongodb % cat /Library/LaunchAgents/org.mongo.mongodb.mongod.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.mongo.mongodb.mongod</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/mongodb/mongod-4.2.6/bin/mongod </string>
      <string>--config </string>
      <string>/etc/mongodb/mongod.conf</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardErrorPath</key>
    <string>/dev/null</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
  </dict>
</plist>

bobby@work mongodb % cat /etc/mongodb/mongod.yml 
########################################################################
# File managed by Salt at <salt://mongodb/files/default/config.yml.jinja>.
# Your changes may be overwritten.
########################################################################

net:
  port: 27017
storage:
  dbPath: /var/lib/mongodb/mongod

bobby@work mongodb % cat /etc/mongodb/mongos.yml
########################################################################
# File managed by Salt at <salt://mongodb/files/default/config.yml.jinja>.
# Your changes may be overwritten.
########################################################################

sharding:
  configDB: rs1/127.0.0.1:27018

bobby@work mongodb % ps -ef | grep mongo
  501 40481 53147   0  3:36am ttys004    0:00.00 grep mongo

bobby@work mongodb % launchctl list | grep mongo                             
-   78  org.mongo.mongodb.mongod
-   78  org.mongo.mongodb.mongos

myii commented 4 years ago

Thanks for all of this, @noelmcloughlin -- merged with standardising commit (4616edc) on top.

noelmcloughlin commented 4 years ago

Thanks @myii hugely appreciated.

saltstack-formulas-travis commented 4 years ago

:tada: This PR is included in version 1.0.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

saltstack-formulas / mongodb-formula