search

marinebon / sdg14

Products for Sustainable Development Goal 14 on Life in the Sea
http://marinebon.github.io/sdg14
3 stars 0 forks source link

add mbon vm to IMaRS puppetmaster? #21

Closed 7yl4r closed 6 years ago

7yl4r commented 7 years ago

We could add the mbon server as a node on IMaRS's puppet configuration. I have a very small configuration I'm adding to all our nodes that would add:

  1. monitoring via telegraf to push resource usage & security info to our graphite server
  2. a few security features like a fail2ban setup

Honestly, I haven't developed any love for puppet in my few months using it and am happy to leave management of this server in your capable hands, @bbest . But I thought I would offer in case this arrangement might help lighten your load. What do you think?

bbest commented 7 years ago

Hi @7yl4r, I'm unfamiliar with puppet, telegraf, graphite, fail2ban; ie all the technologies mentioned. Feel free to provide links or give me a ring to chat. Happy to defer to you on management of server loads - would be great to have a more user friendly view of CPU and memory. I can ask more server tech savvy friends for recs too.

7yl4r commented 7 years ago

Here's a screenshot from graphite for mbon's hypervisor:

image

and one from my security dashboard:

image

I guess the core of what I'm asking is: would you like for me to dig into the mbon server a bit more to add features I'm already pushing to the other servers, or would you prefer to avoid the potential risk and complication in the mbon setup?

bbest commented 7 years ago

Hi @7yl4r,

Yeah, these look like good features to have on the server. Please feel free to proceed.

By the way, a tech startup friend recommends the free version of New Relic.

bbest commented 6 years ago

Hi @7yl4r,

I can't seem to ssh ben@mbon.marine.usf.edu and all web services are down. Can you try to bring up the server please?

7yl4r commented 6 years ago

There was a power outage last night and one of our switches died. I'm working to bring everything back up now.

On Thu, Jul 20, 2017 at 8:16 AM, Ben Best notifications@github.com wrote:

Hi @7yl4r https://github.com/7yl4r,

I can't seem to ssh ben@mbon.marine.usf.edu and all web services are down. Can you try to bring up the server please?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14/issues/21#issuecomment-316685364, or mute the thread https://github.com/notifications/unsubscribe-auth/ABAK_oMzbE_xcOTfYBb0jVjWyPi_Luhyks5sP0UqgaJpZM4OHCA8 .

bbest commented 6 years ago

Ok, thanks Tylar!

On Thu, Jul 20, 2017 at 6:31 AM, Tylar notifications@github.com wrote:

There was a power outage last night and one of our switches died. I'm working to bring everything back up now.

On Thu, Jul 20, 2017 at 8:16 AM, Ben Best notifications@github.com wrote:

Hi @7yl4r https://github.com/7yl4r,

I can't seem to ssh ben@mbon.marine.usf.edu and all web services are down. Can you try to bring up the server please?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14/issues/21#issuecomment-316685364, or mute the thread https://github.com/notifications/unsubscribe-auth/ABAK_oMzbE_ xcOTfYBb0jVjWyPi_Luhyks5sP0UqgaJpZM4OHCA8 .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14/issues/21#issuecomment-316703761, or mute the thread https://github.com/notifications/unsubscribe-auth/ACtLCTqnkOvRiHxWVfS_qwEIQu2kUlWfks5sP1bFgaJpZM4OHCA8 .

7yl4r commented 6 years ago

puppet has been added to the mbon vm and it has been connected to our puppetmaster. telegraf is not working (probably a permissions error), so no advanced logging for now. Next time I can come back around to it I will push metrics into this dashboard.

In the meantime the only two things that may be of use to you are fail2ban & etckeeper:

# to see fail2ban at work:
tylarmurray@mbon:/etc$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed: 195
|  `- File list:    /var/log/auth.log
`- Actions
   |- Currently banned: 4
   |- Total banned: 4
   `- Banned IP list:   116.31.116.6 59.45.175.29 221.194.47.236 221.194.47.233

# etckeeper auto-tracks all your /etc/ configs in a git repo
tylarmurray@mbon:/etc$ sudo git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   telegraf/telegraf.conf

no changes added to commit (use "git add" and/or "git commit -a")

You might consider adding your IP range to the fail2ban ignoreip list so you don't accidentally lock yourself out if you mistype your password several times in a row.:

tylarmurray@mbon:/etc$ sudo fail2ban-client set sshd addignoreip 131.247.0.0/16
These IP addresses/networks are ignored:
|- 127.0.0.1/8
|- 131.247.0.0/16
`- 192.168.0.0/16
7yl4r commented 6 years ago

monitoring is up: http://graphite.marine.usf.edu/dashboard/#mbon

image