Setup production server for TraceBase

lparsons commented 2 years ago

FEATURE DESCRIPTION

Feature Inspiration

Having a "production" server provides developers and researchers a place where certain standards of stability can be expected. Developers will continue to have a place to showcase and test new features (tracebase-dev).

Feature Description

[x] Production VM - tracebase.princeton.edu
[x] Document deployment practice
[x] Protect the site using Apache CAS auth and allow list (similar to Galaxy)
[x] Setup regular database backups
[x] Setup regular filesystem backups
[x] Setup zabbix monitoring for critical services
[ ] Have critical services managed by systemd

Alternatives Considered

A brief description of any alternative features that may accomplish the same ultimate goal.

Comment

Add any other context or screenshots about the feature request here.

ISSUE OWNER SECTION

Assumptions

List of assumptions made WRT the code
E.g. We will assume input is correct (explaining why there is no validation)

Requirements

List of conditions to be met for the feature
E.g. Every column/row must display a value, i.e. cannot be empty

Limitations

A list of things this work will specifically not do
E.g. This feature will only handle the most frequent use case X

Affected/Changed Components

Files
Environment variables
External executables
Database tables
Cron job settings
Etc.

DESIGN

GUI Change description

Describe changes the user will see.

Code Change Description (Pseudocode optional)

Describe code changes planned for the feature.

Tests

A test should be planned for each requirement (above), where possible.

hepcat72 commented 3 weeks ago

@fkang-pu I recall you said that database backups are happening for TraceBase and I see that it's documented, so I just checked that item in this issue description. Do you think that's appropriate, or is there more to do that I'm not aware of WRT the backups?

@anthropotron - I am unaware of anything that may have been done WRT these items in the issue description above:

Setup regular filesystem backups
Setup zabbix monitoring for critical services
Have critical services managed by systemd

Do these exist? Did Lance ever approach you about any of these goals?

fkang-pu commented 3 weeks ago

@fkang-pu I recall you said that database backups are happening for TraceBase and I see that it's documented, so I just checked that item in this issue description. Do you think that's appropriate, or is there more to do that I'm not aware of WRT the backups?

yes. The cron jobs listed in wifi doc are correct. I get email notifications daily about postgres database dumps (both tb9andtb9-dev).

hepcat72 commented 2 weeks ago

@svenmh - Not sure if this tag will work. I just sent you an invite for the team. But I was wondering if you can speak to any of these admin goals in this issue?

svenmh commented 2 weeks ago

it looks like all I have to do to make it public is to open it up on the firewall. Should I do this?

jcmatese commented 2 weeks ago

it looks like all I have to do to make it public is to open it up on the firewall. Should I do this?

Are the 3 checkboxes (above) completed?

Setup regular filesystem backups
 Setup zabbix monitoring for critical services
 Have critical services managed by systemd

svenmh commented 2 weeks ago

it's a VM, so it gets backed up automatically. Zabbix is monitoring it. And I believe it only requires httpd & postgresql. Both are enabled to start at boot.

hepcat72 commented 2 weeks ago

Sorry for the slow reply. (Been doing focussed debugging.) I take it that, by your response about making the site public, that things like zabbix monitoring only make sense for public-facing sites? We're not ready to open it up to the world yet. So let me ask this:

Is there anything that needs to be done for:

Setup regular filesystem backups
Setup zabbix monitoring for critical services
Have critical services managed by systemd

that can be accomplished before making the site public?

And once it goes public, is there anything further to do that relate to these items?

Sorry for the naïve questions.

hepcat72 commented 2 weeks ago

it's a VM, so it gets backed up automatically. Zabbix is monitoring it. And I believe it only requires httpd & postgresql. Both are enabled to start at boot.

OK. I can check off the zabbix and backup items. So the only remaining items are the put httpd & postgresql under systemd management?

hepcat72 commented 2 weeks ago

What is the frequency of the filesystem backups?

Princeton-LSI-ResearchComputing / tracebase