coding-to-music / coding-to-music.github.io

https://pandemic-overview.readthedocs.io/en/latest/index.html
MIT License
2 stars 8 forks source link

Cloud Home Base Setup - Data ingestion, processing, storage, display #213

Open coding-to-music opened 3 years ago

coding-to-music commented 3 years ago

Running with systemctl

Content Status Start Stop Local CloudPod BigPod
Grafana systemctl grafana-server Start Stop 3000 3100
Redis systemctl redis Start Stop 6379
InfluxDB systemctl influxdb Start Stop 8086
InfluxDB Sandbox systemctl Start Stop 3010
Backstage yarn start Start Stop 3000
Cockpit Status Start Stop 9090 36867
Prometheus Docker Start Stop
Node Status Start Stop 8086
Postgres systemctl postgresql Start Stop 5432
MongoDB systemctl mongodb Start Stop 27017

Currently Running with Docker

Content Status Start Stop Local CloudPod BigPod
cAdvisor docker Start Stop 8080
Chronograf docker Start Stop 8888
kapacitor docker Start Stop 9092
Telegraf tcp 8082 docker Start Stop 8082
Telegraf tcp 8092 docker Start Stop 8092
Telegraf tcp 8094 docker Start Stop 8094
InfluxDB systemctl influxdb Start Stop 8086
InfluxDB Entrypoint systemctl Start Stop 3010

4860 8082 8086 8088 8092 8094 8125 9092

See what's running

sudo systemctl status redis mongod grafana-server influxdb telegraf prometheus postgresql docker

sudo systemctl status grafana-server influxdb telegraf postgresql docker

docker ps

Currently running with docker

Visualizers:

Scrapers:

Databases:

export metrics from chromebook, cellphone, Raspberry Pi

Using environment variables in telegraf.conf to connect to influx

influx CLI configs are only available to influx commands. They automatically populate the --host, --org, and --token flags required by most commands. telegraf cannot use the influx CLI configurations.

If you’re using Telegraf to write to InfluxDB Cloud or OSS 2.x, use the influxdb_v2 output plugin. Provide your InfluxDB 2.x host, token, and org in your telegraf.conf. You can either hardcode them or use environment variables:

Store credentials in environment variables

# file ~/.bash_aliases

export INFLUX_HOST=http://localhost:8086
export INFLUX_TOKEN=YourInfluxDBAuthToken
export INFLUX_ORG=myOrg
export INFLUX_BUCKET=myBucket

Use the environment variables in your /etc/telegraf/telegraf.conf

[[outputs.influxdb_v2]]
  urls = ["$INFLUX_HOST"]
  token = "$INFLUX_TOKEN"
  organization = "$INFLUX_ORG"
  bucket = "$INFLUX_BUCKET"
# file /etc/telegraf/telegraf.conf

 [[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  ## urls exp: http://127.0.0.1:9999
  urls = ["$INFLUX_HOST"]

  ## Token for authentication.
  token = "$INFLUX_TOKEN"

  ## Organization is the name of the organization you wish to write to; must exist.
  organization = "$INFLUX_ORG"

  ## Destination bucket to write into.
  bucket = "$INFLUX_BUCKET"

HTML Table example in a Readme

https://raw.githubusercontent.com/imthaghost/goclone/master/README.md https://github.com/imthaghost/goclone/blob/master/README.md

Gary Frederick
Tha Ghost

πŸ’»
Ben Lafferty
Ben Lafferty

πŸ’»
Omar Sagoo
Omar Sagoo

πŸ’»
Gary Frederick
Tha Ghost

πŸ’»
Ben Lafferty
Ben Lafferty

πŸ’»
Omar Sagoo
Omar Sagoo

πŸ’»
coding-to-music commented 3 years ago

See what is running

sudo systemctl status 

● docker-ubuntu-s-1vcpu-2gb-nyc1-01 State: running Jobs: 0 queued Failed: 0 units Since: Sat 2021-08-14 20:47:19 UTC; 13min ago CGroup: / β”œβ”€370 bpfilter_umh β”œβ”€user.slice β”‚ └─user-0.slice β”‚ β”œβ”€session-1.scope β”‚ β”‚ β”œβ”€1285 sshd: root@notty β”‚ β”‚ β”œβ”€1407 -bash β”‚ β”‚ β”œβ”€1418 bash β”‚ β”‚ β”œβ”€1461 sh /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 --conne> β”‚ β”‚ β”œβ”€1468 /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/node /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/vs/server/main.j> β”‚ β”‚ β”œβ”€1524 /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/node /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/bootstrap-fork -> β”‚ β”‚ β”œβ”€1552 /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/node /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/bootstrap-fork -> β”‚ β”‚ β”œβ”€1563 /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/node /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/bootstrap-fork -> β”‚ β”‚ β”œβ”€1617 /usr/bin/bash β”‚ β”‚ β”œβ”€2483 /usr/bin/bash β”‚ β”‚ β”œβ”€2516 /usr/bin/bash β”‚ β”‚ β”œβ”€2539 /usr/bin/bash β”‚ β”‚ β”œβ”€2548 /usr/bin/bash β”‚ β”‚ β”œβ”€2557 /usr/bin/bash β”‚ β”‚ β”œβ”€2566 /usr/bin/bash β”‚ β”‚ β”œβ”€2990 sleep 180 β”‚ β”‚ β”œβ”€3132 /bin/sh -c "/root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/vs/base/node/cpuUsage.sh" 2516 β”‚ β”‚ β”œβ”€3133 /bin/bash /root/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/out/vs/base/node/cpuUsage.sh 2516 β”‚ β”‚ β”œβ”€3136 sleep 1 β”‚ β”‚ β”œβ”€3137 sudo systemctl status β”‚ β”‚ β”œβ”€3138 systemctl status β”‚ β”‚ └─3139 pager β”‚ └─user@0.service … β”‚ └─init.scope β”‚ β”œβ”€1295 /lib/systemd/systemd --user β”‚ └─1306 (sd-pam) β”œβ”€init.scope β”‚ └─1 /sbin/init └─system.slice β”œβ”€irqbalance.service β”‚ └─769 /usr/sbin/irqbalance --foreground β”œβ”€influxdb.service β”‚ └─792 /usr/bin/influxd β”œβ”€epmd.service β”‚ └─763 /usr/bin/epmd -systemd β”œβ”€containerd.service … β”‚ └─788 /usr/bin/containerd β”œβ”€systemd-networkd.service β”‚ └─624 /lib/systemd/systemd-networkd β”œβ”€systemd-udevd.service β”‚ └─667 /lib/systemd/systemd-udevd β”œβ”€cron.service β”‚ └─753 /usr/sbin/cron -f β”œβ”€system-serial\x2dgetty.slice β”‚ └─serial-getty@ttyS0.service β”‚ └─803 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 β”œβ”€docker.service … β”‚ └─895 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock β”œβ”€polkit.service β”‚ └─854 /usr/lib/policykit-1/polkitd --no-debug β”œβ”€networkd-dispatcher.service β”‚ └─771 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers β”œβ”€multipathd.service β”‚ └─539 /sbin/multipathd -d -s β”œβ”€do-agent.service β”‚ └─760 /opt/digitalocean/bin/do-agent --syslog β”œβ”€accounts-daemon.service β”‚ └─748 /usr/lib/accountsservice/accounts-daemon β”œβ”€systemd-journald.service β”‚ └─353 /lib/systemd/systemd-journald β”œβ”€atd.service β”‚ └─787 /usr/sbin/atd -f β”œβ”€unattended-upgrades.service β”‚ └─869 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal β”œβ”€ssh.service β”‚ └─845 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups β”œβ”€snapd.service β”‚ └─777 /usr/lib/snapd/snapd β”œβ”€rsyslog.service β”‚ └─773 /usr/sbin/rsyslogd -n -iNONE β”œβ”€systemd-resolved.service β”‚ └─643 /lib/systemd/systemd-resolved β”œβ”€telegraf.service β”‚ └─782 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d β”œβ”€rabbitmq-server.service β”‚ β”œβ”€ 772 /usr/lib/erlang/erts-12.0.3/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -s> β”‚ β”œβ”€ 861 erl_child_setup 32768 β”‚ β”œβ”€1210 inet_gethost 4 β”‚ └─1211 inet_gethost 4 β”œβ”€grafana-server.service β”‚ └─764 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=deb cfg:default.paths.logs=/var/log/grafana> β”œβ”€dbus.service β”‚ └─755 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only β”œβ”€systemd-timesyncd.service β”‚ └─583 /lib/systemd/systemd-timesyncd β”œβ”€droplet-agent.service β”‚ └─2081 /opt/digitalocean/bin/droplet-agent -syslog β”œβ”€system-getty.slice β”‚ └─getty@tty1.service β”‚ └─811 /sbin/agetty -o -p -- \u --noclear tty1 linux └─systemd-logind.service └─780 /lib/systemd/systemd-logind

coding-to-music commented 3 years ago

Grafana

+++ title = "Install on Debian/Ubuntu" description = "Install guide for Grafana on Debian or Ubuntu" keywords = ["grafana", "installation", "documentation"] aliases = ["/docs/grafana/latest/installation/installation/debian"] weight = 200 +++

Install on Debian or Ubuntu

https://github.com/grafana/grafana/blob/main/docs/sources/installation/debian.md

This page explains how to install Grafana dependencies, download and install Grafana, get the service up and running on your Debian or Ubuntu system, and also describes the installation package details.

Note on upgrading: While the process for upgrading Grafana is very similar to installing Grafana, there are some key backup steps you should perform. Read [Upgrading Grafana]({{< relref "upgrading.md" >}}) for tips and guidance on updating an existing installation.

Hosted Grafana

You can run Grafana on your own hardware or use Grafana Cloud and get Grafana without the overhead of installing, maintaining, and scaling your observability stack. The free forever plan includes Grafana, 10K Prometheus series, 50 GB logs and more. Create a free account to get started.

1. Download and install

You can install Grafana using our official APT repository, by downloading a .deb package, or by downloading a binary .tar.gz file.

Install from APT repository

If you install from the APT repository, then Grafana is automatically updated every time you run apt-get update.

Grafana Version Package Repository
Grafana Enterprise grafana-enterprise https://packages.grafana.com/enterprise/deb stable main
Grafana Enterprise (Beta) grafana-enterprise https://packages.grafana.com/enterprise/deb beta main
Grafana OSS grafana https://packages.grafana.com/oss/deb stable main
Grafana OSS (Beta) grafana https://packages.grafana.com/oss/deb beta main

Note: Grafana Enterprise is the recommended and default edition. It is available for free and includes all the features of the OSS edition. You can also upgrade to the full Enterprise feature set, which has support for Enterprise plugins.

To install the latest Enterprise edition:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add this repository for stable releases:

echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Add this repository if you want beta releases:

echo "deb https://packages.grafana.com/enterprise/deb beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

After you add the repository:

sudo apt-get update
sudo apt-get install grafana-enterprise

To install the latest OSS release:

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add this repository for stable releases:

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Add this repository if you want beta releases:

echo "deb https://packages.grafana.com/oss/deb beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

After you add the repository:

sudo apt-get update
sudo apt-get install grafana

Install .deb package

If you install the .deb package, then you will need to manually update Grafana for each new version.

  1. On the Grafana download page, select the Grafana version you want to install.
    • The most recent Grafana version is selected by default.
    • The Version field displays only finished releases. If you want to install a beta version, click Nightly Builds and then select a version.
  2. Select an Edition.
    • Enterprise - Recommended download. Functionally identical to the open source version, but includes features you can unlock with a license if you so choose.
    • Open Source - Functionally identical to the Enterprise version, but you will need to download the Enterprise version if you want Enterprise features.
  3. Depending on which system you are running, click Linux or ARM.
  4. Copy and paste the code from the installation page into your command line and run. It follows the pattern shown below.
sudo apt-get install -y adduser libfontconfig1
wget <.deb package url>
sudo dpkg -i grafana<edition>_<version>_amd64.deb

Install from binary .tar.gz file

Download the latest .tar.gz file and extract it. The files extract into a folder named after the Grafana version downloaded. This folder contains all files required to run Grafana. There are no init scripts or install scripts in this package.

wget <tar.gz package url>
sudo tar -zxvf <tar.gz package>

2. Start the server

This starts the grafana-server process as the grafana user, which was created during the package installation.

If you installed with the APT repository or .deb package, then you can start the server using systemd or init.d. If you installed a binary .tar.gz file, then you need to execute the binary.

Start the server with systemd

To start the service and verify that the service has started:

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Configure the Grafana server to start at boot:

sudo systemctl enable grafana-server.service

Start the server with init.d

To start the service and verify that the service has started:

sudo service grafana-server start
sudo service grafana-server status

Configure the Grafana server to start at boot:

sudo update-rc.d grafana-server defaults

Execute the binary

The grafana-server binary .tar.gz needs the working directory to be the root install directory where the binary and the public folder are located.

Start Grafana by running:

./bin/grafana-server web

Package details

Next steps

Refer to the Getting Started guide for information about logging in, setting up data sources, and so on.

Configure Grafana

Refer to the Configuration page for details on options for customizing your environment, logging, database, and so on.