How to deploy and configure on miner OS without GUI?

[fanfu13]

My miner works in Ubuntu 20.04 Server version (no graphic user interface). After installation, _curl -s -o - http://localhost:9100/metrics -s |grep "lotus_info" output nothing while /usr/local/bin/lotus-exporter-farcaster.py_ output correctly without mistakes. I guess it's due to ubuntu server version. My question is how to deploy and configure on miner ubuntu server version.

[s0nik42]

farcaster is developped on ubuntu server. You don't need graphical interface.

Do you see something running just : curl -s -o - http://localhost:9100/metrics

[s0nik42]

you need at least to have prometheus node exporter install on the machine

then grafana and prometheus can be deployed on another server

[fanfu13]

farcaster is developped on ubuntu server. You don't need graphical interface.

Do you see something running just : curl -s -o - http://localhost:9100/metrics

Yes. This is the last page of the output of the command.

[fanfu13]

/var/log/syslog something as followed: Dec 9 18:38:01 worker105 CRON[3507296]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:38:01 worker105 CRON[3507295]: (CRON) info (No MTA installed, discarding output) Dec 9 18:39:01 worker105 CRON[3507300]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:39:01 worker105 CRON[3507299]: (CRON) info (No MTA installed, discarding output) Dec 9 18:40:01 worker105 CRON[3507308]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:40:01 worker105 CRON[3507307]: (CRON) info (No MTA installed, discarding output) Dec 9 18:40:55 worker105 systemd[1]: Starting Collect apt metrics for prometheus-node-exporter... Dec 9 18:40:55 worker105 bash[3507339]: awk: cmd. line:1: warning: regexp escape sequence `\"' is not a known regexp operator Dec 9 18:40:56 worker105 systemd[1]: prometheus-node-exporter-apt.service: Succeeded. Dec 9 18:40:56 worker105 systemd[1]: Finished Collect apt metrics for prometheus-node-exporter. Dec 9 18:41:01 worker105 CRON[3507366]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:41:01 worker105 CRON[3507365]: (CRON) info (No MTA installed, discarding output) Dec 9 18:42:01 worker105 CRON[3507427]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:42:01 worker105 CRON[3507426]: (CRON) info (No MTA installed, discarding output) Dec 9 18:43:01 worker105 CRON[3507484]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:43:01 worker105 CRON[3507482]: (CRON) info (No MTA installed, discarding output) Dec 9 18:43:10 worker105 systemd-resolved[3472561]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 9 18:43:14 worker105 systemd-resolved[3472561]: message repeated 37 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.] Dec 9 18:44:01 worker105 CRON[3507493]: (worker) CMD (if /usr/local/bin/lotus-exporter-farcaster.py > /var/lib/prometheus/node-exporter/farcaster.prom.$$; then mv /var/lib/prometheus/node-exporter/farcaster.prom.$$ /var/lib/prometheus/node-exporter/farcaster.prom; else rm /var/lib/prometheus/node-exporter/farcaster.prom.$$; fi ) Dec 9 18:44:01 worker105 CRON[3507492]: (CRON) info (No MTA installed, discarding output) Dec 9 18:45:01 worker105 CRON[3507498]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

[s0nik42]

can you run the script manually from your lotus user ? /usr/local/bin/lotus-exporter-farcaster.py

[fanfu13]

Yes. The last page of the output is as follow(I use f0XXXX to represent my miner_id):

lotus_miner_storage_capacity { miner_id="f0XXXX", storage_id="ee1bb55b-2e5e-47f0-a0a8-d915b9116674" } 502467059712 lotus_miner_storage_available { miner_id="f0XXXX", storage_id="ee1bb55b-2e5e-47f0-a0a8-d915b9116674" } 284545888256 lotus_miner_storage_reserved { miner_id="f0XXXX", storage_id="ee1bb55b-2e5e-47f0-a0a8-d915b9116674" } 0 lotus_miner_storage_info { miner_id="f0XXXX", storage_id="f3193fab-bae8-43d8-b2b9-90307bfbec26", storage_url="http://10.10.100.149:1551/remote", storage_host_name="10.10.100.149", storage_host_ip="10.10.100.149", storage_host_port="1551", weight="10", can_seal="True", can_store="False", path="" } 1 lotus_miner_storage_capacity { miner_id="f0XXXX", storage_id="f3193fab-bae8-43d8-b2b9-90307bfbec26" } 7619330605056 lotus_miner_storage_available { miner_id="f0XXXX", storage_id="f3193fab-bae8-43d8-b2b9-90307bfbec26" } 6908559306752 lotus_miner_storage_reserved { miner_id="f0XXXX", storage_id="f3193fab-bae8-43d8-b2b9-90307bfbec26" } 0 lotus_miner_storage_info { miner_id="f0XXXX", storage_id="f907704a-451f-4cde-9530-9e121929735e", storage_url="http://10.10.100.103:1551/remote", storage_host_name="10.10.100.103", storage_host_ip="10.10.100.103", storage_host_port="1551", weight="10", can_seal="True", can_store="False", path="" } 1 lotus_miner_storage_capacity { miner_id="f0XXXX", storage_id="f907704a-451f-4cde-9530-9e121929735e" } 3149314711552 lotus_miner_storage_available { miner_id="f0XXXX", storage_id="f907704a-451f-4cde-9530-9e121929735e" } 2697983602688 lotus_miner_storage_reserved { miner_id="f0XXXX", storage_id="f907704a-451f-4cde-9530-9e121929735e" } 0 lotus_scrape_duration_seconds { collector="Storage" } 3.4015250205993652 lotus_scrape_duration_seconds { collector="All" } 5.0686140060424805 lotus_scrape_execution_succeed { } 1

[s0nik42]

Ok so the farcaster works. Which means the data flow is broken somewhere. Let's try this :

cat /var/lib/prometheus/node-exporter/farcaster.prom=> Should return the same output as your message before (terminating by ' lotus_scrape_execution_succeed { } 1') curl -s -o - http://localhost:9100/metrics -s |grep node_exporter_build_info"=> Should return something otherwize node_exporter is not configured in prometheus

[fanfu13]

cat /var/lib/prometheus/node-exporter/farcaster.prom => cat: /var/lib/prometheus/node-exporter/farcaster.prom: No such file or directory. Futher check found /var/lib/prometheus/node-exporter/ diretory, but didn't find farcaster.prom in the directory.

curl -s -o - http://localhost:9100/metrics -s |grep node_exporter_build_info" => # HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built. # TYPE node_exporter_build_info gauge node_exporter_build_info{branch="debian/sid",goversion="go1.13.8",revision="0.18.1+ds-2",version="0.18.1+ds"} 1

[s0nik42]

I created a check.sh script to assist new users. Could you try it ?

git pull chmod +x check.sh ./check.sh

[fanfu13]

Some problems Check if worker has write access to /var/lib/prometheus/node-exporter/ : [ KO ] Cannot right to folder /var/lib/prometheus/node-exporter/ TODO : Option 1/ rerun the install script OR Option 2/ manually give right permission to worker to /var/lib/prometheus/node-exporter/

PS:My lotus daemon & miner worked under user worker.

[fanfu13]

So I need to extend the user worker to privileged user like root?

[fanfu13]

sudo chown -R worker:worker /var/lib/prometheus/node-exporter/ Then reinstall lotus-farcaster. sudo ./check.sh worker return all OK Check if worker has write access to /var/lib/prometheus/node-exporter/ : [ OK ] Check if metrics are exposed in /var/lib/prometheus/node-exporter/ : [ OK ] Check if /var/lib/prometheus/node-exporter//farcaster.prom is owned by worker : [ OK ] Check if /var/lib/prometheus/node-exporter//farcaster.prom creation date : Check if prometheus-node-exporter is running properly : [ OK ] Check for farcaster data in prometheus-node-exporter : [ OK ]

curl -s -o - http://localhost:9100/metrics -s |grep "lotus_info" return # HELP lotus_info lotus daemon information like adress version, value is set to network version number # TYPE lotus_info gauge lotus_info{miner_id="f0XXXX",network="mainnet",version="1.2.2+git.93d26195f.dirty+myscheduler20201203"} 8

So my problems get solved. Thanks a lot!

[fanfu13]

Right now I have a new problem, how to "Import in Grafana the relevent dashboard file from ./lotus-farcaster/grafana-dashboard" in ubuntu server version. I got a lot of informatinon to import grafana from Internet, but all are about in GUI mode.

[s0nik42]

You've done the hard part.

Grafana is standalone web service you install on a server your miner or another one.

So you install grafana somewhere https://grafana.com/docs/grafana/latest/installation/debian/ Then you configure the prometheus prometheus datasources (2 clicks) Then you import the dashboard Menu (+) / Import

[fanfu13]

I had already installed grafana and prometheus in the miner ubuntu version (without GUI). I guess I can do it in GUI mode. But I really don't know in how to it in cmd mode. For example, 2 clicks is probably in GUI mode. So I have to have a server installed ubuntu version with GUI? That's my concern now.

[s0nik42]

No you don't need, you can connect to grafana with any webrowser from another PC

[fanfu13]

Got it. Thanks

No you don't need, you can connect to grafana with any webrowser from another PC

[s0nik42]

does it work ?

[fanfu13]

does it work ?

Sorry, I didn't get the goal as I don't have a server with a web browser right now. Currently, we are planning to migrate storage from old cephfs to customized cephfs, and I don't have any spare serve to install a server with web browser. But when after finishing the storage migration, I will have servers to installed web browers. The migration will take a 1-2 weeks. I will have a try at that time.

[s0nik42]

You don't need to install the webserver, grafana has a builtin webserver binding port 3000

[fanfu13]

Great! Then you configure the prometheus prometheus datasources (2 clicks) could be elaborated in what you said in slack You’ve got to add prometheus as a datasource in grafana “Configuration” / “Datasource” / Add Prometheus / Fill the IP Address. But I still don't get this one. On ubuntu cmd mode, as far as I know, I can only use grafana-cli or grafana-server commands. I don't find the command for grafana “Configuration”.