search

dsa110 / dsa110-issues

Issue tracker for all DSA-110 work
0 stars 0 forks source link

disk space monitoring #206

Open caseyjlaw opened 1 year ago

caseyjlaw commented 1 year ago

Containers sometimes fill up with logs, data, database. It would help to have monitoring of disk space to avoid surprises. Prometheus?

rh-codebase commented 1 year ago

can be handled with prometheus. assign to @kembyct

kembyct commented 1 year ago

I tried the prometheus installation couple time on a test server, but it has not been successful. I wonder if there is a specific SOP I can rely on to get it done. Thanks.

31 10:26:41 server02 systemd[1]: Started Prometheus Time Series Collection and Process> Mar 31 10:26:41 server02 systemd[3923]: prometheus.service: Failed to locate executable /u> Mar 31 10:26:41 server02 systemd[3923]: prometheus.service: Failed at step EXEC spawning /> Mar 31 10:26:41 server02 systemd[1]: prometheus.service: Main process exited, code=exited,> Mar 31 10:26:41 server02 systemd[1]: prometheus.service: Failed with result 'exit-code'.

caseyjlaw commented 1 year ago

@kembyct Did you get any help on setting up Prometheus? I know it would be helpful for many projects, so we'd like to help you learn to do this.

kembyct commented 1 year ago

The prometheus installations have been successful on lwacalim nodes. I have not tried to install it on DSA110 yet. I confused it with LWA one's, and I would love to learn more about how you usually do it. Thanks

caseyjlaw commented 1 year ago

Great! Rick installed it for us before, so I don't know it well. Can you use the same procedure for DSA-110?

kembyct commented 1 year ago

No problem, but I haven't got any chance to connect to any DSA 110 yet. So which node should I ssh to. Thanks

caseyjlaw commented 1 year ago

The main gateway is dsa110maas.ovro.pvt. I know Rick likes to have special services like prometheus on dedicated containers, but for general purpose work, we use lxd110h23.sas.pvt.

kembyct commented 1 year ago

I am going to need an access to the dsa110 system.

rh-codebase commented 1 year ago

@kembyct. I see quite a few ssh keys for you in the authorized_keys files. Are all of them necessary? If not, please trim down and send which key you are now using.

kembyct commented 1 year ago

Here it is: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1uYd2jPkcGsYmQDfnii8XBStr6i40dY7Kr52pi6nElkrU1+M3IIhvNMjyTyX/xRLYX30Q9lqmAfXfYCSPB6gA7F90MVYmQQv6p+Q9lp34ukhqFGh+ScaPHEcCMkYsk008TTTB6NmplzyNQ0Wfu9IrtCRSS/APggEbMZWVCWUGC9TclDTvi8I8VpHra52FAPISDgK33nCx2k+e6npar9IVf5adMZF+ixf2khiPjjH3k1JXVJ3NJJLflARDGjTLFcMltt3zcjXIH14ZLpUllBhbap1iKnR2pzoN4OpL6coG8xsF8ja+Y1Jb4k7/T9iszWnjd6OncNHCfrPcL75013JKw== kemby@tioga.ovro.pvt

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLpyVQ09iy+Egtw/Xj/ydTJc7f+jBcrrFx9RTtrpvST6K8W8AvXbcWBVUEnw1M0C2rFjX49vsLQ2GISV1K+o+vDvK/n7t04XVb16rUzRxi8+c3aavUpZyOFF/oyJQJhqzsQAX6LQeXo7SF828dhF94hiVXlTcGRrZG9iVnjOCapZx87DzejjOY/3CZp3GHTIkNLFmUduLuWUAvx7SdslDuj/oWWyt+r8u0tHCcNiYel92JI4OwfhCadT3nHVOi9n/kctdlyfrjeXgfreMEEEgNm4q5hHCtbqVfayGbIqxXmGgkCjyrzpME6UF/ycxFzlC9OpcOCzMDJmSXuenxqC6CSJj/ppXtPdG/GIzSAxZ9WjsymlpoO7WXNf9jlmgFohsLY1bMJywk/KS67R9l/8ksWOcX0hyOUKW7HfO7j5StmQY7nzmarFOUR1Z3yfJfk2VbD+4e049mVbbbQRallXyr9+1p4XtJQM/FuqzAP4DOFxJr/XqiQhitfPNKh5PHJZc= nagau@ADMINIS-G20O6Q8

I have been only using those twos. The second one is to connect from my laptop when I'm not on the site. Thanks.

rh-codebase commented 1 year ago

keys added. ssh through ubuntu@dsa110.ovro.pvt, then to the lxd110hZZ.sas.pvt machines where ZZ = [00-23] ex. ssh ubuntu@dsa110.ovro.pvt ssh lxd110h00.sas.pvt

kembyct commented 1 year ago

I still can't connect to dsa110 system.