Failed to show cgi report,why?

[geostar123]

I succeeded in generating static html report,but got failed for cgi report. I used pgcluu_collectd command as below to collect statistics data,and made a cron task to generate bin file.But httpd server reported errors while opening http://addr1.addr2.addr3.addr4:80/cgi-bin/pgcluu.cgi

command:
pgcluu_collectd -D -i 60 --rotate-daily /var/lib/pgcluu/data -h addr1.addr2.addr3.addr4 -U geoadmin -d postgres -p 5432 crontab -e /5 * /usr/local/bin/pgcluu --cache /var/lib/pgcluu/data

httpd webserver error: [Tue Aug 30 12:17:14.855819 2022] [cgi:error] [pid 13678] [client addr1.addr2.addr3.addr4: 57325] AH01215: FATAL: can't write to /var/lib/pgcluu/data/2022/08/30/pidstat_cp u_stat.bin, No such file or directory ..... Any help will be appreciated!

[darold]

Supposing that you run pgcluu_collectd and pcluu as postgres system user you must execute the following commands:

        /bin/chown -R postgres:www-data /var/lib/pgcluu/data/
        /bin/chmod -R u=rwX,g=rsX,o= /var/lib/pgcluu/data/

[geostar123]

thanks to @darold. I use geoadmin as user,not postgres.I have already open all access (rwx) for folder /var/lib/pgcluu/data.You maybe take a look at web error: can't write to /var/lib/pgcluu/data/2022/08/30/pidstat_cp u_stat.bin, No such file or directory.I found no pidstat_cpu_statbin file existing in folder /var/lib/pgcluu/data/2022/08/30,why?

[geostar123]

@darold.could you provide a cgi example for me like web http://pgcluu.darold.net/example/ showing html ?

[darold]

The HTML outtput is exactly the same as the reports generated by pgcluu, this is just that they are generated dynamically from the *.bin files instead of static HTML.

Actually the CGI should not write to disk so I had a deeper look at the error message above and this the wrong message, it should be a read error. Commit 06fd626 fix this problem.

The problem is that the binary files are not generated by the cron command. You should try to run manually as geoadmin user the command /usr/local/bin/pgcluu --cache /var/lib/pgcluu/data, it will show you the error.

[geostar123]

thanks again.You are right.I mannualy run command "pgcluu --cache /var/lib/pgcluu/data",then it did generated some .bin files,but no pidstat*.bin ,pls see the following:

[geostar123]

In order to generate pidstat*.bin files,I tried running command " /usr/local/bin/pgcluu -i /var/log/sa/sa30 --from-sa-file --cache /var/lib/pgcluu/data".Unfortunately,web page (http://addr1.addr2.addr3.addr4:80/cgi-bin/pgcluu.cgi could not show any graph.

[geostar123]

Now there are pidstat*.bin files in /var/lib/pgcluu/data subfolder,but these bin files' length is less than 100 bytes. The cgi page show nothing as below:

[darold]

Thanks for the additional explanation, commit 76ada65 now verify that the pidstat* binary file exists before.

The CGI doesn't generate reports because the binary files are empty.

[darold]

You can not use a dedicated sar binary file to generate the reports, pgcluu_collectd has already called sar to generate the system statistics.

The pgcluu command should be /usr/local/bin/pgcluu --cache /var/lib/pgcluu/data

[geostar123]

Running command "/usr/local/bin/pgcluu --cache /var/lib/pgcluu/data" will not generate any pidstat*.bin file.

I tried pgcluu with two methods(rpm installation and source code compilation),but got the same result.-:) According to your explanation,why I generate empty bin files?How to fix this problem?

[geostar123]

By the way,my pgcluu version is 3.3.0(both rpm and source code).

[darold]

As I explain to you above, the CGI mode can only be run on a data directory generated by pgcluu_collectd. When you use pgcluu -i /var/log/sa/sa30 --from-sa-file --cache /var/lib/pgcluu/data you are erasing all stats built by pgcluu_collectd.

[geostar123]

Well,thanks for your patience.As I described above:If running pgcluu command without -i and --from-sa-file parameters,there will be no pidstat.bin files generated.If there is no pidstat.bin file in folder,apache httpd server will report errors.So What can I do to fix the issue?

[darold]

Use latest development code from github, I have fixed the error in the commit mentioned above or apply these changes to your current installation.

[geostar123]

ok.let me try.please keep your attention to this issue.thanks again.

[geostar123]

ok.let me try.please keep your attention to this issue.I will not close this issue until it get fixed. thanks again.

[geostar123]

crontab for user geoadmin: /5 * /usr/local/bin/pgcluu --cache /var/lib/pgcluu/data

[geostar123]

@darold,Just now i tried by applying changes https://github.com/darold/pgcluu/commit/06fd626bf1f1c3eb8ade100e81758952cc35c0ed(https://github.com/darold/pgcluu/commit/06fd626bf1f1c3eb8ade100e81758952cc35c0ed) and https://github.com/darold/pgcluu/commit/76ada65daf31a079054fc8a59458f9360f627bfb(https://github.com/darold/pgcluu/commit/76ada65daf31a079054fc8a59458f9360f627bfb) to pgcluu.cgi,there is no pidstat*.bin file generated in /var/lib/pgcluu/data subfolder,why?I assure you that pgbench are always running on remote server addr1.addr2.addr3.addr4.

web server report:

[darold]

The error message still reports "can't write to ..." and in new code after the patches:

$ grep "can't write to" cgi-bin/pgcluu.cgi | wc -l
0

I guess that you have not modified the right CGI file, you should patch the one that is executed by your HTTP server.

[geostar123]

[geostar123]

I commented all 'can't write to....' lines,now web server reported the following:

[geostar123]

The key issue still remains unfixed.There is no pidstat*.bin file generate in /var/lib/pgcluu/data subfolder.With your help I hope this can be fixed.

[darold]

Ok, please do the following:

wget https://github.com/darold/pgcluu/archive/refs/heads/master.zip
unzip master.zip
cd pgcluu-master/
perl Makefile.PL
make
sudo make install
cp -f cgi-bin/pgcluu.cgi /var/www/cgi-bin/

that should override your installion of pgcluu with latest development code.

Then verify with grep "can't write to" /var/www/cgi-bin/pgcluu.cgi | wc -l

Also perhaps you could also try to restart your HTTP server, especially if you use mod_perl.

[geostar123]

ok.let me try

[geostar123]

@darlod.I did these steps you metioned above on a new pc,but got the same.*_As i metioned before,the key issue remains unfixed:pidstat.bin file will not generate in /var/lib/pgcluu/data subfolder,and all generated bin files are almost empty(see picture below)._**

My pgcluu_collectd command line: pgcluu_collectd -D -i 60 --rotate-daily /var/lib/pgcluu/data -h addr1.addr2.addr3.addr3 -U geoadmin -d postgres -p 5432 pgcluu --cache /var/lib/pgcluu/data My crontab task:

/5 * /usr/local/bin/pgcluu --cache /var/lib/pgcluu/data

the latest pgcluu.cgi:

/var/lib/pgcluu/data folder(*.csv):

/var/lib/pgcluu/data/2022/08/31 folder:

httpd webserver report:

[geostar123]

Looking forward to you reply.

[darold]

Thanks for the report, commit 79d2daf might solve this issue. You can just proceed as previously when installing latest development code.

[geostar123]

thanks to @darlod.let me try.

[geostar123]

thanks to @darlod.let me try. I applied the newest patch 79d2daf to pgcluu.cgi,web server did not report errors as metioned above(..Uninitialized $sar_file). As i metioned many times,the key issue remains unfixed:pidstat*.bin file will not generate in /var/lib/pgcluu/data subfolder,and all generated bin files are almost empty(see picture below) while I used the same pgcluu_collectd command line.

pls pay attention to this issue.

[geostar123]

web cgi page:

[geostar123]

apart from pgcluu.cgi bugs,pgcluu_collectd will not generate pidstat*.bin files and other generated bin files are all empty.

[darold]

First you must copied the resources file into your html directory as follow:

sudo mkdir  /var/www/html/pgcluu/
sudo cp -rf cgi-bin/rsc/ /var/www/html/pgcluu/
sudo chown -R www-data: /var/www/html/pgcluu/

the interface will have a better look.

Second, I have installed pgcluu with the command above and create the data directory as follow:

/bin/chown -R postgres:www-data /var/lib/pgcluu/data/
/bin/chmod -R u=rwX,g=rsX,o= /var/lib/pgcluu/data/

then run pgcluu_collectd few minutes:

/usr/local/bin/pgcluu_collectd -i 60 -I /var/lib/pgcluu/data/ -f /tmp/pgcluu_collectd.pid --disable-sar

then build the cache:

/usr/local/bin/pgcluu --cache /var/lib/pgcluu/data --disable-sar

and looking at the CGI screenshot everything is alright.

So my opinion is that your run of pgcluu_collectd is failing but as it is run in daemon mode you do not see the error. Do the following. Stop the daemon:

pgcluu_collectd -k

Run pgcluu_collectd in interactiv mode (without the -D option) and have a look at the errors

pgcluu_collectd -i 10 --rotate-daily /var/lib/pgcluu/data -h addr1.addr2.addr3.addr3 -U geoadmin -d postgres -p 5432

[geostar123]

First,run pgcluu_collectd command without -D option,there is no error occured.

[geostar123]

second:create a folder in /var/www/html and copy cgi-bin/rsc into it.

[geostar123]

according to your steps,all i can do have been done,but got the same:no pidstat*.bin files generated,others are empty,and cgi page shown as below:

[darold]

Well I can not put much more attention to this issue, I'm not able to reproduce the problem.

Maybe you should not use the CGI mode and build static html pages by running periodically pgcluu.

[geostar123]

by the way,must i use postgres user?except geoadmin user It's my advice that you should install pgcluu on non-development computer or virtual machine and verify this issue.

[geostar123]

ok.thank you.