The content of the $pbsdir/server_name file is not the same when it is set by the ncm-pbsclient component than when it is set by the ncm-pbsserver component. In the later case, an additional "\n" character is added.
When the pbs server has a high load, there may be a long time between the execution of the two components. In this case, the jobs submitted during this period does not contains the right server_name and can not correctly removed from the queue at the end of the execution.
I would suggest to remove the trailing "\n" character in the following line:
https://github.com/quattor/configuration-modules-grid/blob/f9f203d97ae6814988e34b378321d49ec536e175/ncm-pbsserver/src/main/perl/pbsserver.pm#L182
The content of the $pbsdir/server_name file is not the same when it is set by the ncm-pbsclient component than when it is set by the ncm-pbsserver component. In the later case, an additional "\n" character is added. When the pbs server has a high load, there may be a long time between the execution of the two components. In this case, the jobs submitted during this period does not contains the right server_name and can not correctly removed from the queue at the end of the execution. I would suggest to remove the trailing "\n" character in the following line: https://github.com/quattor/configuration-modules-grid/blob/f9f203d97ae6814988e34b378321d49ec536e175/ncm-pbsserver/src/main/perl/pbsserver.pm#L182