BlockReplicas API returns not what was requested

[nataliaratnikova]

The API intended logic is https://github.com/dmwm/PHEDEX/blob/07021173f43d0da7910c38cc37f940313adff18b/perl_lib/PHEDEX/Web/API/BlockReplicas.pm#L288-L291

that is if none of block, dataset, node, or create_since is set, the default create_since is set to 1 day ago. If any is set, create_since is set to whatever the parameter is or open. Respective code:

Sometimes however the query returns all data sets subscribed in the last 24 hours even when not intended and confuses the calling application.

Affected version: 2_4_0-pre1 and earlier releases.

[nataliaratnikova]

Findings:

• When called with a missing dataset name: https://cmsweb.cern.ch/phedex/datasvc/json/prod/blockreplicas?dataset= , the API falls back to default behavior. This is consistently reproducible.
• Dynamo plots using this API for monitoring of total number of subscription, have semi-regular spikes, started a few weeks ago, then disappeared: http://dynamo.mit.edu/dynamo/dealermon/main.php?serviceId=2

These were the times (rounded to the quarter-hour) where we queried phedex and got the crazy responses. 2018-10-17 00:30:00 EDT 2018-10-19 02:15:00 EDT 2018-10-20 18:45:00 EDT 2018-10-22 12:00:00 EDT

Since three days I am querying the blockreplicas with the GluGlu dataset, every 20 seconds, and I havent observed a glitch so far. All the JSONs have exactly the same content ...

• Unified:
  • transfer request https://cmsweb.cern.ch/phedex/prod/Request::View?request=1492730 contains non intended data from the recent subscriptions, which indicates a fall-back to default behavior
  • introduced a work around https://github.com/CMSCompOps/WmAgentScripts/pull/330
  • added monitoring for bad returns and recorded a case of wrong output with a non-empty dataset name, coinciding with a frontend error due to expired certificate on the client host

[nataliaratnikova]

Possible solutions:

• A calling client software can check that output strictly corresponds to the input dataset.
• Provide a safe version of the BlockReplicas API for critical applications and to avoid confusions. Make the dataset or block parameter strictly required and disable the 24 hours default behavior. The old API can stay for the backward compatibility with an thing else that may be using the fall-back feature.

[nataliaratnikova]

Second case of misbehavior caught by Unified on Sat Nov 10 04:06 . Again, no errors on the data service side, however, frontend servers that were handling the request have error messages about the expired CN=dmwm/cmsweb.cern.ch certificate (note, this is different from the DN that was used to used with the data service query and belonged to Jean-Roch.)

[bbockelm]

Hi @nataliaratnikova -

What's the error message that was received?

Brian

[nataliaratnikova]

I forwarded it to you in email.

[bbockelm]

For reference, the sanitized version is:

[Sat Nov 10 04:06:14.221283 2018] [ssl:error] [pid 13098] [client $CLIENT_IP] Certificate Verification: Error (10): certificate has expired; subject: $SUBJECT, issuer: $ISSUER, notbefore: Dec  9 15:36:15 2016 GMT, notafter: Jan 13 15:36:15 2018 GMT

One surprising thing I've recently learned is that Apache worker processes never reload CRLs -- meaning busy servers will never reload the CRLs and, in extreme cases, eventually start spewing expiration errors (only fix is to have the parent forcibly reload the child every N requests). I'm not sure this is the case here - but if we're fairly certain the certificate itself isn't expiring, we should look at the average age of the process and compare it to the CRL ages.

[vlimant]

can one explain how that expired cert lead to the query made be changed and replied to with something different than what was requested initially ; sorry but I am missing this point.

The call was for https://cmsweb.cern.ch/phedex/datasvc/xml/prod/blockreplicas?dataset=/MonoHbb_ZpBaryonic_MZp-500_MChi2-1_MChi1-0p5_ctau-100000_13TeV-madgraph/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM and returned block locations for [u'/DoubleMuon/Run2018C-PromptReco-v1/AOD#7057f19b-f644-4fd4-a779-047e63f70ab4', u'/DoubleMuon/Run2018C-PromptReco-v1/AOD#c49a7c8b-525e-460e-b5e4-674ba073369c', u'/DoubleMuon/Run2018C-PromptReco-v1/AOD#fe9bb7ab-0072-4c2c-9c36-995683f9736c', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#018492c6-d733-47e5-ac80-1d63d8b0af5b', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#134e9b0a-a1d5-4587-82e2-22aedf4c997f', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#30d60b8f-0344-4f65-b27b-f6a5368272c0', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#3538588b-5380-4f6f-90e1-523a5d83aec8', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#56bd71ef-4378-4b8e-a826-a867b6f1b51d', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#5f5dc434-6a7a-4a28-9957-2d0febe19430', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#7e78af93-4724-44ab-a976-6a187b3142c4', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#844e11f0-9ef6-4ab3-b1d7-ca5fcd7e52b1', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#93598e28-2ece-4db4-afc1-32e8ed002592', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#b97e9a54-06b7-4b20-b24f-b5f319d79056', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#c1ee19f5-1c11-4bdc-bea6-691de32764a1', u'/EGamma/Run2018A-HcalCalIsoTrkFilter-PromptReco-v3/ALCARECO#de35cfea-d12c-4ca9-98fa-ebccbe78d0b9'] for example

[bbockelm]

@vlimant - my working theory would be Apache has a bug and is spewing crap.

@nataliaratnikova - to make sure that I understand your message correctly, is the CN=dmwm/cmsweb.cern.ch certificate the one that the frontend uses to authenticate with the backend?

[nataliaratnikova]

HI Brian, I do not know much about frontend/backend authentication, I hoped you would know what this DN is for. I will ask via e-group for cmsweb developers and operators, hopefully we get an answer next week.

[amaltaro]

That certificate is "internally" used by CMSWEB services for inter-communication, though the frontend though. Like in ReqMgr2, some ReqMgr2 APIs are internally querying couchdb, those queries are not local, so they again go through the frontends and in order to get those requests authenticated, those services have a copy of the dmwm/cmsweb service certificate in their area.

[bbockelm]

@amaltaro - do we have monitoring for the age of the certificates? Is it safe to assume that these did not really expire?

Back to my earlier comment:

we should look at the average age of the process and compare it to the CRL ages.

The only way I know to test this theory is to have someone log in to the frontend and look at the age of the Apache worker processes. If they are regularly a few days old, we could be hitting this bug.

[amaltaro]

I think Lina is going to reply to this thread, but there is no monitoring. The service certificate gets copied during the deployment to its own service auth area. Meaning either it works all the time, without any room for glitches or very short windows of expiration, OR... I wouldn't discard the change to have a unwanted backend serving requests with an expired certificated (e.g., reqmgr2 on vocms0742, which does not serve any requests, but if it had to, it would have expired certs).

I let Lina comment any further, we should have the backend ip address on the frontend logs and be able to rule out that theory.

[bbockelm]

Gotcha. Since we're not seeing massive failures across the board, I'm going to guess that it's not really expired. I feel like my CRL theory may be going down the right track.

@h4d4 - can you do ps faux | grep httpd on one of the frontend nodes (that's where Natalia is seeing the failure, right)? For example, on one of my development hosts, the output is this:

root     30861  0.0  0.0 273388  9476 ?        Ss   Aug08  11:20 /usr/sbin/httpd -DFOREGROUND
apache   15726  0.0  0.0 287972 11108 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   15727  0.0  0.0 287940 11144 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   15728  0.0  0.0 287980 11108 ?        S    Nov18   0:01  \_ /usr/sbin/httpd -DFOREGROUND
apache   15729  0.0  0.0 287952 11120 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   15730  0.0  0.0 287952 11124 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   23671  0.0  0.0 287988 11136 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   18290  0.0  0.0 287980 11128 ?        S    Nov18   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   25207  0.0  0.0 287980 11084 ?        S    12:09   0:00  \_ /usr/sbin/httpd -DFOREGROUND
apache   26923  0.0  0.0 287916 10944 ?        S    12:18   0:00  \_ /usr/sbin/httpd -DFOREGROUND

So the majority of processes are from yesterday (Nov18); if we had a CRL that expired on 19 November, then that worker process would potentially have similar error messages to what Natalia is seeing. The leap-in-logic is really whether this additionally triggers some other bug related to returning the wrong data. That still feels like a pretty big leap to me!

Anyhow, we could potentially test this by having a custom frontend with one child and doing a steady stream of requests that require authentication for a week or so. Alternately, we could set:

MaxConnectionsPerChild 10000

in the frontend configuration and see if it lowers the average lifetime of the child process -- and see if this bug "magically" disappears.

[h4d4]

@nataliaratnikova @bbockelm @amaltaro Regarding the certificate: As all you know, all requests again CMSWeb services needs to pass by the frontends, and as part of the authentication process, the services/user need to provide a valid certificate. If the certificate is not valid the request is denied at the fronted layer.

CN=dmwm/cmsweb.cern.ch, is a grid host Certificate dmwm/cmsweb.cern.ch for authenticating some CMSWeb services. Accord to deployment scripts[1], more of those queries are agings to CouchDB databases, as for example t0_workloadsummary. The services that use this certificate are: acdcserver crabserver(There was a CouchDB database: ASO, Then certificates exit, but currently ASO was deleted.) reqmon reqmgr2 t0_reqmon workqueue

Therefore, a copy of the certificate needs to be in the authenticated area of the service, in nodes where the service should run. Regarding dmwm certificate expiration date, it’s still valid. Checking for example, at the certificate for workqueue in vocms0740, we got below: [cmsweb@vocms0740 t0_reqmon]$ openssl x509 -in /data/srv/current/auth/workqueue/dmwm-service-cert.pem -dates -subject -noout notBefore=Jan 10 20:13:28 2018 GMT notAfter=Feb 14 20:13:28 2019 GMT subject= /DC=ch/DC=cern/OU=computers/CN=dmwm/cmsweb.cern.ch

And for renewing the certificate, CMSWeb operator receive an email like one month before it expires.

Regarding, ssl:error Certificate Verification, messages in frontends logs, those are in fact showing up there, and comes from vocms0741 and vocms0742, and on both nodes, dmwm certificate for all services listed before are from Dec 14 2016 (this mind in fact expired). But accord to [4], this nodes are dedicated to other services. If there be a mistake with the propagated certificate, the issue will affect directly the services listed above, since their request to GET or PUT data into their CouchDB databases will be denied in the frontends.

Now the question, why those old certificates are there? The answer is the last time that I renewed it, I’ve to propagate it by hand(since that day did not match with the production upgrade, the procedure that includes a step to propagate it). Propagated, means copied from afs private area to the specific authentication area of the node where the service should run. Follow[2],[3],y [4] I propagated the certificate into authenticating area of the services into the nodes that those should be run.

[1] https://github.com/dmwm/deployment/blob/master/t0_reqmon/deploy#L49 [2]https://github.com/dmwm/deployment [3] https://cms-http-group.web.cern.ch/cms-http-group/activity.html [4] https://github.com/dmwm/deployment/blob/master/frontend/backends-prod.txt#L8

[h4d4]

@bbockelm Regarding frontends, proccess I've runned the command on all the 4 nodes, and most of them are from today, as is showed below:

vocms0158:
cmsweb   12678  1.0  0.0 113308  1672 ?        Ss   16:54   0:00          \_ bash -c ps faux | grep httpd
cmsweb   12715  0.0  0.0 112704   948 ?        S    16:54   0:00              \_ grep httpd
root      1537  0.0  0.0  81516 14720 ?        Ss   Nov06  13:20 /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9497  0.0  0.1 104992 24696 ?        S    Nov19   0:07  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 28770  0.0  0.1 106980 26056 ?        S    14:09   0:07  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9305  5.2  0.1 107812 27748 ?        S    16:24   1:36  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9845  5.1  0.1 107712 27700 ?        S    16:29   1:16  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11293  5.7  0.1 106216 26436 ?        S    16:37   1:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11374  4.4  0.1 107276 27076 ?        S    16:38   0:43  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11401  5.6  0.1 107612 27460 ?        R    16:38   0:54  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11453  6.2  0.1 105732 25928 ?        R    16:39   0:58  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11513  5.6  0.1 107568 27236 ?        S    16:40   0:50  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11534  6.6  0.1 107600 27548 ?        S    16:40   0:57  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11557  5.2  0.1 105428 25684 ?        R    16:40   0:45  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11749  6.7  0.1 105388 25636 ?        R    16:43   0:45  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11759  6.7  0.1 107160 26868 ?        R    16:43   0:45  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11822  6.0  0.1 107420 26924 ?        R    16:44   0:37  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11831  6.7  0.1 107412 26888 ?        S    16:44   0:41  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11992  7.1  0.1 107416 27012 ?        S    16:45   0:38  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12006  6.9  0.1 105348 25536 ?        R    16:46   0:37  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12047  6.5  0.1 107244 26856 ?        S    16:46   0:33  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12078  6.6  0.1 105356 25220 ?        S    16:46   0:31  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12188  7.8  0.1 105244 25180 ?        S    16:48   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12191  8.4  0.1 105332 25036 ?        S    16:48   0:32  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12195  7.8  0.1 106956 26400 ?        S    16:48   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12203  7.4  0.1 105236 25300 ?        S    16:48   0:28  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12209  8.6  0.1 107140 26444 ?        S    16:48   0:32  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12213  7.7  0.1 105252 25424 ?        S    16:48   0:28  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12218  6.7  0.1 105308 25252 ?        S    16:48   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12221  8.6  0.1 105392 25296 ?        R    16:48   0:31  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12226  7.7  0.1 106144 26144 ?        R    16:48   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12229  6.6  0.1 105228 25104 ?        R    16:49   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12232  7.0  0.1 107076 26752 ?        S    16:49   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12235  7.7  0.1 105304 25228 ?        R    16:49   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12239  7.4  0.1 105304 25208 ?        S    16:49   0:25  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12254  6.7  0.1 105284 25468 ?        R    16:49   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12266  7.9  0.1 107088 26728 ?        S    16:49   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12271  7.9  0.1 107176 26372 ?        R    16:49   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12272  8.0  0.1 106920 26556 ?        R    16:49   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12274  7.2  0.1 105260 25436 ?        S    16:49   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12278  8.4  0.1 107100 26432 ?        S    16:49   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12281  7.3  0.1 105156 25020 ?        S    16:49   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12324  7.7  0.1 105360 25468 ?        S    16:50   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12326  7.8  0.1 105212 25312 ?        S    16:50   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12327  8.7  0.1 105112 25008 ?        R    16:50   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12610  8.9  0.1 105060 24476 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12612  9.8  0.1 104884 24528 ?        R    16:54   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12613  7.2  0.1 104984 24504 ?        R    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12617 10.2  0.1 105024 24624 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12619  8.6  0.1 105004 24520 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12620  9.8  0.1 104940 24548 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12622  9.4  0.1 104908 24592 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12623  9.3  0.1 104904 24492 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12624  8.7  0.1 104968 24476 ?        R    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12625  9.1  0.1 104968 24668 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf

vocms0760:
cmsweb   13508  1.0  0.0 113308  1672 ?        Ss   16:54   0:00          \_ bash -c ps faux | grep httpd
cmsweb   13545  0.0  0.0 112704   948 ?        S    16:54   0:00              \_ grep httpd
root      1635  0.0  0.0  81432 14716 ?        Ss   Nov06  15:36 /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 21821  0.0  0.1 105668 25860 ?        S    Nov19   0:53  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 11996  6.0  0.1 107536 27332 ?        S    16:36   1:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12067  6.0  0.1 107388 26944 ?        S    16:37   1:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12745  6.4  0.1 107244 26960 ?        S    16:46   0:32  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12809  6.1  0.1 107108 26544 ?        S    16:47   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12876  6.1  0.1 107188 26676 ?        S    16:48   0:25  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12902  6.4  0.1 107236 27104 ?        S    16:48   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 12903  4.4  0.1 107176 26556 ?        S    16:48   0:18  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13040  5.8  0.1 107140 26568 ?        S    16:49   0:18  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13047  4.3  0.1 107088 26748 ?        S    16:49   0:13  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13048  6.3  0.1 107172 26536 ?        S    16:49   0:20  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13049  6.7  0.1 105224 25368 ?        S    16:49   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13099  7.4  0.1 107116 26472 ?        S    16:50   0:22  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13105  6.6  0.1 107052 26212 ?        S    16:50   0:19  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13121  2.9  0.1 106788 25776 ?        S    16:50   0:08  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13141  6.7  0.1 107144 26488 ?        S    16:50   0:17  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13148  6.9  0.1 107176 26408 ?        S    16:50   0:17  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13173  5.9  0.1 107092 26332 ?        S    16:51   0:13  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13183  7.8  0.1 106944 26212 ?        S    16:51   0:17  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13186  6.8  0.1 107072 26680 ?        S    16:51   0:15  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13204  7.1  0.1 107088 26540 ?        S    16:51   0:15  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13254  6.6  0.1 105036 25132 ?        S    16:52   0:11  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13270  7.6  0.1 107084 26132 ?        R    16:52   0:12  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13278  6.2  0.1 106964 25988 ?        S    16:52   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13283  6.7  0.1 107112 26552 ?        S    16:52   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13290  6.0  0.1 104952 24696 ?        S    16:52   0:09  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13301  3.6  0.1 105028 24460 ?        S    16:52   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13306  6.5  0.1 107060 25928 ?        R    16:52   0:08  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13336  4.3  0.1 105032 24620 ?        S    16:52   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13356  5.5  0.1 105004 24560 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13358  4.3  0.1 104988 24488 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13364  1.1  0.1 103600 23060 ?        S    16:53   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13369  6.2  0.1 104944 24432 ?        R    16:53   0:06  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13371  5.4  0.1 104996 24488 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13373  6.6  0.1 106164 25492 ?        S    16:53   0:06  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13374  5.7  0.1 104928 24468 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13375  6.0  0.1 104952 24488 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13376  6.1  0.1 104876 24348 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13377  5.4  0.1 106836 26284 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13378  5.5  0.1 106980 25728 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13385  6.0  0.1 105020 24568 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13386  6.2  0.1 106552 25824 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13387  5.5  0.1 104996 24500 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13388  5.7  0.1 104984 24588 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13390  5.7  0.1 104972 24624 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13392  4.3  0.1 104896 24452 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13393  4.8  0.1 104972 24484 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 13477  3.6  0.1 103576 22948 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf

vocms0162:
cmsweb   32237  0.0  0.0 113308  1668 ?        Ss   16:54   0:00          \_ bash -c ps faux | grep httpd
cmsweb   32274  0.0  0.0 112704   948 ?        S    16:54   0:00              \_ grep httpd
root      1472  0.0  0.0  81520 14720 ?        Ss   Nov06  14:19 /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8125  0.0  0.1 103440 22572 ?        S    Nov19   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 14141  0.0  0.1 103568 22936 ?        S    Nov19   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 30007  5.7  0.1 107864 27752 ?        S    16:27   1:36  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 30738  5.7  0.1 106844 26824 ?        S    16:35   1:08  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 30861  6.3  0.1 105980 26216 ?        S    16:36   1:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 30894  6.1  0.1 107660 27292 ?        S    16:36   1:06  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31112  6.2  0.1 105916 26220 ?        S    16:40   0:55  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31115  5.9  0.1 107184 26840 ?        S    16:40   0:53  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31121  6.3  0.1 107464 27068 ?        S    16:40   0:56  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31122  5.9  0.1 105552 25816 ?        S    16:40   0:53  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31179  5.8  0.1 105532 25512 ?        S    16:41   0:48  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31287  5.8  0.1 105468 25732 ?        S    16:42   0:43  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31336  5.5  0.1 107308 26896 ?        S    16:43   0:38  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31356  6.0  0.1 105436 25340 ?        S    16:43   0:40  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31581  1.9  0.1 105084 25196 ?        S    16:46   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31584  5.9  0.1 107212 26604 ?        S    16:46   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31586  6.6  0.1 107308 26676 ?        R    16:46   0:33  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31587  5.9  0.1 107280 26704 ?        S    16:46   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31592  7.0  0.1 105416 25708 ?        S    16:46   0:35  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31665  6.9  0.1 105304 25248 ?        S    16:47   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31667  7.6  0.1 107292 26644 ?        S    16:47   0:32  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31839  7.9  0.1 105240 25348 ?        R    16:49   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31842  7.9  0.1 105228 25392 ?        S    16:49   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31855  7.4  0.1 105204 25372 ?        S    16:49   0:22  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31868  7.6  0.1 107080 26424 ?        S    16:50   0:22  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31888  8.2  0.1 105144 25240 ?        R    16:50   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31891  8.1  0.1 105212 25408 ?        S    16:50   0:22  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31896  8.1  0.1 105220 25092 ?        S    16:50   0:22  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31897  7.8  0.1 105204 25108 ?        S    16:50   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31904  7.5  0.1 105124 24968 ?        S    16:50   0:20  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 31906  7.6  0.1 105104 25016 ?        R    16:50   0:20  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32079  9.0  0.1 105060 24696 ?        S    16:53   0:09  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32140  7.9  0.1 104908 24788 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32160  4.4  0.1 104960 24424 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32161  7.9  0.1 104884 24372 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32164  7.6  0.1 104888 24388 ?        R    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32165  4.3  0.1 104844 24204 ?        R    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32166  6.7  0.1 104876 24304 ?        R    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32167  6.6  0.1 104868 24324 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32169  8.7  0.1 104940 24332 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32170  7.0  0.1 104992 24532 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32171  6.3  0.1 104856 24312 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32172  5.6  0.1 104844 24232 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32173  7.8  0.1 104840 24300 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32174  5.6  0.1 106168 25696 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32175  6.5  0.1 104876 24368 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32176  8.0  0.1 104980 24428 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32178  8.0  0.1 104844 24308 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32180  8.0  0.1 106828 25524 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32181  5.2  0.1 104904 24316 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32182  2.5  0.1 104920 24340 ?        R    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32183  8.3  0.1 104904 24408 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32185  5.5  0.1 105020 24420 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32186  9.2  0.1 104920 24412 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32187  7.5  0.1 104876 24432 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32188  8.2  0.1 104956 24468 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32190  7.2  0.1 104852 24660 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32196  6.7  0.1 104996 24468 ?        S    16:54   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32208  2.9  0.1 104864 24256 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32212 10.1  0.1 104852 24304 ?        R    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32216  3.8  0.1 103552 22936 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32221  7.1  0.1 103732 23188 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32224  9.2  0.1 103564 22968 ?        R    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32227 11.2  0.1 103692 23072 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32231 14.5  0.1 103560 22980 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 32275  0.0  0.0  94012 11560 ?        R    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf

vocms0164:
cmsweb    9662  1.0  0.0 113308  1668 ?        Ss   16:54   0:00          \_ bash -c ps faux | grep httpd
cmsweb    9699  0.0  0.0 112704   944 ?        S    16:54   0:00              \_ grep httpd
root      1474  0.1  0.0  81512 14708 ?        Ss   Nov06  20:47 /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  3857  0.0  0.1 104828 24260 ?        S    Nov19   0:02  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 15425  0.0  0.1 106804 26120 ?        S    Nov19   0:25  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 15769  0.0  0.1 106496 26008 ?        S    Nov19   0:20  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 15986  0.0  0.1 105088 24908 ?        S    Nov19   0:13  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 16183  0.0  0.1 105168 25380 ?        S    Nov19   0:23  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+ 17576  0.0  0.1 105076 25104 ?        S    Nov19   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8463  5.8  0.1 107304 26720 ?        S    16:42   0:44  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8770  7.5  0.1 107388 27128 ?        S    16:46   0:39  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8845  7.9  0.1 107012 26768 ?        S    16:47   0:36  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8867  7.4  0.1 107000 26188 ?        S    16:47   0:32  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8906  7.1  0.1 107116 26484 ?        S    16:47   0:30  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8938  6.6  0.1 107128 26436 ?        S    16:48   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8961  7.2  0.1 105440 25288 ?        S    16:48   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8982  7.7  0.1 105292 24984 ?        S    16:48   0:29  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8984  7.4  0.1 107192 26844 ?        S    16:48   0:27  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  8985  7.1  0.1 107004 26668 ?        S    16:48   0:26  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9011  8.0  0.1 107148 26632 ?        R    16:49   0:28  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9132  8.7  0.1 105532 25156 ?        S    16:50   0:24  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9138  7.5  0.1 106700 26236 ?        S    16:50   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9153  8.0  0.1 106392 25916 ?        S    16:50   0:21  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9198  7.3  0.1 106932 25956 ?        S    16:50   0:17  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9199  6.5  0.1 107064 26056 ?        S    16:50   0:15  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9217  7.9  0.1 105016 24576 ?        S    16:51   0:18  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9224  7.5  0.1 107040 26076 ?        S    16:51   0:17  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9232  7.3  0.1 107056 26212 ?        S    16:51   0:16  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9234  6.6  0.1 105772 25240 ?        S    16:51   0:14  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9243  7.1  0.1 107052 25968 ?        S    16:51   0:15  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9296  7.9  0.1 105092 24740 ?        S    16:51   0:15  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9312  7.0  0.1 104996 24972 ?        S    16:51   0:13  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9325  6.9  0.1 106644 25888 ?        R    16:51   0:12  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9355  7.3  0.1 106904 25876 ?        S    16:52   0:12  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9378  7.3  0.1 107080 25984 ?        S    16:52   0:12  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9408  7.6  0.1 105220 24848 ?        S    16:52   0:11  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9419  6.9  0.1 105116 24968 ?        S    16:52   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9429  7.3  0.1 104920 24964 ?        S    16:52   0:10  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9493  6.6  0.1 107072 25996 ?        S    16:52   0:08  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9512  6.8  0.1 106880 25736 ?        S    16:53   0:07  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9522  8.4  0.1 104892 24700 ?        S    16:53   0:08  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9528  5.9  0.1 104936 24352 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9531  5.6  0.1 105024 24568 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9543  8.5  0.1 105160 25144 ?        S    16:53   0:07  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9548  6.4  0.1 105152 24644 ?        S    16:53   0:05  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9566  6.0  0.1 105100 24540 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9572  9.4  0.1 104892 24456 ?        S    16:53   0:06  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9576  6.4  0.1 106888 25528 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9579  7.3  0.1 104924 24512 ?        S    16:53   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9582  6.1  0.1 106884 25680 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9588  7.8  0.1 106928 25604 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9590  5.8  0.1 105016 24388 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9591  7.3  0.1 104988 24536 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9593  3.9  0.1 104964 24440 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9596  8.4  0.1 104996 24580 ?        S    16:54   0:04  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9599  8.8  0.1 104944 24444 ?        S    16:54   0:03  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9602  4.3  0.1 104936 24436 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9647 12.2  0.1 103588 23016 ?        S    16:54   0:01  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9651 11.4  0.1 103576 22988 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf
_fronte+  9656 17.0  0.1 103524 22868 ?        S    16:54   0:00  \_ /data/srv/feHG1811h/sw/slc7_amd64_gcc630/external/apache24/2.4.18gsi/bin/httpd -f /data/srv/state/frontend/server.conf

[nataliaratnikova]

Findings:

* When called with a missing dataset name: https://cmsweb.cern.ch/phedex/datasvc/json/prod/blockreplicas?dataset=  , the API falls back to default behavior. This is consistently reproducible.

Apparently this was an intentional feature: https://github.com/dmwm/PHEDEX/blob/d2602abdcdf53195c47e7e1e5391d13d6d059e4e/perl_lib/PHEDEX/Web/Util.pm#L230