search

OpenIdentityPlatform / OpenAM

OpenAM is an open access management solution that includes Authentication, SSO, Authorization, Federation, Entitlements and Web Services Security.
https://www.openidentityplatform.org/openam
Other
769 stars 149 forks source link

openam agent chokes openAM with create session requests after openAM pod upgrade #303

Closed spetix closed 9 months ago

spetix commented 4 years ago

Describe the bug Upgrade openAM/openDJ pods. After some time, agents configured on application pods start to flood openAM with create token requests. Only way to stop this seems to restart application pod. This problem seems not to affect those pods that are restarted after openAM/openDJ upgrade.

In activity.csv log hundreds of lines like the following related to the agent of the offending pod without any attempt of DELETE:

66442ec3ce5601""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"5f1666442ec3ce5601";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153366";"2020-08-19T21:41:28.643Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153362";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""44a4
6633afba238601""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"44a46633afba238601";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153388";"2020-08-19T21:41:29.019Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153384";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""19d6
4d01263615c201""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"19d64d01263615c201";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153410";"2020-08-19T21:41:29.415Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153406";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""e1b3
a72b4b1daaf101""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"e1b3a72b4b1daaf101";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153432";"2020-08-19T21:41:29.823Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153428";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""af3f
4d9a4616746001""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"af3f4d9a4616746001";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153454";"2020-08-19T21:41:30.316Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153450";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""dcab
079bcbcd3c2401""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"dcab079bcbcd3c2401";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153476";"2020-08-19T21:41:30.686Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153472";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""ee57
87498fd0026801""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"ee5787498fd0026801";"CREATE";;;;;"Session";"/"
"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153498";"2020-08-19T21:41:31.102Z";"AM-SESSION-CREATED";"d034c3c5-ffed-4e40-b7a7-7a69eb894b78-153494";"id=ewpmanagement587dc77b6dkxsdg,ou=agent,dc=openam,dc=forgerock,dc=org";"[""24e7
306aa792ae4801""]";"id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org";"24e7306aa792ae4801";"CREATE";;;;;"Session";"/"

In openDJ situation reflects the log above (snippet is taken from another syste in a moment where other application were acting the same on activity.csv) - some of the offending pod were already been restarted:

root@ems-opendj-0:/opt# /opt/opendj/bin/ldapsearch -h localhost -p 1389 -D 'cn=Directory Manager' -w 'XXXXXXXXX' -b 'ou=tokens,dc=openam,dc=forgerock,dc=org' -s sub '(objectClass=*)' coreTokenUserId | grep coreTokenUserId | sort | uniq -c
      1 coreTokenUserId: id=barb,ou=user,dc=openam,dc=forgerock,dc=org
     26 coreTokenUserId: id=dsameuser,ou=user,dc=openam,dc=forgerock,dc=org
      2 coreTokenUserId: id=esaportal55b4d89756ff69f,ou=agent,dc=openam,dc=forgerock,dc=org
      1 coreTokenUserId: id=ewpauditf485f59949lhtj,ou=agent,dc=openam,dc=forgerock,dc=org
      2 coreTokenUserId: id=ewpdashboards667f49f499sc55h,ou=agent,dc=openam,dc=forgerock,dc=org
   3701 coreTokenUserId: id=ewpdashboards667f49f499spzkj,ou=agent,dc=openam,dc=forgerock,dc=org
      1 coreTokenUserId: id=ewpdpa558b885c989s2zz,ou=agent,dc=openam,dc=forgerock,dc=org
   1556 coreTokenUserId: id=ewpisrestapi74cc766dbfhs789,ou=agent,dc=openam,dc=forgerock,dc=org
      2 coreTokenUserId: id=ewpisrestapi74cc766dbfvsgqn,ou=agent,dc=openam,dc=forgerock,dc=org
      1 coreTokenUserId: id=ewpmanagement587dc77b6d2td75,ou=agent,dc=openam,dc=forgerock,dc=org
      1 coreTokenUserId: id=ewpmdm9bcfdb7bcd888m,ou=agent,dc=openam,dc=forgerock,dc=org
      2 coreTokenUserId: id=ewpmediationd7b8cf47fb5kpc,ou=agent,dc=openam,dc=forgerock,dc=org
   4262 coreTokenUserId: id=ewpmediationd7b8cf47fngbnc,ou=agent,dc=openam,dc=forgerock,dc=org
      1 coreTokenUserId: id=ewppcap67d98c648c6t6rz,ou=agent,dc=openam,dc=forgerock,dc=org
      2 coreTokenUserId: id=ewptracec7998549626z2z,ou=agent,dc=openam,dc=forgerock,dc=org

A workaround that seems to work is to restart all application pods after openAM/openDJ upgrade.

To Reproduce Steps to reproduce the behavior:

  1. Upgrade openAM/openDJ pods
  2. Access an application whose pod has not been restarted after openAM/openDJ. Its agent will start (sometimes after a few hours of usage) to flood openAM with CREATE-SESSION requests.
  3. Openam pod becomes unhealty showing "no healty upstreams". Liveness check on pod fails

Expected behavior We expect that agents renews session properly without flooding openam without the need of application pod restart

Desktop (please complete the following information):

Additional context

vharseko commented 9 months ago

please provide 1) agent config 2) debug log