search

p2-inc / keycloak-themes

Themes and theme utilities meant for simple theme customization without deploying a packaged theme
https://phasetwo.io
Other
28 stars 11 forks source link

A vast amount of folders are created in /tmp causing inode starvage #21

Closed gringostar closed 6 months ago

gringostar commented 6 months ago

Hi, we are running a version of keycloak bundled with the phasetwo keycloak-themes utilities on a kubernetes setup. We are experiencing inode starvation on the nodes, and have narrowed down the issue to a vast amount of folders being created in the /tmp folder of the container. It seems to be a lot of seemingly random named empty folders, as the snippet you see here:

drwx------ 2 keycloak     3000 4096 Feb  7 11:13 6888217457948314625
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 16666497599011410436
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 2194727438320025022
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 11109565532642929170
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7045059405371698628
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 10434855246425943235
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7881562790629509270
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8310356290661959999
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8158484512323151611
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7183474864929927987
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 17742112019516436090
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 517507398132951816
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 17213526254849155520
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 3219170250654840034
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 12392673609669382556
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 12256837402144043448
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 2543966388458921869
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9817882537941558180
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7823412426059011019
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 5253353637584204642
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 12092106895361614939
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 16902560068674059387
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9124482098864398693
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7928165158031177216
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 6277286449182196614
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 4368413232393414706
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 5535962613799196828
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9208959293958624080
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8669782482714906514
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8670101507093596484
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 2823111343467424000
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8678464518059225494
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 13475007947718579007
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 14928196600787297796
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7965186811096806551
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8804761063038193411
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 1994944435558107569
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 2063096457190358971
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7135863084320414630
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7395970464151729455
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 5647119207205038527
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 16673791779862873879
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9639927870629870646
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 4526560051324899905
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 147823781901448437
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 4568528101076596283
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8627978574603014782
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 12848976203507138240
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 4703927020762260417
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 18087475116439737204
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 10576731939831528596
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 18249649704622196426
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9254938369582831060
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8157552672873702103
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 7607175376944505594
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 8610325487607964270
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 1297111702902390736
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 12541660459658263017
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 14218173030982759112
drwx------ 2 keycloak     3000 4096 Feb  7 11:13 9178896915726605253
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3731604942917790411
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 1859373387297451333
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3423447017829927743
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 838700807373558131
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 10628508248014236583
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 13537017727227557306
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 14570876310467472993
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 7508940692572585086
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 11220209779543075882
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 5456272065163355518
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 10945391202295228608
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 13101159980058252297
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 283102556641068097
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 7365848465052532158
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 4818434741656324253
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 17361963968092816717
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 2694603592102171005
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 8367857123460478894
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3870196176174547843
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 1989975730529877178
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 7780728339066839117
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3353822948708780341
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 5984349189143189030
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 940908892108645851
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3821097460875450069
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 1782205378589064397
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 18196052192523808531
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 6711255424619955693
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 6354838730717672510
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 5101926847265450365
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 16133191174386819474
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 1198953234585458235
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 18203351962164528080
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 5383030093215646637
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 7252303285517356871
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 13642848165963002448
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 2159965872389797622
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 16999888962925836976
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 260122639178204101
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 10092573819669002059
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 14412988424940407429
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 4620818016629137576
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 7827531609056656542
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 14180494158153208131
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 16984850674880567146
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 9057894762963975239
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 15993124353707372055
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 14780652827642543075
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 11785390261645744327
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 2494822582574532552
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3411141718900717550
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 4536465999128113247
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 3743046641122271575
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 15446366339205774533
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 1554179099711132120
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 9041089313660788313
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 8523272011615143519
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 12154232816580021149
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 14083305094766178814
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 2796099900314221994
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 5997921576505710676
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 15428347365962035369
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 17417814678998909907
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 16048881074826506879
drwx------ 2 keycloak     3000 4096 Feb  7 11:12 6941371936701483386
drwx------ 2 keycloak     3000 4096 Feb  7 11:11 8509969757572283816
drwx------ 2 keycloak     3000 4096 Feb  7 11:11 13568072778659286600

After some investigation, it turns out the folders are created by the AttributeThemeProviderFactory:

https://github.com/p2-inc/keycloak-themes/blob/92a7c527408d9fd54441170661ef1abe46979b06/src/main/java/io/phasetwo/keycloak/themes/theme/AttributeThemeProviderFactory.java#L25C5-L25C41

Are we missing some configuration to make sure this does not happen, or is there a bug in the code?

We are building from the phasetwo-containers version 23.0.6.1706914808.

xgp commented 6 months ago

@gringostar Thanks for the report. It looks like this will happen for large numbers of instantiations of AttributeThemeProviderFactory. tmpdir should get (recursively) deleted in the close().

xgp commented 6 months ago

Pending an answer to the concurrency problem in #21, a temporary fix is out in #25. This works, but may not be as elegant as what you were hoping.

gringostar commented 4 months ago

Sorry for not getting back to you sooner, and commenting on a closed issue. Have not had the chance to try out the fix before now. After running with the latest release for 83 mins, just above 9000 folders have been created in the /tmp folder, with only a few users logging on. There seems to still be an issue. I will try to have another look at the concurrency issues, you mentioned on the previously open PR.

xgp commented 4 months ago

@gringostar Thanks for getting me to look at this again. The creating of the dir was done in the create method of the factory, creating a directory for every instance of the provider and then not deleting it. I updated that. Please check it in the 0.29 release to see if it solves your problem.

gringostar commented 4 months ago

Perfect, I have tested today, and so far it looks like the issue is solved.