openbmc / bmcweb

A do everything Redfish, KVM, GUI, and DBus webserver for OpenBMC
Apache License 2.0
156 stars 131 forks source link

bmcweb_persistent_data.json grew so big caused login to fail #167

Closed gtmills closed 3 years ago

gtmills commented 3 years ago
# ls -lh
-rw-r-----    1 root     root        6.1M Nov  6 16:44 bmcweb_persistent_data.json

# grep -o -i csrf_token bmcweb_persistent_data.json | wc -l
32035

Need to delete old sessions. This was a test system running https://github.com/openbmc/openbmc-test-automation that hit this. @anoo1 FYI.

gtmills commented 3 years ago

@edtanous Should we persist a lastUpdated ? https://github.com/openbmc/bmcweb/blob/04e438cbad66838724d78ce12f28aff1fb892a63/include/sessions.hpp#L264

And don't restore the sessions if the session has timed out ? https://github.com/openbmc/bmcweb/blob/c0ea7ae1c502fa2c67b0a58aee05b75581dafa7a/include/persistent_data.hpp#L97

This testing was continue to reboot the bmc / boot, reboot, power off the host. https://github.com/openbmc/openbmc-test-automation/blob/master/extended/obmc_boot_test.robot

gtmills commented 3 years ago

https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/38177 merged which might help a little here. I believe what is happening is openbmc-test-automation is creating a lot of sessions and not cleaning them up and with this particular testing, the bmc reboots at least every hour, resetting the timeout of these sessions so the number of sessions just continues to grow until bmcweb_persistent_data.json gets so large it fills the filesystem.

If we wanted to do more here it would be https://github.com/openbmc/bmcweb/blob/83cf8189e5f9d414efb5c5fa73227e977cf5f84d/include/sessions.hpp#L130

I opened https://github.com/openbmc/openbmc-test-automation/issues/2102 to make changes in test to clean up their sessions.

Closing this issue since don't plan to make any additional changes at this time to bmcweb.