bradbishop
commented
4 years ago Another way to help here would be to get rid of all the mapper-wait processes (move them to the application that needs those objects).
Open geissonator opened 4 years ago
bradbishop
commented
4 years ago Another way to help here would be to get rid of all the mapper-wait processes (move them to the application that needs those objects).
Our Hardware CI runs about 20-30 jobs a day test openbmc master gerrit jobs. Approximately 5-10% of our runs fail with this issue. It used to be more intermittent but has started to ramp up. We've always had performance issues in the early startup of the BMC. The removal of python helped but then new functions keep getting added (ssl key generation, certificates, ldap, ...). My theory is that this intermittent issue is related to performance.
The data shows you get 33 seconds for "r = sd_event_loop(loop);" call to return in mapper wait. That times out for all the initial mapper waits but then on the subsequent retry it works within 16 seconds.
I'm thinking about adding a retry to the sd_event_loop(loop); call on a timeout ERRNO.