Closed IvanKuchin closed 7 years ago
OK. I appreciate the reproducer.
I will have a look at this over the weekend.
It is surprising wrt Autobahn. When I did the last fix I updated the logs on libwebsockets.org from autobahn, they are all fine.
https://libwebsockets.org/clients/index.html
Autobahn strictly uses echo semantics though. And it always takes data as fast as it provides it. So there are probably significant gaps between what that synthetic test actually stresses and what might have trouble hiding in the real world.
I started looking at this and noticed that the fixes on the server side rx state machine did not get added to the client side one... since the communication on the one connection stalls it can be either side responsible.
I pushed a patch aligning the client rx state machine with the server one on v2.1-stable, v2.2-stable and master, can you see if this makes any difference before I go any further into it?
This probably sheds light on why Autobahn is happy as well, we only test the server side of lws on it.
Will run tests tomorrow
I got the stable-2.2.0 and it still stuck.
I was thinking how to makes reproducing more easy than described in my first message. I ended up with far more easy procedure based on investigation of my team. We tested a lot of files and found a pattern , if take max compressed archive with lot of repeatable patterns in archived files, it brings the issue on the spot.
Send "killer string" to socket_status:
test-server ----------- Killer string 1:
socket_status.send("{ \"RequestType\":\"SendImage\", \"RequestID\":\"283463389\", \"toType\":\"toUser\", \"toID\":\"1036\", \"fileType\":\"image/jpeg\", \"blob\":\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAGqAoADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD9U6KKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArlviX43Pw78H3fipdHl1RreSGJLSKQRtK0kqxgBiDjlvSuprhvjNpmoav4INnpljPdznU9Ok8qCMu2xbuJmbA5wFBJPYA0AL4r+K2j+G/hxH8Rra0k1G3ureOezto3CPOXXcFBwcHbk9O1bmg+L9H12wW7juEimSygvrmAtlreOVCy7jjngN+VeOal4L8Vz3Wv8Ag99DvG0Hw5b317pEixMUupLpS0UUf94w5ZMD1FWdLfW/CFxqcF34S126l1vwxpttZfZdPllU3EUMitFIyqREcuOXIHXninYD0y8+KPgCwNkt34ntU/tCGO4tyA7BopPuOSBhQc8bsVYXxtpEc+tHULm2tbPRVheW5afIKyAkEjHy9OOTnNeEHQNX0bw/pTQaH4l0/W38NabbNCNFkvbS/kjgUeTMqrmF1bKncVx1rZ1Lwz4tuV1bVL3wpdSJDf8Ah+/urCGBmFzFAxaeOIf8tNv90ZzgDvRYD2fw34w8NeLoppvDurRXqwNslChlZD1GVYAj8q0rq9tLIRtd3CRCVxGhc4BY9Bn3rlfBl/Z61reqazp/g660q3kjhiW8u7SS0muyoOR5UiqwVc4DEc89q2fFQSTRprVtFk1VrjESWyjhmPQs38AHXdkYx60gNivPb74uwWXhPW/EX/CPzyXmjal/ZQ05Zh5k85ZFjCtjjcZFxxXXeGdO1LSdDtNP1bUPtt1CmJJvXngZ74HGcDOM4FeYXfhPXm+Nq28elXDeG7yaHXbi68s+SLmGORRGW6bi7RNjr8hoA6XU/i3p1pYeHr+w043a+INPk1NMziMQwIiNuY7T/fA/OtS8+J3gnSntLbWtftbO6u4IpxESzhA4BG5gMKPc4ryPSvBHipIvGdheaDei20HT7jRtB/ct/pMLzSSK0Yx8w8t4k47oaj8XaLqVglzJpmjeIbfV7rRLSAWw0WS/sdUdIiBFJsX9yQSVJZhgHPanYD1B/jD4PsfEWseHNdvo9Nm0q7itUeRiwn8yGOQN8q/IP3m3k9R1ra8ceLoPBfgzVfGTWpvYdMs3vPKSQL5qqucBsHGfXFebDwpq15YfE681LwvIuoaotuLdRbl/NK2EAKxHHzgSBxx3Bre+IukavqHwA1TRbPTbq41GXw75C2scTNM0vkAbAgGS2eMYzmgDt9c1xNF0CfXXiRlhjWQo8mwckDlsHHX0rMf4leCYNUg0O78QW0OoT+WogO4gO+Nql8bQTkYBIrjfFvjJfFvgjUfDeleE/Fq3s1qgQXHh68gQlXTI3PGBnr3rjdc0rWtM1K8Ph3QdeXVbq6tJW0250iS50/UGUIPN89V2wEYOSzfKVziiwHsPhL4k+GfGer6zoujXLvcaLcfZ5gyMA3yg7gSOmTj8K6qvPfh9aT6V418b2d3ot5bG81NL22uDZusE0Jt4l+WXGwkMrZUHI9K9CpAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVma34m8OeGvsf/CQ67YaZ/aNyllafa7hIvtFw/3Yk3EbnPZRyaANOiisbQfGfhHxTc39l4a8T6XqtxpUohvorO7jme1kOcLIFJKHg8HHQ0AbNFZ2u+ItA8MWa6j4j1qx0u1eVIFmvLhYUMjnCoGYgbieg6mi08R+H7/VrnQbLW7G41KzjSa4s47hWmhjb7rOgOVB7EjmgDRoorO1/wAQ6D4V0uXXPE2tWOladAVEt3ezrDChZgq5diAMkgDnqRQBo0UgORkUtABRVbUtT07RrCfVNWvreys7VDJPcTyCOONB1ZmPAHuaNO1LT9XsYNT0q9gvLO5QSQzwSB45EPRlYcEe4oAs0VT1fWNJ8P6Zcazrup2un2FonmT3V1KsUUS+rMxAA9zU9tc297bRXlnPHPBOiyxSxsGV0YZDAjggggg0AS0UUmRnGeaAFooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooqrqGpWumQ+dcsQCcKBySaaTk7ITairstUVz3/AAmdh/z63H/jv+NH/CZ2H/Prcf8Ajv8AjW31ar/KZfWKfc6Giue/4TOw/wCfW4/8d/xo/wCEzsP+fW4/8d/xo+rVf5Q+sU+50NFc9/wmdh/z63H/AI7/AI0f8JnYf8+tx/47/jR9Wq/yh9Yp9zoaK57/AITOw/59bj/x3/GtDTNcstVLJBuWRRkow5xUyoVIK8kVGtCTsmaNFFFZGgUUUUAFFFFABXin7YfhS78TfAbxBeaUCup+HVj16ykUZaOS1cSkr7lVYfjXtdVtRsLXVLC50y+hWW3u4nhljboyMMEH8DQBxNz8UNOb4IH4tQyrHaz6AurREH7peEMg/wC+mAr52/Zu8OXnwh+KfhZtT3xn4seHLi9u1JO3+0Y5DcA/UxMwA9qxNG1PUX+COl/s0vcNJrFt46m8ITg9Xs7aYzE/TyvK9sGvaf2ptPTwl4J8LfEvS4WB+HGuWGolYx8xsg4imT6eU7E/SgBn7Qif8Jp8Ufhb8KU/eQTanJ4h1KP0gtgBG3/fbH8q6zw7rXw8j+M/jO3stIltPENlpdrNq2oSSnypbfLbAATgbcEk4rmPh40fj39pfx146jdZ7DwtYWnhrT5lOV8woJ7gA/70gH4Vm6DDpFx+0b8XLfX7mO30yXwzZpeTSSCNY4T5gdix4UAZ57UAbEH7S2r67YyeJfAvwY8TeIfDCFzHq0UkcX2iNTgyRRN8zrwcHIzXK/tQ+P8Aw78Sf2QNf8X+Grh3sriWzVllXZJDIt5EHjcdmB6irvg3wv8AtE/BnwxZ6B4BPhj4ieEbGEDSkkujZXv2Xqi7yDG+FxghuRiuE+Mvivwf4x/Y78djw34Tl8MXVtq8EOs6TKuJLe+a7hMhPJB3cEEcGgD2LU/2htUjt5tY8J/CHxL4j8O2u7fq9u0cccqL96SFG+aROCQeMjpXo/gHx34f+JPhSx8Y+GLhpbC/Qsu9drowOGRh2YHgitXTNNsdM0q10mxtkitLaBIIoVGFWNVwFA9McV4l+xt8vw28QQrwkPjDV0Reyr5q8D86AO2/aF/5Ij40/wCwRN/KpPgF/wAkX8G/9giD/wBBqP8AaF/5Ij40/wCwRN/KpPgF/wAkX8G/9giD/wBBoAn+Nt74T074UeJb7xzpcupaDDZFr+0ico8se5flBBBHOO9Y3iH4w+Gfh/4c8Jabo3h/UtV1HxDZxLoWh2QDTyRLCG5ZjhVVcZY1D+1X/wAm8eOv+wWf/Q1rktd+Hep+Lrb4aeLPh5440vRvHnhbQIns7W+Hmw3NrNbxrKskaneFPHzAH9aAOn8P/H+ZvF2neC/iJ8OtY8GX2tFk0uW6lSe3unUZKCRcbWx2IryTxZ8WPF1p+1VodxD8JvFM/wBj0TULaOxjeLddruT9+g3Y2jHfnmuq174jeNPDus+HbL9o/wCEumHSm1SKHT/EWk3v2m3t71uI2eMhZI89Bwa1vEP/ACeL4Q/7FbUv/Qo6APcrOeS6tIbmW3e3eWNXaKT70ZIztOO46VNRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXKeNSfNtRnjax/UV1dcp41/wBda/7rfzFdOD/jI58V/CZzVeear8Q9V0r4tWHg6e3g/sa8tlDT4PmR3D58vJ6YYgL9WFeh15/4t+HOo+IdY1fVrW/t4JLnTYobB23boLuKQSRyHA+6GVTxzxX0eF9lzNVdrf0/keFiPacqdPe5Qb4sXMPxG1fRbiGIeH9Js2Zp1UmWSdCBJjnGFZtv1U11Ws/ELw1oKeZqc8sajS5NX3bMj7OhQHv97Mi8Vx8fwf1MWcEMmq2rTto89reTYbMt7LIZXlAx90uSfXmn/wDCAePdSujfaxceHo3h8PzaNDAvmTxu7PEdz7lX5SI8EDkZ4zXVKGFk009Fo/Pz+ZzxniIp3Wr/AK/A2n+JNneQwG3t77S5jfQWzx39n8zrIpK7drYwcdcnGOlWrP4m+H77V49LtrXUmimuTaRX/wBm/wBFeYdUD5znIxnGM965DTvhN4lW4M00+n6fafbbW5XT4Lya4iTyg+91aRQQW3gbeny1veGPCXjXw+bTQFvdK/sKyvJLlZlLm6ljZy4jKFdo+ZuWDHipqU8Mk+V/j/V/QcJ4htcyO/rV8LkjWoAD1D5/75NZVavhf/kNwfR//QTXk1v4cvRnp0f4kfU7uiiivBPZCiiigAooooAKKKKAPEbP9mu3tf2kLv47HX1ayngLx6P5JxFetFHG9wGzjJEYzxnnrXqHjzwlZePfBWueC9RIW31uwnsJGIztEiFdw9xnNb1FAHl/7O/wZl+B/wAPh4Tv9eGt6ncXk99f6j5Rj+0SyOSCQSTwu1eT2p0HwVtp/iH428Wa3qEV7pfjPSItIn0/yirLGoYPl88hgxHAFenUUAeC+G/hh+0R8OdFh8D+CPiV4fvvD9kgttNm1nS3lvLK3UYSPcjqsm0YA3L0Apus/suyan8E/EXw0HjBpNc8VX8Wp6prdzb7vNuFmRziNSMLhNoGeM175RQAijaoX0GK8/8Agp8L7j4TeG9T0G51iPUm1DW73VhIkJjCLOwITBJyRjr3r0GigDm/iP4Tl8d+BNc8HQ3q2b6vZvarOybxGW7lQRn868h8L/CT9qDwj4e0/wAM6R8dPC62WmQLbwB/C+5gijjJMnNfQVFAHhmsfCL46eNfBviXwX8QPi1oWo2euaebSE2ugfZzBIWB3kh8sMDGOK0PGHwM128bwh4o8B+MF0Pxh4Q09NNivHtvNtr238tUkimiJ5VtoI5yp6GvY6KAPDL/AOEXxa+Juo6Snxj8ZaIdB0i9i1AaVolg8Qu54jmNpZJHZtoPO1cVr/FL4ReMvEPxB8PfFD4c+MLLRNa0O1uLGSO+sftMFzBNjIIDKQRtGCDXrlFAENmt0lpCl9KklwsaiV0XarPjkgdhntU1FFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFZ+r6RDq0IR2KOnKOO1aFFVGTg+aO4pRUlZnKf8IVN/wBBBP8Av2f8aP8AhCpv+ggn/fs/411dFb/XK3cw+q0uxyn/AAhU3/QQT/v2f8aP+EKm/wCggn/fs/411dFH1yt3D6rS7HKf8IVN/wBBBP8Av2f8aP8AhCpv+ggn/fs/411dFH1yt3D6rS7HKf8ACFTf9BBP+/Z/xrT0bw7FpUhuHl82XGAcYArYoqZ4mrNcrZUcPTg7pBRRRWBsFFFFABRRRQAVFcPcRqDbwLK2eQX24H5GpaKAKX2jVP8AoGx/+BH/ANjR9o1T/oGx/wDgR/8AY1dooApfaNU/6Bsf/gR/9jR9o1T/AKBsf/gR/wDY1dooApfaNU/6Bsf/AIEf/Y0faNU/6Bsf/gR/9jV2igCl9o1T/oGx/wDgR/8AY0faNU/6Bsf/AIEf/Y1dooApfaNU/wCgbH/4Ef8A2NH2jVP+gbH/AOBH/wBjV2igCl9o1T/oGx/+BH/2NH2jVP8AoGx/+BH/ANjV2igCl9o1T/oGx/8AgR/9jR9o1T/oGx/+BH/2NXaKAKX2jVP+gbH/AOBH/wBjR9o1T/oGx/8AgR/9jV2igCl9o1T/AKBsf/gR/wDY0faNU/6Bsf8A4Ef/AGNXaKAKX2jVP+gbH/4Ef/Y0faNU/wCgbH/4Ef8A2NXaKAKX2jVP+gbH/wCBH/2NH2jVP+gbH/4Ef/Y1dooApfaNU/6Bsf8A4Ef/AGNH2jVP+gbH/wCBH/2NXaKAKX2jVP8AoGx/+BH/ANjR9o1T/oGx/wDgR/8AY1dooApfaNU/6Bsf/gR/9jR9o1T/AKBsf/gR/wDY1dooApfaNU/6Bsf/AIEf/Y0faNU/6Bsf/gR/9jV2igCl9o1T/oGx/wDgR/8AY0faNU/6Bsf/AIEf/Y1dooApfaNU/wCgbH/4Ef8A2NH2jVP+gbH/AOBH/wBjV2igCl9o1T/oGx/+BH/2NH2jVP8AoGx/+BH/ANjV2igCl9o1T/oGx/8AgR/9jR9o1T/oGx/+BH/2NXaKAKX2jVP+gbH/AOBH/wBjR9o1T/oGx/8AgR/9jV2igCl9o1T/AKBsf/gR/wDY0faNU/6Bsf8A4Ef/AGNXaKAKX2jVP+gbH/4Ef/Y0faNU/wCgbH/4Ef8A2NXaKAKX2jVP+gbH/wCBH/2NH2jVP+gbH/4Ef/Y1dooApfaNU/6Bsf8A4Ef/AGNH2jVP+gbH/wCBH/2NXaKAKX2jVP8AoGx/+BH/ANjR9o1T/oGx/wDgR/8AY1dooApfaNU/6Bsf/gR/9jR9o1T/AKBsf/gR/wDY1dooApfaNU/6Bsf/AIEf/Y0faNU/6Bsf/gR/9jV2igCl9o1T/oGx/wDgR/8AY0faNU/6Bsf/AIEf/Y1dooApfaNU/wCgbH/4Ef8A2NH2jVP+gbH/AOBH/wBjV2igCl9o1T/oGx/+BH/2NH2jVP8AoGx/+BH/ANjV2igCl9o1T/oGx/8AgR/9jR9o1T/oGx/+BH/2NXaKAKX2jVP+gbH/AOBH/wBjR9o1T/oGx/8AgR/9jV2igCl9o1T/AKBsf/gR/wDY0faNU/6Bsf8A4Ef/AGNXaKAKX2jVP+gbH/4Ef/Y0faNU/wCgbH/4Ef8A2NXaKAKX2jVP+gbH/wCBH/2NH2jVP+gbH/4Ef/Y1dooApfaNU/6Bsf8A4Ef/AGNOSfUWdRJp8aqSMsJ84HrjFW6KACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACioZry0t/8AX3Mcf+8wFUZvEekxcfaN5/2VJosBqUVz03jC3GRBau3+8QKozeLb9/8AVRRxj86fKwOvorg5tc1Sb7124HoOKjtdUvba4WcXDtgjILZBFPlA9AoooqQCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACori5gtYzLcSqijualrkPFsztfpAWO1EBA9zTSuBqzeK9NjyIxJIfYYFUpvGMh/497NR/vtn+Vc3RV8qA1pvE+rS/dlWMf7Kj+tUZtRv7j/XXcrD0LnFV6KdkAUUUUAFFFFABQOtFA60AelDpS0g6UtZAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFcb4r/5Cv/bJf612Vcb4r/5Cv/bJf61UdwMaiiirAKKKVVZztVSSewFACUVch0fU5/8AV2UuD3K4H61eh8KanJ/rDFF/vNn+VF0Bi0V08Pg5BzPeE+yrV2HwxpUX3o3kP+01LmQHF0DrXb3ui6b9ilCWqIVQkEdQRXEDrQncD0odKWkHSlrMAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArjfFf8AyFf+2S/1rsq43xX/AMhX/tkv9aqO4FfSNGl1UuVkEaJwWIzzW7D4RsU5lmlkPpnApvg//jzn/wCuv9BW/Q27gZ8Og6TD92zRv9/5v51djhihG2KJUHooxT6KkAooooAKKKKAIbz/AI9Jv+ubfyrzoda9FvP+PSb/AK5t/KvOh1q4gelDpS0g6UtQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVxviv8A5Cv/AGyX+tdlXG+K/wDkK/8AbJf61UdwNPwf/wAec3/XX+grfrA8H/8AHnN/11/oK36T3AKKKKQBRRRQAUUUUAQ3n/HpN/1zb+VedDrXot5/x6Tf9c2/lXnQ61cQPSh0paQdKWoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK43xX/yFf+2S/wBa7KuN8V/8hX/tkv8AWqjuBp+D/wDjzm/66/0Fb9YHg/8A485v+uv9BW/Se4BRRRSAKKKKACiiigCG8/49Jv8Arm38q86HWvRbz/j0m/65t/KvOh1q4gelDpS0g6UtQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVxviv/AJCv/bJf612Vcb4r/wCQr/2yX+tVHcDT8H/8ec3/AF1/oK36wPB//HnN/wBdf6Ct+k9wCiiikAUUVDc3dtZp5lzMsY7ZPX6UATUViy+K9MQ4QSye4XH86j/4S+w/59p/yH+NOzA2Lw4tJs/882/lXnQ610GqeKBdW7W9pCyBxhmbGce1c+OtXFWA9KHSlpB0pazAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiioLy9t7GEz3MgVR09SfQUAT0Vy9z4vkJItLYAdi55/Kqh8Vaqe8Q+iVXKwOzri/FDq+qttOdqKp+vNNfxNqrqV81Fz3C81mO7yOXdizMcknvTirAdV4P/AOPOb/rr/QVv1geD/wDjzm/66/0Fb9S9wCiiikBXv7tLG1kuX/gHA9TXBXd3PeztPO5Zj09h6Cuq8WFhpqgdDIM/rXH1cUAUUUVQBQOtFS2sTT3MUKjJdwv5mgD0UdKWkpayAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuH8QX73l+67j5cR2qP5mu4rzzUI2ivZ43HIc/zqo7gV6KKKsAooooA6zweP8AQpj/ANNf6Ct+snwzbNb6WjMMGUl/w7fpWtWb3AKKKKQFXU7IX9lJbHgsMqfQ1wU8EttK0MyFWU4INej1UvtLstQXFzECezDgiqTsB5/RXUyeDoScxXjKPQrmmr4OT+K+P4J/9eq5kBzFdH4Y0h/MGo3CFQv+rB7n1rSs/DWnWrB2VpWHQv0/KtUAAYAwBUuXYBaKKKkAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/xFocl03220XMgHzr/e966CimnYDzZ0eNtsiFSOxGKbXok9laXP8Ar7dH9yOar/2FpH/PjH+tVzAcGAScAZNbOjeH57yRZrpCkAOcHgt7V1EOm2Fucw2kan6Zq1ScgEVQoCqAABgAUtFFSAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUV4145vPGfw28Sv8A8InG2oW3jeZbK0jnmythqTD5ZOTnyioJIHdR60Aey0V4pot5490ix1zSvB+pWctl4LPk3UupK8s+qXIjE07bs/uwS5A68+grpPCfxLv/ABPLr08dvFFb2Oj2WoWyEfMJJUlLqx7gGMYp2A9HorzjVPEWp678BR4qlYR3+oaBDev5JKASvGrHb3AyaxtB8X/ELw7NpFh4ludIvYNT0CW9tgmYPIlhiVgskrHDKRnLHGDSA9gorwvw/wDFbxvqGrSaLDrOj61PdaLcajbS2dlIkEU0WCYlkJ2zAhsblP4Vrat8a7qy0hPENnZxT2lj4dl1fUI8c/aMhI4Qf4cvv59ENOwHr1FeA/8AC8fFNlb38S3mn6zP/Z5u4JYbCa3jt5lkRTE+8fMCH4bg8Hiuz8P+IfHL+K7fwp4tu9Omi1zRZdRtZLOFo2tmVo1ZGyfm4k4PHTpRYD0uivJNOtPG2hfFTw/4WTxTHcaRZ+H4xLHLAS8/l/Izls/fLDOfTitz4oeLdY8OzaXZaVruk6QL1pDJcXiGeVto4SKBSGkJ7kdAKQHf0V4x4c+KXjbxtpnhvStE+wWWs6oL57u7uLd/KjjtZjESsJw25yM7SRt71l6B4v8AH2malq3h1prKTxFrPij+zop3LNbQItuZXkCHnGxGwvHJ/GnYD3uivAL3x54w+HviHxjdeKLy0vbyKPTbW0liV1t2MgbEhi5K4AOQM5IqxafG7xKTceH7aSy1fU7uezttNv1s5bW3Ek8yxESK4z8m4NwTkA9KLAe70V5R4afxxa/GWXTPF2qWl6q6BHJDJaI0UbgzSZLRkkBgQRnJyAPpTvin4+8SeHNXaw0bX9H08Q2LXSwy27Xl1dOOiiGM7lX/AG8HrSA9Vorxaw+I/wAR/GCyT+Gm0nToYfDljrjG5haUl5oi5iGCOOOvb0Naum/Fy9tlj1TxNBBBpt14ck1iFkHPmwf69M98g7gPRTTsB6pRXC6v4n8W6D8JD4q1C2tR4gisIZ5odpESzuVDJjrgFiPwrmbfxD8Y7vxPH4STVPD8ct5pX9rrdm1ci3UMqmLZn5yTIvzZHAPFKwHsFFeY/Dn4ma14vv8AQre/treFdR8PR6lOsYJxP5jo20/3fl4FcrffGDxsdO0y/mu7HQ7C5F75uqz6dLcWwlimKRxPs4iBUElmOOKdgPeKK8XvPibfaZfT+IpYbG6n/wCER0+8X7LcmSCS4muJUARgdpTcQQwGSK7fwkfiZDqYj8Xy6Ve2Fxa+cs9mhia3myP3RUklxgn5h6dOaQHY0V5D4m8S+L/DfxV1e9l1iCXQ9P8ADceoiwEJ3N+9lUDdn7xZfvemB2q5pPjD4gaXqXhe68XSaXcab4skEEcVpEySWUjxNJGCxP7wELtJ4wTTsB6lRXI+J/Fl74Z8WaNb3nlDRNThuI5JSvzx3KAMoz6Mu7j/AGTXIaV48+IXi2+0/QdFfTdNury1uNVkubmAyLFarKI4kCAjczbgxORgUgPXaK8s8JfE7xBrHiPRvDWqWtrFdedq1lqZiyVaa0dEDxk9FbcTg+uO1ZsnxM+IGq67beHPD0GmLcXusahp6zXCNshighVw5xyTyeO5wOOtFgPZaK5Pxz4i1jwnpWl6rH5EsS39vBqRKHHkvlSy+nzlfzrhrD4weJNTn1TToLS1S6m1mytNGJUkSWssxjkkbnkoEkY49BRYD2WivBrn4i+KtFspTYPDpmnyeINcgutVlspryK3MV46xq6qcoGAPzEhRjirEGv8AjrxV408MTeHNf0dLi98OXslxcxMbmzIW4hAkRAQGJ4HJGNx9KdgPcaK4zwL4n1TxX8Pm1jV44Y9Qj+2WlwYMhDLBI8TMueQCUJH1rzPRfFvibS9J0a40XTpNX1NPCmrXkELO7PNLHdIFXGfm6+hPGBRYD3+ivALz4i+Mte8F+JU0zxto095YW8E4kjs3trq3LPh45LdzvXthz15GK9p8J/2ofDOmNrV3Hc3rWsbTTRpsV2IzkDtxilYDWooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACsPxJ4TsvE11ot3eXM8TaJqCajCI8Yd1UgK2QePm7YNblFAHCa58K4tU1PVb7TPFWraNBrwX+1bWz8ry7ohAhbLoWRigCkoVJAqG/+D2nyXckuieI9U0W1u9Oi0u9tbPy9lxbxghRllLI2GYblIPJ5r0Gii4HNxeBtOh8AwfD1bu5Nlb6dFpqzEr5pjRAoY8Y3YX0xWf4l+FeheKrS1sdSu7wQ2ulXGkBY2Ub4povLYk4zux0x3rtKKAOD0b4XXGneItL8T3/AI21fUrvSraSzjWaOBI2gfb8hVI1GcqDuHzH1xTtK+D3hTTNL8R6M/2i6tPE25blJnH7uMhv3cZABVQXcjuCa7qigDgH+FDahY3Nl4h8ba5qwmtltImnMKmGMOGyAiAMx2jLMCfeuh/4RCxHiTTPE32mf7Rpeny6dHHxsZHZCWbjOf3Y6HHJreooA57UvB0V94v07xjDq13aXNhA1q8MQjMdzESTtfcpI5OcqQfeqninwE+v6/YeJtN8SX+jajYW8toJbaOGTfDIVLKVlRgOVUggZ4rrKKAPNrD4KWekWNlDo/i3WLS90y7u7iyvx5TzRpcPvlibehEiljn5wT70sHwT0u3spwPE2sNqkuqLrMeqM0Znhugu3co27SCuQVIIwSMYr0iincDzmP4L6dM2s3eteJtX1PUNb+zvLeSmNHhkh/1bxBFCoR0wBj25NWbj4Tw6rp93B4h8Waxqd7cNBJBfu0cUlo8L743iWNVRWDAEnbzjnIrvaKVwOM8OfDh9G8UzeMtU8WaprWqT2a2LPdLEiCNWLAKkaKq8seg569aj174YprHiG+16z8U6ppn9rWkdlqEFsIitxEm7ADMheM4ZhlCp5rt6KAOH8IfCrTfCNlNaRazf3hm0mDRy84jBEMKFEI2qBu2nGfauf8UfDf8AtF/Bvgmz028l07RLtbq41GRlCmBQ2+FsY3eYG2kYxgk9q9YoouBleJ/D1r4q0G78P3k0sMF2qqzxY3ABg3GQR2qtb+D7G28SxeJ0uZzcQ6YdKEZxsMZdG3dM7sxjvjk1vUUAedW/wZtNLg0ZNA8Wavpk+kWTad9oiELNcW5YttcMhAOScMoBGetGn/B86BYWdn4Y8c65pr2scsLyfuZ/PSR953pKjLuB6MAD716LRQB55ZfBDwjZae2lJLePato0OjGNnH3I5XlWXIGQ+9yfTgcVr+GPAb6DqY1jU/E+p65dxWpsrd7wRqIISQSFWNVBJ2rljknHWusooA5TXvh7p+v+JD4huNRuoxNpp0q7tFCGK5t9zsA25SwIMjcqQaz9D+FUWl6npV7qXirVtYt9B3f2VaXfleXakqUDZRAzkKxUFyxANd3RQBwHxh0C/wDF+jWXhWw0q4ma8u45GvEYKlmqn5mY5zkqWUAepq/rXw5t7y70zVNA12+0HUdKtDYRXNmsbl7chcxukisjDKqRkZBHFdhRQB55/wAKc0+0tNK/sTxJqunanpc9zc/2mhjknuJLhg07SiRWVtzDPTjtirHhj4R6P4Y1Cy1ODWNSu57K7u70NcOrGSS4jCPuwo44yMfy4ru6KLgZfifw/Z+KtAvvD2oPIlvfRGJ2jIDLzkMM9wQCPpXM6V8IfDuk614c1uC7vGm8N2ktrAjsu2YuCDJIMcuNzYIx9413VFAHBn4VfY2a48PeMNY0m6a+vr1pYhFIr/apmldGjkRkIDMdpIyB3qfwl8K9E8IajZ6rY3t3LPa2dzaMZSuJTPKkjyMABhtyDgYGCeK7WigDC8N+EbHwzoM3h+0uZ5YZp7qcvJjcDPK8jDgAYBcgewFc6vwc0NbW1tU1fVI/sel3WlRSRSiOQJPKspkDKAQ6soxjj1Brv6KAPPf+FQwX76jdeJvFWp6xe39gNNW5ljgiaGAMWG0RIqk7jnLA10Wg+F73RZ7SWfxVqmoJa2IsvJn8tY5CGBEpVFA3gDbxgYzxmugooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD//Z\"}")
----------- Killer string 2:
socket_status.send("{ \"RequestType\":\"SendImage\", \"RequestID\":\"788346414\", \"toType\":\"toUser\", \"toID\":\"1036\", \"fileType\":\"image/jpeg\", \"blob\":\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCAHgAoADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD9U6KKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAopNy/wB4fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/AHh+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv8AeH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/wB4fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/AHh+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv8AeH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/wB4fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/AHh+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv8AeH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/wB4fnRuX+8PzoAWik3L/eH50bl/vD86AFopNy/3h+dG5f7w/OgBaKTcv94fnRuX+8PzoAxst6mky3940UmRQA7J9TRk+ppKKAAFh/EaCWP8RoooAXJ9TRk+ppKKADL/AN40Zf8AvGkwKMCgA3N/eNG5v7xoI70Ad6AF3Mf4iPxpMsP4j+dHSlxmgBCWH8R/OlBb+8fzpCM96OgoAVWb+8fzoLN6n86OBTSc0AOy3940Zb+8aTg0YFACkv8A3jRlv7xpNvvSYx3oAXLn+I0Ev/eNAwOc0YBNABubHU/nSbm/vH86VaMc5oATc394/nTt3+0aTb70baAAl8/eP50uW/vGk20baADc3940B27k0babQA/Lf3jRlv7xpMAdTRgHpQA4lj/EaTLf3jTSMUvFAC5b+8aMt/eNJjHNGAelAC5b+8aMt/eNJxQSD3oAXLf3jRlv7xpNvvRjHegBSW/vGgFv7xpuB60YHrQA7Lf3jRlv7xpAQO9IMDvQA7Lf3jRlv7xpuM96XbQAuW/vGjLf3jSfLR8tAC5b+8aAW/vGkAFJgetADst/eNGW/vGk2+9IAPWgB5LH+I0mW/vGmkYpeKAFy3940Zb+8aTGOaMA9KAFy3940Zb+8aTigkHvQAuW/vGjLf3jSbfejGO9ACkt/eNALf3jTcD1owPWgB2Wz940mWzjcaP4aMDFAC5b+8aPm/vGmjA70pGaAF+b+8aPm/vGm4HrRgetADst/eNALf3jTcD1pQPegBQX/vGjLf3jSAClOO9AAS2fvGgFs/eNJgYzRgYzQApLf3jQS3940hA9aQjHegB2W/vGjLf3jTcD1owPWgB1FFFABRRRQAUUUUAGc0UnzUtACZFGRRk+lGT6UALSZx2paMigApMr1paTJ9KAA57UDPelpPmoAMCjAoyKMigA+Wj5aMn0oyfSgBcYowKKKAEwKCAO1Lz6UhJoAMg0ZApfwo/CgBMCjApAT0FOoATAowKWigBMCjgUZPpRk+lAAeDmlpDycUtACfNSgGkzntRz9KADn1oOR3oyBQSD3oAXoKRTnqaXp1ozmgAooooAKKT5qPmoAXPGaQEmlBzScCgAwfWjB9aMijIoADntQM96BnvQc9qAFpP4qXPOKOM0AITilHNHtRQAnzUoBpM57Uc/SgA59aDkd6MgUEg96AF6CkU56ml6daM5oAKKKKACik+aj5qADqOKFpQc0ZFACEZpQMUgPrS0AIc9qBnvQc9qBnvQAtJto3UtACE54FKOKAc0ZxQAUUm72o3e1AAc9qTBFKCaDntQAtFITigHNACg5oJxQBigjNAATQPeikyKAFoozijOaACg57UUUAJgUv4Ugz3paAEwKMCl59KOfSgAooooATgUGl59KKADgUmRS0mT6UALRwKKTJ9KADg0vAoOR2pMn0oACfalyKTJ9KUnFABRjFFGRQAcdBzQeODwPSuN+IXxa8EfDLTW1DxZq8NrGASFJyx/Ac189aj/AMFFvhRaXTQ2K/aVU4D/ADCgD635AyRj3oPTI596+evh3+218HvHl8mlx6wlrdyHARwcZ+pr362vrW8tFvbW4SSJ13KynIIoAm45/hPal6ngY968W179q34XeHfGjeBtQ1YDUldY/L2Hqa9lgnS6hSdPuSorg+xGRQA89KWkPSloAQDFLRRQAc+lI1LnFITxQANSjgUUUAIc8sB16ilHp+leZ/Fz4++Bfgy1jH4xvxbPqG7yvlJzt+ldN8PfH+hfErw1beKfDtx59ldZ8t8Yzg4oA6XHt06UuOpHJpMkHPf0rB8ceM9H8A+HLnxNrkvlWVqAZG69aAN4nC5xk96Mc4HXtXlvwq/aJ8AfF2+ubHwjqQmltSBINpHX616iB6jPvQA7A60UUhGaAA57UtIc9qWgABzSYpQMUZoAQ9KWkPSloAQDFLRRQAc+lI1LnFITxQANSjiiigAooooAMYoPNIc9qWgAHSiijPOKACiiigBPmo+ag57Upz2oATrQDmjpQBigBaKKKAEBz2paQYzSk4oAQnFKeKQjNKeaAEY0ueM0YzQelAAcnvR+NJxnOaOtAC4zRSEZpaACiiigAooooAKKBzRQAUUUUAH40UHHeigAopMCjAoAX8aPxpDjvScetADufWjn1pDjvQMdqADB9aMH1oyKMigA59aXn1pDjvSkjvQAgyc4rmfiR40sfh94O1LxPqMgWO0iJBJ744/Wum6jBr5Z/wCChOuXen/Bi6s7aRkW5xvIPo1AHxBdah8R/wBrj4vnTo7u4aGe4ZQoY7Y0B/LpX2j4Q/4J4/CnTdESDX4ft96yAyS5K849K8x/4Ji+FbSX/hIvEVzErzr5XkuRyvXNfoGAoHAz70Afl5+1T+x3cfBy2HjPwJPM2nodzxxAgxc8c9TXuP7Anxn1/wAV+Hb3wT4j8+STTY8RSSggnIPrX2B4h8N6N4p02TStbtEubab78bDg1z/hv4V+BfBBnu/DWix2csincy9+KAPzF+M8m39rm4ZjhRdxdT06V+j+rfHf4X+BrCytPEfii3t5fs8YKr838I9K/MH9qI6j/wANIawulyFblp40jYDueK+h/CP7Auv+OvDCeJfG3jO4i1O6hDrEyltvHHf0oA+1vBnxW8B+P4xJ4X1+C6B7AgH8utdaCBkk5B6e1fjday+NP2avjpHoEGq3BksrqOMqWO11cgdM46Gv0i/aI+Nn/Cqvg5/wk0L41G6tIzAueSWUZP60Ad74w+MHw+8CKW8SeIbe229VDBiPwFZ3hX4+/CzxpdCz8PeKLeeU8YY7f51+dfwU+A/j39qnVr3xb4p8Tz2tk8m4SsxIcEnjGe1bHx6/ZA8SfAnSR448G+LJruCyIeXYSu054780Afp7G6yrvQh1PRgeKz9a8RaN4dtGvdX1CG0iQZLO4Gfzr5i/Yq/aIu/iN4GvNK8S3e/UdDiy0jHlxg/0FfLPx9+LPjv9oP4vt4A8MXs6WInMMMUTEdPvHIoA+/JP2pfgjHfGwk8YwifO3btOM/WvQ/D/AIr0DxVbC70LU4LqMjIKOCfyr4bsv+CbUkvhcXdx41mGpvF5m3YchsZxnNeS/BzWfiz+zv8AGhPC98uoXGmpOIrhSrMrg8Kc84oA9N/4KiFheeEDuwczYHr0r6E/YZOf2fNAYntJwe3zV86f8FN5zcDwRc7cNMkjn2yAa2vhh+0h4f8Agn+y3owa5SbWJ45RBbhuc7utAH2jrXxE8HeHbr7Hq+u20Ew6ozjNeTftU69pev8A7P2v32lXAmhZF+ZenWvjP4OfCf4lftUeO38ceKdQu4tIabe77yFK54AANfX/AO0f4N0rwN+zTq+g6OhjhtokVcsWJOeetAHy/wD8E6NWsNI8R+Ib7VLyO2hjZSWdgB39a+1Zf2nfgvFqR0mTxhCtyG2425GfrX5UfAnwN45+JnimTwV4TvJrW3vpMXUkf8IBr6s8Xf8ABOO7sfC8moaL4xmn1OCIyFdpBY4yec0Afd+i6/pHiG0F9o2oRXMDAEFGBq+8ixxtJKQEHcnFflj+yj8a/GPwg+K8fw78V3s7afLMYZklYnZjgdfU19O/tw/tEX/w08I2eieGLny9Q1qM7ZF6oMD+hoA9y8VfHn4XeDJWt9f8UW8MgOMKd38qd4T+Ovwy8bTeToHii3mc8BWIX+dfnz8AP2Q/F3x7sH8b+L/EN1Z2t0S0buWbfzzxnioPj/8AsneMv2ereLxr4T8Q3V5ZW7b5HQlAnPHGeaAP1Gmu7eC3N08gMaruLA8Yrmbb4p+BLu8TT4fEVs1ySVEe8A5/Ovn/APYt+Ok/xm8DXXhjxROZb+yjEUjE8sDx/KvmD9rj4NeKfgj8QI/G/hm5vP7Knm85cSMRHg5PfuaAP1Ha4hihNzJIqx7dxYngCuWh+KngWe7+wQ+ILd7gttCKwJJ/CvhLxj+3edT+BltoemztH4luofJlYdUxx+oq5+wr+z7rXiPUh8VfGU939n3mSzSSRsSEk7uM0AfoYjh1DL0IBFOPtTY0WNAi9FAApSaAFOT3opOBRkGgBaKKKAAmijIoJxQAmD60YPrRkUZFAC/jQOKQAUvAoATPGaM8ZoPSg9KAFooooAAc0E4oooAKKM5ooATgUvBpODS8CgAopMijIoAWiiigBM84paT+KjHOaAAUZ5xS0e9ABRRRQAUUUUAJ81HzUmB60YHrQAvzUfNSYHrRgetAClvSgZ70tJkUALRSA5oJxQAtI1GRQT2oAGoPShqD0oAAcmj+KgHn60fxUAB7YoPSgnmgnigAB4znrXzR+3v4TvPEHwS1C7so2lktApCqOeWr6XyNpBXkdKy/Evh+x8T6Jd6HqcKvFdxlCp5HSgD8+f8Agml4/sNH1vXPB+ozrHPfmNbdWOOVzmv0ZJ6gYr8mPjT8BviX+zf8Qz4u8JJcPYrMZra5hUngnJGB+VereEv+Ckuv6XoyWPiXwXNdXUKBfOZypcgemKAPsT9oX4sR/B/4dX3ilNjXUQHkxFgCxzivF/2Vf2pfFvx31nVLLUtNMNvZr97dkcg18ffFn41fFX9qjxFbaLY6Rdw2Uj7YrRVJVQfVsV94/slfs/j4LeAWfU4h/a9/FuuOORxxQB8MfGlFk/a4uFcAj7ZD1/Cv1f0dVTRbJEGB9mj/APQBX5UfGWyvH/a2uHWznK/a4fmEZI7d6/VjSfl0ezXv9mj/APQRQB+U/wC2QAP2oJ8Lg/bLbt7ivb/+Ch73q/DXwWsDN5JtB5uOn3VxXi/7Y1jezftNzSRWk7j7ZbfMsZI6r3r7e/aK+DUvxf8AgfDpmnW+7Uraxia3Hf7oJ/lQB8Ufs/eG/wBrDUvBFvP8JrnGjHOxVKZHPfPNdx4q+FX7c3ifRbjR/EtwJrCcASI7RqPzrhPgN+0J45/Zf1a78I+JvDt1PYb9vlEECPB7HHeu5+Nf7dPib4laGfCXgHw7dWcl5hWmj3E5zx2oA2P2afgB8T/hND4l1XxBbRwQ3FuxPlzq+fkPYGvKP2NTbP8AtMRPqpUuLqfbv79fWvrb9jb4Z+OrHwXfav8AEW5nkl1iLEcMpJ2jBH9a+Tfjp8LfHv7OvxjPjnw7p85sTOZraWJSQRnLdKAP1aBVUBBBGPyFeb618Qvg1ZeJX0jVbywGrKyqytEpbcenNfJw/wCCj8p8IG1bwfMdX8nys7jndjGeleU/s7/Df4h/H74zr478QxXkVgJ/NuJX3KMdVGKAPRf+CoLo7eDnh6MsxXHcYGK+VdI+C/xL8TfD6bx3a2E8mjaeAyHdkcnnAr6u/wCCmunXIbwXb20EswgWRPkQnoAO1e7/ALHPh201j9mrS9I1awGy5ilR1dMHknrQB5F/wT7+PdlNp/8Awq7W3igurc7bbIAL85NfQX7YRB+A2vt1BROfxr8+/jp8MvFP7N/xmXXvDkVwtq9x5ttJEhIIzlhxX1r45+KsPxf/AGTdR1eCGUXawRrPEUO4EHFAHj//AATVOlDxTr3nbBdl18vPXvnFfovPsMTBz8m07s9xivxa+Cni/wCIPwo8Uv448PaReTW9lJm4QIwyCcelfVnij/go7cXPhOW10nwjPDqs8Xl5ycqcYJxigDwj9oQRp+1tfroG3aLy32lOnUZrr/29/tv/AAmvhtrzd5XlQ7M9Purmsz9lb4Q+MfjV8YY/Hviayn/s+GYzXM0qkbs8r1r6l/bh/Z51H4leDrPXPDFoZNQ0OM7Y1HMg4/kBQB67+zCbD/hTegiwKeWYjkrjk1T/AGsf7OPwR13+0dhTyxgN65r4c/Z7/a68W/AnTj4H8WeHrq7sbViibwy+Wc89uai/aF/ay8X/AB/ij8FeEvD11a2M7BHCBm83J47cUAdD/wAE2zc/8J9riwqRbiQbj2xzivq79sXxF4E0j4TalB4yjhla4TFvG33ic9u9cN+x78G5PgT8Or3xj4xgMF5eRCaRCMlcAkfnXyf8cPHnjX9qD4vw+G9Ftbs6atx5dvFsYKFzhjQB876W9tZ67bajfWbyaeLoOFIIBTf6/Sv2n+Avinwh4q+G+kah4N8lbIQhfKTqpAAOfxr5/wDHv7E3h6b4EW+haPZr/bunW5lS4A+ZifmIr54/ZA+NXiT4JfECTwL4qhu00u4n8ly0bERkHA7dzQB+poPpSDOeaitLqG+tormE/LKgcfQjNS8YwaADvR3o70d6AAnFHQUEZpaAAc80HnikJxS4xQAU3JpQc0EZoAU8UgNKeaQCgA3UZFHy0fLQAZFGRR8tHy0AGQaMgUny0fLQAuRRn3pDtxSjaaADvR3o70d6ADn1oOR3oIHrRgHpQAEkDrQARzRjPNAAoACcUtITiloAQDvS0UUAJ3o70d6O9AC0UUGgBAeaCOc0mBig4xQDVg5zkUvPrQMdKBjPFJK/UNWHA4ozQwxQRTt5hdCE0ueKRqXHFHzC6BaAecYoFIetO6DQUkjvQcgdaQ896OtK6DQXIHSjIoA4pB1o1DUM0ZoHWlajUNRaMUh6YoWlqBS1PRdM1u2az1OyhuYmGCsiA/zrzLUv2V/gjrF013f+DIWlY5JDkD9K9ZODgmg4DYLZFPUZyHhL4SfD7wOAPDvh62tj2YoGI/E115A/1eARjFKeRjHSj5cZzk+lOzE7HJXnwr8DX+sHXbzQoJL1iG84gZyK6tFVVWNQAEGAPQU8YIwBSdyCdxPalqLc5TWvhd4I17UjrOraFBPeFg3mMBnI6V1EUccUaQxgeWi7QuO1PHXnj0o3EfL2FFhnGeLPg98PPG0vmeI/DltPJ/eChT+grO8O/AL4U+FrsX2jeFreKVTkFhu/nXooOBn0oJzyOlFgGRxRQxpHDGqIvACjAFZ+u+GtE8SWr2OuWEN1E4xh0BxWkOufXtRz0zg0rtB7zPK3/Zf+Cz3f2tvBkPmk7t27jP0r0LQvDei+GrNNP0TT4bWKMYARAP1rSIz06+tKD2LZI600nILM57xR4C8LeMjC3iPS4rtrfOwuM4zWlouh6X4c0+PStHtVt7SP7iKOBV7nlg2c0vG0ZGCKAOf8TeBPCvjARf8ACQaPDeeVnbvUZGarWXw08GaZpU+i2mjRR2VxjzIh0P4V1HGN4GCetB9hgetAHJad8KvAWmW1xaWfh21SO5H71SgOf0rmpv2Zvgxc3IvJfCEAk3bg2e/0r1IEgbs5IoXptxwaAMzQfDWheGLJdP0OwhtYIhgBEAzWk6LIrCVAynjBGQaXg8E5pM8H+8egoA4PxN8C/hj4vuTda74Xt5ZW5LKoX+VL4X+B/wAMfB9yLnQfC9vFKvIZlDY/MV3mcDBPTtS8AdeT2p2sw0XQpalpOn6rZNpl9CJLZxhkAwMVzuifCjwJ4d1AappPh+3guUJKOFBPPWuu7YB4WjqcjqaSTb0YIR1V1MTKCMYxXGXvwe+Heo6k2qXXh23a5dg28KAcjvXaZ6ZOcUDjhjgetF0O5Fb20NpAlvAMKgCgegFSnOQc0ZA+ajjv3ouguLRSAYpScUm3HYQhOKWjIFFHqAhGaWk3Uuad0AUUgOaDzTugFHFNPWnZpo60XQCnpQOlLRRdAIM96Dk0ZFGRS0QW1uGD60YPrRkUZFK6YO973A9KAMUtJupgAOaCcUZA4oPSgBc0mD60ZAoyKADJ9KMn0oyKMigANABpcZpuB60ALg+tGD60ZFGRQAvPrRRkUUAJg+tGD60YFGBQAHPagjNGBQCACc9KG1b3gbSXvBjvnHvRjuR+NeX/ABB/aE8BfD+8bTr7UUe7X70YycfiK4cftneAedqcDqea46mMoU92cVTMMNSdnI+iM/7X6UZ/2v0r54/4bO8Bdk/nR/w2d4B/ufzqf7QofzELNMK/tH0OcHvQMDvXzv8A8NneAv7n86P+GzvAX9z+dH9oYf8AmD+1ML/MfRHPrRz6187/APDZ3gP+5/Oj/hs7wH/c/nR/aFD+YP7Uwv8AMfRB3HvSYP8AEa+eD+2d4C7J/Omy/tpeAIo8uvT61rTxtGbsmH9qYX+Y+icn1oHHevmj/huj4c/3P50f8Nz/AA6/ufzr2FleKkrqI/7Uw38x9L4PpQQT2r5nP7c/w4/55/zo/wCG6Phx3jP60/7KxX8of2phv5j6ZNA4/wD118zf8N0fDf8A55n9aP8Ahuj4b/8APM/rR/ZWK/lF/aWH/mPpgcUdTXzP/wAN0fDf/nmf1o/4bn+HHaM/rT/srFfyh/aeG/mPpnnOB0pAMn6V8zn9uf4c8/uzx9asWf7bnw6vrlLaKP5n+tH9lYr+Ul5thIK8pH0jn/apPoa8NH7WHgkgEJ/Oj/hq/wAFdk/nT/snFP7Bxf6yZbF2lUPc8H1owfWvDP8Ahq/wX/d/nR/w1f4L/u/zo/sjF/yh/rLlv/Pw9zz/ALX6UZ9/0rwz/hrDwV/c/nR/w1f4K/ufzpf2Ti/5Q/1ly3/n4e5k+9A69a8M/wCGsPBX9z+dH/DV/gofwfzprKMU/sjXEuXJfGe5kkKTnk0AgMQK8MP7V/gv72z+dXNL/ae8Iatex2FvH+8f61FTLMVBX5S6Gf4CtKymez5HrRn/AGv0rhB8VtNIBFu2D9aX/haunf8APu361wNW0Z7SaaujusH1owfWuE/4Wrp3/Pu360f8LV07/n3b9aBnd5/2v0oz/tfpXCf8LV07/n3b9aP+Fq6d/wA+7frQB3fXvSH1zXC/8LV07/n3b9aP+Fq6d/z7t+tAHdHjk85pQDjrxXC/8LV0wHi3bH41R1T41aJpVv8AabqEhR9axxFanhoe2qOyNKVGVeXLBHpBDY4NAz3NePf8NI+Ev7v86P8AhpHwlj7v868KXFWWN6VLHpvIsdNXjA9hyfSjn0rx7/hpHwn/AHf50n/DSXhP+7/Oj/WrK/8An4P+wcd/IexfhRz6V49/w0n4U/u/zo/4aT8J+n86P9asr/5+B/YWO/kPYefSgZHavHv+GkvCfp/Oj/hpLwp6fzprivLLXdQP7Ax/8h7CB37mgnHJPSvHz+0l4U/u9PrWN4j/AGtvBHhu3FxdD5W+taUeJMtxMuWNQqPD+Pe0D3nPtRn2/WvmA/t2/Dj+7/Ok/wCG7fhx/cP616Sx9D+Y2/1WzP8A59M+oM+360Z9v1r5f/4bu+HH9w/rR/w3d8OP7h/Wj6/Q/mD/AFWzT/n0z6gz7frRn2/Wvl//AIbu+HB/gP60f8N3fDj+4f1p/XqH8wf6rZp/z6Z9QZpMj0r5g/4bt+HH9w/rSj9u34b/ANw/rS+v0P5h/wCq2af8+mfTxzwc8GlIGeelfMcP7dfw2aRUYbVY8nnivZfhx8X/AAZ8TrP7V4Z1RLgoBuToR+dXTxdGo7JnLisix+DhzVoWR3GBRnFIDmlxmult9DyNluIRmgDFAIoJxQAfNR81Hy0fLQApz2oGe9J8tHy0AAOe1BOO1G6jdQAZ5xR/FS55xSfxUALRz6UnzUc+tAC0UUUAFFFFACcHLetY3i6+k07w7f3kJ+ZIWI/I1skk4OOBXPeP9p8I6mB3gb+RqKvwMyry9x+h+ZHiPUr7Xtcu9Sv5WkkkmcfMc9CapfZMZIA5q3sBvLknn9/Jj/vo1YWMAZ71+e1qj52flderLnkjM+yn+7R9kI6CtTyh6UeUPSsXOSdkY+0ktDK+yn+7S/ZT6VqeUPSjyhjOKfPJD9pK9rmX9k9qPsp9K1PKHpR5Q9KFO60H7WWzMo2xGeKo6pBttJDjtXRGNcEkVl62irYyYHQV6GVyviYJ9wjVu7HExQbh071L9m5HFS2Y+Q8d6thAa/pXDQjKlGy6IcqjUtWUPsgoFoK0PKo8ut/ZQWpPtJt6Gf8AZBR9kHpWh5dHl0eyXQftGt2Z5tKPsnPStDy6PLpuiJVuhnG268Ve0CDbq0P1pzR8ZqxoqY1WH60vZR7GGJqP2TPSoLYsi8dqn+xHd0q1Yx5iX6VdEIHOKtU0fn9WvJS3Mj7H7Uv2L2rW8kelHkj0o9mk0kiPrGm5k/YvQUfYj6VriIelHlewodKKshfWJRd4syPsXtR9i9q1vJHpR5I9KPZK6G8RJK9zHNlz0rc8CwbPFFnx/FUTQgdK0vByAeKLPA/jrz8ypWw0rHs5BXlLHU031Pqe00xXhjIQcqO3tVj+x1I/1Y/KtbS4Fa2iyP4F/lWkLZcdK/LJ/Ez+jKXwL0OY/sdP+eQ/Kj+x0/55D8q6j7MPQUfZl9BUlnL/ANjp/wA8h+VH9jp/zyH5V1H2ZfQUfZh6CgDl/wCx0/55D8qP7HT/AJ5D8q6j7MPQUfZh6CgDljpA/wCeYx9K4H4tWSw+HnOwD8PevZZLdQM4ry741RKvhtyPUfzr5/idtZbUPWyTXGwR4BHaZHAqUWRx0rQtIlKjirawJ6V/M9SvKMnqftlOlHkWhifYj6UGxz2rb8lPSgQIe1R9Zk1uV7GOljE+xcfdo+w/7Nbfkp6UeSnpU/Wp7Jh7GLaujE+xe1H2I+lbnkL6UeQvpVfWJ8urGqMXJJGEbJvSvNfjNBs0mH8a9kkhXB4ryj42pjSocep/nXs5DXlLGQVzowlH98rnhywbuop32fBq1EgqXy81+tqbaPs44eLitCh9nGelH2cDtWh5Q9KPL9qfPbqX9VjfYz/s6+lH2celaHlD+7R5Y9KTqX0QfVoJNWM/7MB2pfs6+lX/ACx/dpPLX0oc7ISw0exnmADOT16V7T+yX4n1TQ/i3pem21w/kXjMJV3ccDjivJWjHOF6da9F/ZsG340+Hzjje+fyrrwVR+1jqeDxHhYf2fUuuh+qcZLRo3qoNPOccVFBnyY/9wfyqT5q+6Xwn8vTilKXqLQBimg4pQc1RAfxUNRjnNH8VABkUuRRz6Uc+lACZFGRRk+lHPpQAHpQOlLRQAUUUUAFFFFABRRRQAh6Vzvj/wD5FLUv+uDfyNdEOc1zvxA/5FLUv+uDfyNZVvgZlX/hy9D8yf8Al5uP+u8n/oRqyOlVx/x9XH/XeT/0I1aAr86qfGz8nr/HL1A9RluvQ0AHgbcepz1o7EYz6V6P8Lvgn4i+JQlns42htYlyZcZyadKlOtLlp7ioYedafLT1Z5zgnIxmun+H/wAP9b+IGsw6VpVs5RmHmNjhRW7pfwR8YXvjM+E202RXjkAd8cBfX8q+0/AHw/8ACvwd8K+dKYklSPdNO3UnFelg8ulXqc1TRI9fL8oliJ81TRLc+Zvi5+zTN4H8PRa3pdyJljXNxkYwa8AKkEgr0OGr3r9oH493XjO7m0DRJCumxkqWH8deDHnG7qefrXPmHsvaWo7HLmaoKrbDidjWRrufsMn0rXI4NZGu/wDHjJ9KrKrPFwXmcCepylkBtH1NXF6VTs/uD8auL0r+n8M7YeK8kKau7sXoQSeOwpOnbHtS4J4ByB0969Z+Fn7PHiz4oabcarZwNDbwrujcj79KtXpYWHNVZpRoVMTK1M8mABbIruvhP8J/EHxR12HTdMt38ncPOl28KK2vB37PvjXxH43bwpNp0sIgkAmkI4Uf/qr738G+DPBfwI8Fb5DDB5EW6aZgMsa8TM86jh48lDWTPZy7KJV3zV9Io+Pfj/8Asv3Pwy0iLxDpFwJrJFHng8bD/XmvnVjghvzFe9ftG/tD6h8TtSl0XSnMejwsVAB4krwckkliceld+VLEOhzYndnDmUqCq8uH2Qh6VNo//ITi+tQnpU+jf8hOH616LR5GJ/gs9Z08fukq+DVHT/8AUp9KvDvVxR+d1viF6YK9uvtR1BP3g3X2o5+WvRPBnwW8ReMdGn1eCMxxqu6IY+/61jiMRTw0eao7GuDwNbHVHToRuzzvAHAGAK7T4cfDXVfH2pi3ghZLYEeZLjgCr3gr4P8AiDxJ4jOk3No8Mdu+JnI7V9X6bp/hn4VeGOTFBHAmST1Jrw81zqNL93h9ZM+u4f4XliZe3xq5acT5l+LnwYn8CRrqFk4ltSBu7bT/AFrycAZGOlel/F74sX3j3UmtYnKWEbEKg715qMngngdK9PK/b/V08R8R4GffVPrklgvgQ1hxitHweP8Aip7PP9+s9q0fB/Piiz/36WaL/ZpJmnDzisfTXW59jaT/AMe0X+4v8q1FrL0n/j2i/wBxf5VqDpX5RP4mf0lS+Begc8DGcdqD97nr29qGIXndhQCSfSvCfH37W3gHwJ42tPB9xcpIXcpPID/qj2qSz3bv0+prxn9oL9ozw58F9Ckle4S41KUfuIA3JP8ASs745ftQ+Efh14OXUtNv4ry+vYibaNG7kf8A16/O1U+IX7SHxCABnu7i8l99qLn8hxQB+hP7MX7TVl8arB7G/iaDU7cncuCQwJ45+lfQGSTnHTrXjf7O37PehfBfw6kcSh9WuEU3E2OSa9lOeMNn8KAGS/dryz43f8i5J+H869Tl+7Xlnxt/5FyT6j+dfP8AE7Tyyoevkf8AvsDxW05UfSrvQVStBhV+lXB0r+X63xP1P3Ck7U0GB/hS8HqKFUllK9+MV3WmfCTxBqXh+TWkjYEDdHFjlh3roweW4jMZNYdXsYYnG0MIlKu7XOF2jcAOQf1rufh18M7/AMY3PmyQtHZp1cjrVj4d/C3VPE2pbr+FobWFvm3DrX0BqGpeH/h1oG3EcSxJgKOpNfa8OcJqSeLzBWprv1Pl864hcGsNgdZv8D5x+I3w9n8GXow++3f7prjM/wAIHPb2rqPHnji/8Y6m9zKSLdSfLT0Fct83ds4r5LOvqzxklgvgPosqWIeGj9Z+LqNkHymvJfjeP+JVDj3/AJ1605yDXk3xw50mH6n+dbcO6Y6ET2cIl7eJ4zF0qeoYugqev2B7H29NqMUg4+8TgUnQYIx71JDDLcTLDCpZ2IUL6k17Rbfss+ObnwL/AMJgsDl9m8W+3kitaeHlVWhw47M8Nl0l9Zmo32PFApY/J1PQV7v+z9+zVrHxRn/tPVIXt9LQH5mX73HH61e/Z5/Zm1jx7rSal4jspLfTLaTL7xjfg/8A1q+zvHPjnwT8B/BXlRiCHyItsUS4yxxXr4HLo8nta2x8FxPxhUVRYDKveqy6rofn18dPgxqHwi8QiwkkElpOT5MnHzAdeO1eY46DORXc/Fn4qa38U/EUmrapKRAHPkx/3BXC8keory8T7P2j9nsfdZOsX9Sh9dd6ltRH6V6H+zf/AMlq0D/ff+VeeP0r0P8AZv8A+S16B/vP/KrwS/fROTiPXL6nofqfbHMMf+4KkJxUdt/qI/8AdH8qlr71fCfyrW+J+oUgHNLRVGYhGe9KRmiigApPmpaT5qAFPtSDPelPtSDPegBaMiijAoATrR0paKACiiigAooooAMVzvxAA/4RLUuf+WDfyNdDj3rnviAMeEtSP/TBv5Gsq2kGY19KTPzJH/H1cf8AXeT/ANCNWR1FV1/4+rn/AK7yf+hGrA7V+d1fjZ+UV9arFP6Gvoz9mr462/hMxeEteVFs3O2ObAG3/GvnPHG3OPUUqM8Tb0Yqy8hga0wteeHmpwNcJip4OoqkT9RZ9S8M2FhJ4ldrdUZN5m4yRj1r4z+Pvx+v/Gd5LoegzNHpsRKsVb79eb3XxX8Y3vhmPwrPqsn2KIYxnkj61xuSSWLEnv716eLzV4iHJT0PYx+ePE0/Z0lbuKxLEsxJJ9aKQUA5rxNz53fcQ96ydd/48ZPpWse9ZOu/8eMn0r0Mq/3uHqVHc5Sz+4KuLVSy+4PqatrX9PYbWhG/ZClJ82gobDAgYAORX1v+yl+0ZZ6EsPgfxN5cUBIWGUgDFfJBA4+bOf0p0c8tvIkkLFGQ5BBwc1jj8DTzCl7OR04HFTwVX2iP1y1bW/CfhnTJvFVw9rFGyeY0wAy3HFfAn7Rf7ReqfEnVZdH0e4eLSIWKqqnHmV57rfxi8ca94atvCmoavLJZW4KhemR9a4jcxADHNeNluQrCy9rX1fQ9bMM6liV7Ojog5PJPWgZ70c0gz3r6bW10fP2V7MG61Po//ITh+tQN1qfR/wDkJw/WkzDE/wAFnrdgcxJ9BV/PSqFgMRJ9BV/0qon53W+MVSVYSDqpyPevpH4BfGWyihh8K63sjI+WKQ8Bq+bQOPX29KfBPJbyLNBKUKHII4INcOY4Cnj6bpz3PTyXN6mT4lVobdT9BNa1vw74Z06bW53hhTbuLgD5vSvkL4s/FvUfHWoyW9vM0dhGxEag9a5zXPiJ4m1/TLfS9Qv3eCEFQOma5gf7TV5eVZEsHP2lfWXQ+gz/AIulmUVRw/ux6hnNFFFfSHw71dxDWh4OP/FT2n+/Wea0PB3/ACM9p/v15uaf7tI93h7/AJGEPU+x9J/49ov9xf5VqLWXpP8Ax7Rf7i/yrUWvyifxM/pKl8C9Bs8azRPC4yJFKt9CK/PX9sD9ljV9H1G5+IPhKOW4tpG8yaMEkx/jX6Gjhsk9aq6lplpq1nLY6hAs8MylWVhkEVJZ+MngnwV43+KviK28MWSXVy6vsO8kiMd+tfp9+z1+zz4d+Dfh6FRbRy6pMgM9wV5zXVeBfgt4E+Ht/d6j4d0aK3mum3O/U59s13g65z06UAAAPNAwe1LSAY70ANk/1Zryv42f8i3J+H869Uk/1Zryv42f8i3J+H86+f4n/wCRZUPXyP8A32B4ta/dH0q6eoqla/dH0q6eor+Xq/xM/caX8NDomMUqP3Qgivo74RfFCx1i0j0PVCkdyg2qT0YV83H72c/Sp7O9ubC4jubSUxOhySK9zh7PquR1lUirxe6PJznKaebUeSWjWzPsbxJ4k0Pwfpcl6/lJxkBcDJr5b8deOdQ8X6k8s07CIE+XGOgFUfEPi7WfEpRdSu2kSMYVegFYoO3kHr39K9niXi6ebr2ND3YHnZJw5HLf3tb3ph0FA5FIOuKOp4r4Y+pW42Toa8m+N/8AyCYfqa9ak6V5L8cP+QVF9TXucPf79A6MJ/GR41D92phyKhh+7Uw4r9etdWZ9vT1SZa02/m028hv7fG+FwwyM5wa/Q39mv9obQfH+jw+GtcaGHUIkCbWxh/6V+dJycnOCa0dB1/VPDmpRappFy8E8JyCprvwWM+pvyPm+JuHKfENDlbtNbM/Ur4n/ABP8H/CDw3Les0ETlSYokABY/hX5vfFf4seIfijr8+p6lcP5Ic+VDu4UVn+OviX4q+IN0s/iHUZJ/KUCNScDp6VynGPQ1vjsyeI92HwnmcK8H08kXtsR71TuLRRRXk+h92rX0GSdK9D/AGbv+S16B/vP/KvPJfu16H+zd/yWrQP99/5V2YP+NE8LiPTL6iXY/VG34gj/ANwfyqSo4P8AUx/7g/lUlfexfuo/lOprNt92FAGKCcUVRmJjnNB6UtFACHpS0h6UtABRRRQAUUUUAFFFFABRRRQAUUUUAGMsR61zvxAwPCGp5/54N/I10OTuA9Ky/E2ntquh3lggy0sLAfXBqKq5oMyrq9Nn5eJzdXBzx50n/oRqyvI54Bq/4q8N6l4U8RXuk6payRPHMxUlTggknrWasqbfvZPpX53WpyjN6H5Xiaco1XoS0UzzV9aTzU9RWXJNdDD2c+iJKTB9ab5qetIZU7mhwl0QOnN7okoqMyoO4pfNX1/Wjkl2Dkl2HHpWVrv/AB5S/StEypjiszXHU2UmPSvRyqLWLhddSowl2OUs87B9aueuTVKzYBOfWrXmJzzX9M4acPZRbfRBKEnokSYPrRg+tM80daXzV9a256b1TJ5Ki0aHUUzzEHejzEPWhVV3B02ug/B9aMH1pvmrSeatNVIN2uHsprWw5j2qxo3Opw896qmRc1Y0ZgdVi+tJzj3McTCXsXoeuaef3S89hV8YwOazrB1ES5PYVd81OOaqM49z88rUanN8LJKKZ5y+opfMT+8Kp1ILVsyWHqb8rHUUzzk9R+dHmp6inzR6MPYztqmPopnnJ6j86POT1H50c8e4OjU/lY5ulaPg8Z8T2f8Av1lGZDxkVp+D2U+J7PB/jrzcznF4eVme5w/TmsfBtM+yNJ/49ov9xf5VqD2NYulX1oLeMGZPuD+IelaI1Cz7zx/99CvyqfxM/o+l8CLLUo+tVf7Qs/8AnvH/AN9Cg6hZ9p4/++hUllnB9aMH1qr/AGhZ/wDPeP8A76o/tCz/AOe8f/fVAFoYoOKrf2hZf890/wC+hR/aFl/z3T/voUATy/dxmvK/jdn/AIRyTnuP516W+oWewkTx/wDfQrzD403EEvh6QJIrHjgHPevA4mTeW1Ej1skaWNg2eN2h4H0q4elUbWRNo57Va82PH3hX8w1qNTmfus/b6dSHs1qSUH61H5sfrR5sfZh+dZOjWbvys09pB63Hk89aUnio/MT+8KPNTGMj86FRqr7LEqkU7XHnpQtM8yP1A/GgSx+o/On7Gp/KxKpC+4rk7SDXk3xw/wCQVF9T/OvV3ljKk5FeTfG8g6XDj1P869zh6lNY6LaZ0YWpD2y1PHIcham5x1qCJuKl3DGK/XeSUldI+2hXpctnJfePopu8elIXHalyN6NF+2pS2kvvHUtM3il3ijkklZIft6ezkvvHUU3eKTeKfI3okL29GWikr+oPggdh6V6J+zfx8a9Ay3LO/wCHFedFwScjPpXt/wCyR4B1rX/ihY67HZyCz09iZJGXA5HFdmChL2sdD5zijF06WXT95bH6S25P2eL/AHR/KpabEu2NU/ugCnGvu0vdR/LtWXNNpAelIBigHNLTICiiigAooooAKRaWigAopcH0NGD6GgBo6UtAVsdDS4PoaAEowKXB9DRg+hoASilwfQ0bT6GgBoPFID+tP2N/dP5Umw56Gl7z0YWvuch4u+Ffgfxud3iLRI7lu7A7T+YrlG/Zb+C+Qf8AhFj/AN/mr1raR2NG3sQR+FYywtOW6OWeEoTeqPJf+GW/gvjJ8Lf+R2o/4Zb+C2M/8It/5HavWtr47/lRtOO/5VH1Sl/KT9Rw/Y8lH7LnwWH/ADK//kZqP+GW/gt/0K//AJGavW9rf3T+VG1v7p/Kj6pS/lD6jh/5TyQ/sufBU/8AMrn/AL/NR/wy78F/+hW/8jtXrWxvQ/lRhvf8qHhKP8ofUcP/ACnkv/DLnwXzz4WP/f8Aakk/ZX+Ckow3hXI/67NXrZR+pB/KlKsRwD+VXDD0ou6QfUcP/KeM/wDDI3wJ/wChR/8AI7Uf8Mi/Ar/oUv8AyO1eyhWx0P5UYbP3TXorG10rKTH9Rw/8p4z/AMMi/Av/AKFL/wAjtS/8Mi/Ar/oUv/I7V7LsPoaNh9DT+u4j+Zh9Rw/8p41/wyL8Cv8AoUv/ACO1H/DI3wK/6FL/AMjtXsuG9D+VGG9DR9exH8zD6jh/5Txr/hkb4Ff9Cj/5Hag/si/Ar/oUf/I7V7LsPoaNh9DR9dxH8zD6jh/5Txo/sjfAwAf8UlwP+m7VLbfsn/A+1mE9t4Sww/6btXsAQg5IODQVPoQPpR9er/zMmWX0JKzieZ/8M6fCYdPD2P8Atq1H/DO3wn/6F8/9/Wr0zBHY0YJ7H8qP7Qr/AMzOZ5LgnvBHmZ/Z1+FA/wCZfP8A39agfs6/Cg/8y+f+/rV6YVbPQ/lQA3ofyp/X6/8AMw/sTBfyI8z/AOGdvhP/ANC8f+/rUf8ADO3wn/6F4/8Af1q9M5/umjn+6aX16v8AzMP7EwX/AD7R5of2dPhT/wBC8f8Av81N/wCGdPhV/wBC8f8Av61emkE/wGk2n+4aax9f+Zg8jwT+wjzT/hnP4UFiP+EePP8A01arFj8Avhhp1yt3Z6BslTofNNeiEEjaSQfpS7Tx8pI7molja8tHJmlPKMHSd1BXOVHwz8I9rBx/21NH/CtPCX/PjJ/39aup2n+6aNp/umuZu56KVlY5b/hWnhH/AJ8ZP+/rUf8ACtPCP/PjJ/39aup2n+6aNh7KaBnLf8K08Jf8+En/AH9aj/hWnhL/AJ8JP+/rV1JDd0NADdkNAHLH4a+Ev+fCQ/8AbVqT/hWnhP8A58JP+/zV1O0/3DRtP9w0Acv/AMK08In5Rp8mP+uzVXvfhP4Hvo/Ju9LZl95WNdkEI4OfypMMOCpIrKrRjiI8s1oVCq6UrxPPj8Cvhz/DouP+2ho/4UV8Of8AoDH/AL+GvQQpH8JoCE/wGvLWQZfJ3nTR3vM8Sl7s39558fgV8Ou2jH/v6aT/AIUV8O/+gN/5FNehFTn7po2kdFNP+wMv/wCfaJ/tTF/zv7zz7/hRfw7/AOgMf+/rUn/Civh3/wBAb/yK1eh4/wBk0Y/2TR/YGX/8+0P+1MX/ADv7zz0/Ar4ddtGP/f00n/Ci/h0P+YL/AORGr0LaT2NJtP8AdNC4fwFv4aB5pi/53955/wD8KL+HWMHRj/38NZ2sfs2/CfXIfJ1PQPNT08xq9SKnGMGja4GApAqqWTYKjK6poazXFr7b+88U/wCGP/gX28J/+Rmo/wCGP/gZ/wBCn/5Hava9r/5FG1/Su76rSX2TX+2sb/z8f3s8UP7IHwK7+Ff/ACM1A/ZA+BX/AEKn/kZq9rCnupoKHspo+q0v5Q/trHf8/X97PFP+GPvgX/0Kf/kZqP8Ahj/4Gf8AQp/+R2r2sq3YGja/p+lH1Wl/KH9tY7/n6/vZ4mf2QPgX28K/+RmpR+x/8CyP+RU/8jNXtW1v7hpdpHRTR9Vpdg/trHf8/H97PF4v2QvgdG4b/hE+VOR++avSvCvgnw54LshYeH9MjtYlGPlHP51vBTkkgk+lCq3J2EVUcPGHQwrZlisQuWrNteoCil2n+6fyo2n0NbX6HBs7oTpRS7T6GjafQ0AJSYFO2n0NGD6GgBM4oo2k9jS7T6GgBKM5pdp9DSbSOxoA2tq/3R+VG1f7o/KlooATav8AdH5UbV/uj8qWigBNq/3R+VG1f7o/KlooATav90flRtX+6PypaKAGlR2UflSbB6CnfjSHPehXFZMNqjsPypML/dFKtLkdKA2EwvoKMJ6D8qNvvRt96Bi/L6CjC+gpNvvRt96A0FwPQUbV/uj8qTHvRt96QC4X0FJhfQUbfejb70wDC/3R+VGF/uj8qXHSjHSgBML/AHR+VGF/uj8qXHSjHSkGgmF/uj8qML/dH5UuOlGOlAaCYX+6Pyowv90flS46UY96YaBtX+6PypNo/uj8qdSZFAtRNq/3RRgDHyilyKTbS1GLtX+6Pyo2r/dH5UYo20C0Dav90flRtX+6Pyo20baA0Dav90flRtX+6PypNvvS7aY9g2r/AHR+VN2+wp9FK7Fa4m1f7o/Kjav90flS0UxibV/uj8qNq/3R+VLRQAm1f7o/Kjav90flS0UAJtX+6Pyo2r/dH5UtFACbV7gflSYT0H5U6kyKV2hbjSF7AUYHoKfRRa4xuxfQUbF9BTqKYDdi+go2L6D8qdRQAzavt+VKFX0FKDmlpagN2p3Ao2r/AHRTqTFMWobV/uj8qNq/3R+VGKNtINA2r/dH5UbV/uj8qNtG2gNA2r/dH5UbV/uj8qNtG2gNA2r/AHR+VG1f7o/KjbRtoDQQKOuBS7V9BR+NLRqwGkKP4RS7V/uj8qWm7fejYYu1f7o/Kjav90flS0UwE2r/AHR+VG1f7o/KlooATav90flRtX+6PypaKAE2r/dH5UbV/uj8qWigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAP/9k=\"}")
As soon as "killer string" sent to server it stuck and not doing anything anymore. Behavior very similar to what we get in production:
lwsts[30119]: pmd test: RX len 41
jj1Brv6KAPPf+FQwX76jdeJvFWp6xe39gNNW5ljgi
lwsts[30119]: pmd test: RX len 127
aGAMWG0RIqk7jnLA10Wg+F73RZ7SWfxVqmoJa2IsvJn8tY5CGBEpVFA3gDbxgYzxmugooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiii
C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
- the only way to kill server
`ikuchin@dev:~$ killall libwebsockets-test-server`
**test-fraggle**
Test with huge input form client to server also enforce test-fraggile to stop handling any packets/requests.
May I ask you couple questions about socket operations ? (I know this is not the right place for teaching, but give me a chance...)
We look for a way sending files over WebSocket. We can't use the XMLHttpRequest, because of the state-machine and proprietary-protocol used for message exchange.
Also, I tried to send non-Unicode symbols (for ex: russian alphabet) over ws and some symbols get broken. Therefore we encode files to base64 and sending them.
From WS architecture point: what is the right approach to send files over ws ? Is it possible to send binary/non-utf8 data to WebSocket ?
accordingly to callgrind all the costs after stuck associated with lib/service.c:lws_service_flag_pending
ikuchin@dev:/usr/src/libwebsockets-2.2.0-20170325/build_debug/bin$ callgrind_annotate callgrind.out.1562
--------------------------------------------------------------------------------
Profile data file 'callgrind.out.1562' (creator: callgrind-3.11.0)
--------------------------------------------------------------------------------
I1 cache:
D1 cache:
LL cache:
Timerange: Basic block 0 - 472799998
Trigger: Program termination
Profiled target: ./libwebsockets-test-server (PID 1562, part 1)
Events recorded: Ir
Events shown: Ir
Event sort order: Ir
Thresholds: 99
Include dirs:
User annotated:
Auto-annotation: off
--------------------------------------------------------------------------------
Ir
--------------------------------------------------------------------------------
10,401,599,950 PROGRAM TOTALS
--------------------------------------------------------------------------------
Ir file:function
--------------------------------------------------------------------------------
10,401,599,950 /usr/src/libwebsockets-2.2.0-20170325/lib/service.c:lws_service_flag_pending [/usr/src/libwebsockets-2.2.0-20170325/build_debug/bin/libwebsockets-test-server]
Ok... I will try it in the morning.
Ws defines two message types, TEXT and BINARY. TEXT must carry UTF-8. BINARY has no restriction... this is the meaning of the flag you give lws_write().
It should be fixed... I added all three "strings of death" to the test HTML and it can pass them OK.
Patch added on v2.0-stable. v2.1-stable, v2.2-stable and master.
Thank you for fixing.
I run huge-file-testing on test-fraggle ..... as soon as WebSocket client-initiated-closed, libwebsocket-test-fraggle drops "segmentation fault".
Does it looks like an issue ? I can share core-dump.
*** Error in `./libwebsockets-test-fraggle': munmap_chunk(): invalid pointer: 0x0000000000b5e0e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fd5c167e7e5]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7fd5c168aae8]
./libwebsockets-test-fraggle[0x414520]
./libwebsockets-test-fraggle[0x41454e]
./libwebsockets-test-fraggle[0x406b19]
./libwebsockets-test-fraggle[0x407e18]
./libwebsockets-test-fraggle[0x407d4a]
./libwebsockets-test-fraggle(lws_service_fd_tsi+0xd1b)[0x40c404]
./libwebsockets-test-fraggle(_lws_plat_service_tsi+0x39c)[0x41ed83]
./libwebsockets-test-fraggle(lws_plat_service+0x25)[0x41ee29]
./libwebsockets-test-fraggle(lws_service+0x20)[0x40c482]
./libwebsockets-test-fraggle[0x406964]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd5c1627830]
./libwebsockets-test-fraggle(_start+0x29)[0x405d19]
======= Memory map: ========
00400000-00438000 r-xp 00000000 fc:00 927450 /home/ikuchin/src/libwebsockets-2.2.0/libwebsockets/build_debug/bin/libwebsockets-test-fraggle
00637000-00638000 r--p 00037000 fc:00 927450 /home/ikuchin/src/libwebsockets-2.2.0/libwebsockets/build_debug/bin/libwebsockets-test-fraggle
00638000-00639000 rw-p 00038000 fc:00 927450 /home/ikuchin/src/libwebsockets-2.2.0/libwebsockets/build_debug/bin/libwebsockets-test-fraggle
00b5d000-00b8c000 rw-p 00000000 00:00 0 [heap]
7fd5c11ed000-7fd5c1203000 r-xp 00000000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd5c1203000-7fd5c1402000 ---p 00016000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd5c1402000-7fd5c1403000 rw-p 00015000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd5c1403000-7fd5c1406000 r-xp 00000000 fc:00 783666 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd5c1406000-7fd5c1605000 ---p 00003000 fc:00 783666 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd5c1605000-7fd5c1606000 r--p 00002000 fc:00 783666 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd5c1606000-7fd5c1607000 rw-p 00003000 fc:00 783666 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd5c1607000-7fd5c17c6000 r-xp 00000000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fd5c17c6000-7fd5c19c6000 ---p 001bf000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fd5c19c6000-7fd5c19ca000 r--p 001bf000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fd5c19ca000-7fd5c19cc000 rw-p 001c3000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fd5c19cc000-7fd5c19d0000 rw-p 00000000 00:00 0
7fd5c19d0000-7fd5c19e8000 r-xp 00000000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd5c19e8000-7fd5c1be7000 ---p 00018000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd5c1be7000-7fd5c1be8000 r--p 00017000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd5c1be8000-7fd5c1be9000 rw-p 00018000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd5c1be9000-7fd5c1bed000 rw-p 00000000 00:00 0
7fd5c1bed000-7fd5c1e07000 r-xp 00000000 fc:00 784742 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd5c1e07000-7fd5c2006000 ---p 0021a000 fc:00 784742 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd5c2006000-7fd5c2022000 r--p 00219000 fc:00 784742 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd5c2022000-7fd5c202e000 rw-p 00235000 fc:00 784742 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7fd5c202e000-7fd5c2031000 rw-p 00000000 00:00 0
7fd5c2031000-7fd5c208f000 r-xp 00000000 fc:00 784746 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd5c208f000-7fd5c228f000 ---p 0005e000 fc:00 784746 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd5c228f000-7fd5c2293000 r--p 0005e000 fc:00 784746 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd5c2293000-7fd5c229a000 rw-p 00062000 fc:00 784746 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7fd5c229a000-7fd5c22b3000 r-xp 00000000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd5c22b3000-7fd5c24b2000 ---p 00019000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd5c24b2000-7fd5c24b3000 r--p 00018000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd5c24b3000-7fd5c24b4000 rw-p 00019000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd5c24b4000-7fd5c24da000 r-xp 00000000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fd5c26c7000-7fd5c26cc000 rw-p 00000000 00:00 0
7fd5c26d6000-7fd5c26d9000 rw-p 00000000 00:00 0
7fd5c26d9000-7fd5c26da000 r--p 00025000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fd5c26da000-7fd5c26db000 rw-p 00026000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fd5c26db000-7fd5c26dc000 rw-p 00000000 00:00 0
7fffacff4000-7fffad015000 rw-p 00000000 00:00 0 [stack]
7fffad050000-7fffad052000 r--p 00000000 00:00 0 [vvar]
7fffad052000-7fffad054000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
Yes it's an 'issue' one way or another, either from fraggle or lws. But your backtrace is not useful.
Can you do cmake .. -DCMAKE_BUILD_TYPE=DEBUG
and re-make / retry running under valgrind?
Testing the test server under valgrind here, simply using the 'pmd test' button to send the strings-of-death is completely clean. So it needs something more.
Our app-server can't start with the same stack. Before last change - works perfectly,
I diff'ed my chat-server(based on lws-server-pthread 2.0.x) server and the one from 2.2.0 distributive - they are the same except
1) dumb_increment_protocol frame size (my - 32, yours - 10)
2) my is disabled http_protocol (from enum demo_protocols
and static struct lws_protocols protocols
)
I tried to build lws-server-pthread w/o http and it works.
Here is the valgrind and gdb traceback's. Hope it help. If you wish I can grant you the access to my server. You'll take whatever you need.
--------- valgrind (look into lws_service_fd_tsi : service.c:728)
ikuchin@dev:~/src/dev.connme.ru$ valgrind ./chat-server -d65535
==27279== Memcheck, a memory error detector
==27279== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==27279== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==27279== Command: ./chat-server -d65535
==27279==
lwsts[27279]: libwebsockets test server pthreads - license LGPL2.1+SLE
lwsts[27279]: (C) Copyright 2010-2016 Andy Green <andy@warmcat.com>
Using resource path "/usr/local/share/libwebsockets-test-server"
lwsts[27279]: Initial logging level 65535
lwsts[27279]: Libwebsockets version: 2.2.0 ikuchin@dev.connme.ru-v2.0.0-339-g5dd1b41
lwsts[27279]: IPV6 not compiled in
lwsts[27279]: libev support not compiled in
lwsts[27279]: libuv support not compiled in
lwsts[27279]: LWS_DEF_HEADER_LEN : 4096
lwsts[27279]: LWS_MAX_PROTOCOLS : 5
lwsts[27279]: LWS_MAX_SMP : 32
lwsts[27279]: SPEC_LATEST_SUPPORTED : 13
lwsts[27279]: sizeof (*info) : 392
lwsts[27279]: SYSTEM_RANDOM_FILEPATH: '/dev/urandom'
lwsts[27279]: default timeout (secs): 20
lwsts[27279]: Threads: 1 each 1024 fds
lwsts[27279]: mem: context: 9272 bytes (5176 ctx + (1 thr x 4096))
lwsts[27279]: mem: http hdr rsvd: 30848 bytes (1 thr x (4096 + 3616) x 4))
lwsts[27279]: mem: pollfd map: 8192
lwsts[27279]: mem: platform fd map: 8192 bytes
lwsts[27279]: Compiled with OpenSSL support
lwsts[27279]: SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
lwsts[27279]: Creating Vhost 'default' port 7681, 3 protocols, IPv6 off
lwsts[27279]: insert_wsi_socket_into_fds: 0xc2be220: tsi=0, sock=8, pos-in-fds=1
lwsts[27279]: Listening on port 7681
lwsts[27279]: LWS_MAX_EXTENSIONS_ACTIVE: 2
lwsts[27279]: mem: per-conn: 568 bytes + protocol rx buf
lwsts[27279]: canonical_hostname = dev.connme.ru
lwsts[27279]: pthread_create: 0 created
lwsts[27279]: lws_protocol_init
==27279== Thread 3:
==27279== Jump to the invalid address stated on the next line
==27279== at 0x328: ???
==27279== by 0x587A134: lws_service_fd_tsi (service.c:728)
==27279== by 0x588DC69: _lws_plat_service_tsi (lws-plat-unix.c:156)
==27279== by 0x587AEAF: lws_service_tsi (service.c:1285)
==27279== by 0x413582: thread_service(void*) (chat-server.cpp:202)
==27279== by 0x56576B9: start_thread (pthread_create.c:333)
==27279== by 0x63B682C: clone (clone.S:109)
==27279== Address 0x328 is not stack'd, malloc'd or (recently) free'd
==27279==
==27279==
==27279== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==27279== Bad permissions for mapped region at address 0x328
==27279== at 0x328: ???
==27279== by 0x587A134: lws_service_fd_tsi (service.c:728)
==27279== by 0x588DC69: _lws_plat_service_tsi (lws-plat-unix.c:156)
==27279== by 0x587AEAF: lws_service_tsi (service.c:1285)
==27279== by 0x413582: thread_service(void*) (chat-server.cpp:202)
==27279== by 0x56576B9: start_thread (pthread_create.c:333)
==27279== by 0x63B682C: clone (clone.S:109)
==27279==
==27279== HEAP SUMMARY:
==27279== in use at exit: 176,919 bytes in 48 blocks
==27279== total heap usage: 245 allocs, 197 frees, 542,065 bytes allocated
==27279==
==27279== LEAK SUMMARY:
==27279== definitely lost: 0 bytes in 0 blocks
==27279== indirectly lost: 0 bytes in 0 blocks
==27279== possibly lost: 37,358 bytes in 33 blocks
==27279== still reachable: 139,561 bytes in 15 blocks
==27279== suppressed: 0 bytes in 0 blocks
==27279== Rerun with --leak-check=full to see details of leaked memory
==27279==
==27279== For counts of detected and suppressed errors, rerun with: -v
==27279== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed
--------- gdb (look into lws_protocol_init)
ikuchin@dev:~/src/dev.connme.ru$ gdb ./chat-server ./core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
...skip...
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./chat-server...done.
[New LWP 27286]
[New LWP 27285]
[New LWP 27284]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./chat-server -d65535'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000328 in ?? ()
[Current thread is 1 (Thread 0x7ff27f8fb700 (LWP 27286))]
(gdb) where
#0 0x0000000000000328 in ?? ()
#1 0x00007ff286ac3558 in lws_protocol_init (context=0x1d8a110)
at /usr/src/libwebsockets-2.2.0-20170326/lib/context.c:198
#2 0x00007ff286abc135 in lws_service_fd_tsi (context=0x1d8a110, pollfd=0x0, tsi=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/service.c:728
#3 0x00007ff286acfc6a in _lws_plat_service_tsi (context=0x1d8a110, timeout_ms=50, tsi=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/lws-plat-unix.c:156
#4 0x00007ff286abceb0 in lws_service_tsi (context=0x1d8a110, timeout_ms=50, tsi=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/service.c:1285
#5 0x0000000000413583 in thread_service (threadid=0x0) at chat-server.cpp:202
#6 0x00007ff286cf06ba in start_thread (arg=0x7ff27f8fb700) at pthread_create.c:333
#7 0x00007ff285fe382d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) quit
Hmmm I sincerely doubt this is anything to do with the last patch from me. At that point there are no incoming connections negotiated or pmd. You can try leave everything the same and revert the patch. But why aren't the lws test apps affected by something so basic?
I can't help you debug your app, that part is your problem. Busted things in lws are arguably my problem and I can usually help with those. However when I see pthreads in the bt I wonder if you are racing something critical during init. If it's v2.2-stable then this seems somehow related to your struct lws_protocols. But I think whatever's up with that is in your app.
Got you, understand. Do you want me to close the case or I can take the debug from server-fraggle ?
A usable bt / valgrind on that, and tell me which lws version it was using, would be good thanks. It looks like it's freeing an invalid pointer, knowing where the free() is in the code might be enough to know where to look.
2.2-stable Fraggle-server under valgrind control is going to be very slowly and not able to work appropriate time. I waited about hour and fraggle took about 2 mins to receive new chunk. Here is GDB bt/bt full/list and local variables. Let me know if this is not enough , I'll work on more attempts with valgrind. My assumption: lib/alloc.c -> _realloc(0)
Spamming session over, len = 196227. sum = 0x17DFC62
Spamming 419 random fragments
Terminating...
[2017/03/26 22:08:41:2592] NOTICE: lws_context_destroy: ctx 0x2582b50
*** Error in `./libwebsockets-test-fraggle': free(): invalid next size (normal): 0x00000000025810d0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe76bef17e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fe0a)[0x7fe76bef9e0a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fe76befd98c]
./libwebsockets-test-fraggle[0x4128d7]
./libwebsockets-test-fraggle[0x412905]
./libwebsockets-test-fraggle(lws_context_destroy+0x3ec)[0x41259b]
./libwebsockets-test-fraggle[0x4050da]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fe76be9a830]
./libwebsockets-test-fraggle(_start+0x29)[0x404449]
======= Memory map: ========
00400000-00432000 r-xp 00000000 fc:00 653035 /home/ikuchin/111111111/libwebsockets-test-fraggle
00631000-00632000 r--p 00031000 fc:00 653035 /home/ikuchin/111111111/libwebsockets-test-fraggle
00632000-00633000 rw-p 00032000 fc:00 653035 /home/ikuchin/111111111/libwebsockets-test-fraggle
02581000-025b0000 rw-p 00000000 00:00 0 [heap]
7fe764000000-7fe764021000 rw-p 00000000 00:00 0
7fe764021000-7fe768000000 ---p 00000000 00:00 0
7fe76bc64000-7fe76bc7a000 r-xp 00000000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe76bc7a000-7fe76be79000 ---p 00016000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe76be79000-7fe76be7a000 rw-p 00015000 fc:00 783667 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fe76be7a000-7fe76c039000 r-xp 00000000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fe76c039000-7fe76c239000 ---p 001bf000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fe76c239000-7fe76c23d000 r--p 001bf000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fe76c23d000-7fe76c23f000 rw-p 001c3000 fc:00 783670 /lib/x86_64-linux-gnu/libc-2.23.so
7fe76c23f000-7fe76c243000 rw-p 00000000 00:00 0
7fe76c243000-7fe76c25b000 r-xp 00000000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe76c25b000-7fe76c45a000 ---p 00018000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe76c45a000-7fe76c45b000 r--p 00017000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe76c45b000-7fe76c45c000 rw-p 00018000 fc:00 783682 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fe76c45c000-7fe76c460000 rw-p 00000000 00:00 0
7fe76c460000-7fe76c479000 r-xp 00000000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe76c479000-7fe76c678000 ---p 00019000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe76c678000-7fe76c679000 r--p 00018000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe76c679000-7fe76c67a000 rw-p 00019000 fc:00 783603 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fe76c67a000-7fe76c6a0000 r-xp 00000000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fe76c88e000-7fe76c892000 rw-p 00000000 00:00 0
7fe76c89c000-7fe76c89f000 rw-p 00000000 00:00 0
7fe76c89f000-7fe76c8a0000 r--p 00025000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fe76c8a0000-7fe76c8a1000 rw-p 00026000 fc:00 783373 /lib/x86_64-linux-gnu/ld-2.23.so
7fe76c8a1000-7fe76c8a2000 rw-p 00000000 00:00 0
7fff2bcb5000-7fff2bcd6000 rw-p 00000000 00:00 0 [stack]
7fff2bdbf000-7fff2bdc1000 r--p 00000000 00:00 0 [vvar]
7fff2bdc1000-7fff2bdc3000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Aborted (core dumped)
#0 0x00007fe76beaf428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fe76beaf428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fe76beb102a in __GI_abort () at abort.c:89
#2 0x00007fe76bef17ea in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7fe76c00a2e0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007fe76bef9e0a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>,
str=0x7fe76c00a438 "free(): invalid next size (normal)", action=3) at malloc.c:5004
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3865
#5 0x00007fe76befd98c in __GI___libc_free (mem=<optimized out>) at malloc.c:2966
#6 0x00000000004128d7 in _realloc (ptr=0x25810d0, size=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/alloc.c:60
#7 0x0000000000412905 in lws_realloc (ptr=0x25810d0, size=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/alloc.c:68
#8 0x000000000041259b in lws_context_destroy (context=0x2582b50)
at /usr/src/libwebsockets-2.2.0-20170326/lib/context.c:1037
#9 0x00000000004050da in main (argc=1, argv=0x7fff2bcd3758)
at /usr/src/libwebsockets-2.2.0-20170326/test-server/test-fraggle.c:380
51 static void *_realloc(void *ptr, size_t size)
52 {
53 if (size)
54 #if defined(LWS_PLAT_OPTEE)
55 return (void *)TEE_Realloc(ptr, size);
56 #else
57 return (void *)realloc(ptr, size);
58 #endif
59 else if (ptr)
60 free(ptr);
61 return NULL;
62 }
(gdb) bt full
#0 0x00007fe76beaf428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
resultvar = 0
pid = 13091
selftid = 13091
#1 0x00007fe76beb102a in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x3030316364623266, sa_sigaction = 0x3030316364623266},
sa_mask = {__val = {7075830554540059952, 8223625903103763300, 3472328295963457581,
4192904167887482928, 2314885531086893104, 2314885530818453536, 2314885530818453536,
8528445641706184736, 7378697234196427620, 3919933115663279718, 7378634724483280944,
7378697629483820646, 8223625903103619126, 3472328295963457581, 4192904167887482928,
2314885531086893104}}, sa_flags = 538976288, sa_restorer = 0x68}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007fe76bef17ea in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7fe76c00a2e0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
ap = <error reading variable ap (Attempt to dereference a generic pointer.)>
fd = 6
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
#3 0x00007fe76bef9e0a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>,
str=0x7fe76c00a438 "free(): invalid next size (normal)", action=3) at malloc.c:5004
buf = "00000000025810d0"
cp = <optimized out>
ar_ptr = <optimized out>
str = 0x7fe76c00a438 "free(): invalid next size (normal)"
action = 3
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3865
size = <optimized out>
fb = <optimized out>
nextchunk = <optimized out>
nextsize = <optimized out>
nextinuse = <optimized out>
prevsize = <optimized out>
bck = <optimized out>
fwd = <optimized out>
errstr = <optimized out>
locked = <optimized out>
#5 0x00007fe76befd98c in __GI___libc_free (mem=<optimized out>) at malloc.c:2966
ar_ptr = <optimized out>
p = <optimized out>
hook = <optimized out>
#6 0x00000000004128d7 in _realloc (ptr=0x25810d0, size=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/alloc.c:60
No locals.
#7 0x0000000000412905 in lws_realloc (ptr=0x25810d0, size=0)
at /usr/src/libwebsockets-2.2.0-20170326/lib/alloc.c:68
No locals.
#8 0x000000000041259b in lws_context_destroy (context=0x2582b50)
at /usr/src/libwebsockets-2.2.0-20170326/lib/context.c:1037
protocol = 0x632378 <protocols+56>
pt = 0x2582c30
vh = 0x0
wsi = {u = {http = {ah = 0x0, ah_wait_list = 0x0, preamble_rx = 0x0, stash = 0x0, preamble_rx_len = 0,
new_wsi_list = 0x0, filepos = 0, filelen = 0, fop_fd = 0x0, range = {start = 0, end = 0,
extent = 0, agg = 0, budget = 0, buf = '\000' <repeats 127 times>, pos = 0,
state = LWSRS_NO_ACTIVE_RANGE, start_valid = 0 '\000', end_valid = 0 '\000', ctr = 0 '\000',
count_ranges = 0 '\000', did_try = 0 '\000', inside = 0 '\000', send_ctr = 0 '\000'},
multipart_content_type = '\000' <repeats 63 times>, request_version = HTTP_VERSION_1_0,
connection_type = HTTP_CONNECTION_CLOSE, content_length = 0, content_remain = 0}, hdr = {ah = 0x0,
ah_wait_list = 0x0, preamble_rx = 0x0, stash = 0x0, preamble_rx_len = 0, ups = URIPS_IDLE,
ues = URIES_IDLE, lextable_pos = 0, current_token_limit = 0, esc_stash = 0 '\000',
post_literal_equal = 0 '\000', parser_state = 0 '\000'}, ws = {hdr = {ah = 0x0,
ah_wait_list = 0x0, preamble_rx = 0x0, stash = 0x0, preamble_rx_len = 0, ups = URIPS_IDLE,
ues = URIES_IDLE, lextable_pos = 0, current_token_limit = 0, esc_stash = 0 '\000',
post_literal_equal = 0 '\000', parser_state = 0 '\000'}, rx_ubuf = 0x0, rx_ubuf_alloc = 0,
rx_draining_ext_list = 0x0, tx_draining_ext_list = 0x0, time_next_ping_check = 0,
rx_packet_length = 0, rx_ubuf_head = 0, mask = "\000\000\000",
ping_payload_buf = '\000' <repeats 140 times>, ping_payload_len = 0 '\000', mask_idx = 0 '\000',
opcode = 0 '\000', rsv = 0 '\000', rsv_first_msg = 0 '\000', close_in_ping_buffer_len = 0 '\000',
utf8 = 0 '\000', stashed_write_type = 0 '\000', tx_draining_stashed_wp = 0 '\000', final = 0,
frame_is_binary = 0, all_zero_nonce = 0, this_frame_masked = 0, inside_frame = 0,
clean_buffer = 0, payload_is_close = 0, ping_pending_flag = 0, continuation_possible = 0,
owed_a_fin = 0, check_utf8 = 0, defeat_check_utf8 = 0, pmce_compressed_message = 0,
stashed_write_pending = 0, rx_draining_ext = 0, tx_draining_ext = 0, send_check_ping = 0}},
pending_timeout_limit = 0, context = 0x2582b50, vhost = 0x258f750, parent = 0x0, child_list = 0x0,
sibling_list = 0x0, protocol = 0x632340 <protocols>, same_vh_protocol_prev = 0x0,
same_vh_protocol_next = 0x0, timeout_list = 0x0, timeout_list_prev = 0x0, user_space = 0x0,
rxflow_buffer = 0x0, trunc_alloc = 0x0, active_extensions = {0x0, 0x0}, act_ext_user = {0x0, 0x0},
desc = {sockfd = 0, filefd = 0}, position_in_fds_table = 0, rxflow_len = 0, rxflow_pos = 0,
trunc_alloc_len = 0, trunc_offset = 0, trunc_len = 0, chunk_remaining = 0, cache_secs = 0,
hdr_parsing_completed = 0, http2_substream = 0, listener = 0, user_space_externally_allocated = 0,
socket_is_permanently_unusable = 0, rxflow_change_to = 0, more_rx_waiting = 0, conn_stat_done = 0,
---Type <return> to continue, or q <return> to quit---
cache_reuse = 0, cache_revalidate = 0, cache_intermediaries = 0, favoured_pollin = 0,
sending_chunked = 0, already_did_cce = 0, told_user_closed = 0, do_ws = 0, chunked = 0,
client_rx_avail = 0, client_http_body_pending = 0, extension_data_pending = 0, c_port = 0,
count_act_ext = 0 '\000', ietf_spec_revision = 0 '\000', mode = 0 '\000', state = 0 '\000',
state_pre_close = 0 '\000', lws_rx_parse_state = 0 '\000', rx_frame_type = 0 '\000',
pending_timeout = 0 '\000', pps = 0 '\000', tsi = 0 '\000', protocol_interpret_idx = 0 '\000',
redirects = 0 '\000', chunk_parser = 0 '\000', reason_bf = 0 '\000'}
n = 0
m = -1
__func__ = "lws_context_destroy"
#9 0x00000000004050da in main (argc=1, argv=0x7fff2bcd3758)
at /usr/src/libwebsockets-2.2.0-20170326/test-server/test-fraggle.c:380
n = 0
port = 7681
use_ssl = 0
context = 0x2582b50
opts = 0
interface_name = '\000' <repeats 127 times>
ads_port = ".N=\366\000\000\000\000\261$@\000\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "\020\244\350k\347\177\000\000\000\020\211l\347\177\000\000\210\032\212l\347\177\000\000\240\065\315+\377\177\000\000\000\000\000\000\000\000\000\000\220\065\315+\377\177\000\000&\260be\000\000\000\000GZ\000l\347\177\000\000\377\377\377\377\000\000\000\000\000\000\200\000\000\000\000\000\200\022\334+\377\177\000\000\000\027\212l\347\177\000\000\b\364El\347\177\000\000x!c\000\000\000\000\000 D@\000\000\000\000\000P7\315+\377\177", '\000' <repeats 26 times>, "\366\230hl\347\177\000\000\001", '\000' <repeats 17 times>...
iface = 0x0
wsi = 0x7fe76c245fc5
address = 0x0
server_port = 7681
info = {port = 7681, iface = 0x0, protocols = 0x632340 <protocols>, extensions = 0x427240 <exts>,
token_limits = 0x0, ssl_private_key_password = 0x0, ssl_cert_filepath = 0x0,
ssl_private_key_filepath = 0x0, ssl_ca_filepath = 0x0, ssl_cipher_list = 0x0,
http_proxy_address = 0x0, http_proxy_port = 0, gid = -1, uid = -1, options = 0, user = 0x0,
ka_time = 0, ka_probes = 0, ka_interval = 0, provided_client_ssl_ctx = 0x0, max_http_header_data = 0,
max_http_header_pool = 0, count_threads = 0, fd_limit_per_thread = 0, timeout_secs = 0,
ecdh_curve = 0x0, vhost_name = 0x0, plugin_dirs = 0x0, pvo = 0x0, keepalive_timeout = 0,
log_filepath = 0x0, mounts = 0x0, server_string = 0x0, pt_serv_buf_size = 0,
max_http_header_data2 = 0, ssl_options_set = 0, ssl_options_clear = 0, ws_ping_pong_interval = 0,
headers = 0x0, reject_service_keywords = 0x0, external_baggage_free_on_destroy = 0x0, fops = 0x0,
simultaneous_ssl_restriction = 0, _unused = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
Could you advice how to correctly remove lws instance, if CMAKE_INSTALL_PREFIX:PATH=/usr/local
?
I can do it manually (install_manifest.txt), but I'm not sure that I didn't miss anything (like ld.so.cache update).
One of my thoughts is that some marks from prev installation still alive.
About remove after make install
$ sudo rm -rf `cat build/install_manifest.txt`
will remove everything (check in the txt file first to see what it will remove, it should all be reasonable, not, eg, / ^^). Other general gotchas
your shell may remember it found the file on, eg, /usr/local/bin before and cache that; you can defeat it by running eg, /usr/bin/xxx...
explicitly.
run sudo ldconfig
to update the system's understanding of what libs are found where
The bt says we're dying in context destroy, but the line no for v2.2-stable "lib/context.c:1037" is the end of that function... so no clue which free. It's dying trying to free() 0x25810d0 but there's no clue in the dumps where it got that pointer from.
If I want to reproduce this myself, what exactly do I have to do?
I didn't test the master branch with d58353f-commit.
Reproducing steps: attached array elements in random order with pacing 0.2Hz have to be sent to fraggle-protocol-server about 10-15 mins, then stop, you'll get the crash.
My script provides with simple statistics:
requestsSentCounter (img1 size: 1976597): 53
requestsSentCounter (img2 size: 1267600): 78
requestsSentCounter (img3 size: 18022): 25
requestsSentCounter (img4 size: 24630): 32
responceReceivedCounter: 3187
Do you need it ? Otherwise I can test new patches on your behalf.
About debugging.
line 1037
(gdb) l 1037
1032
1033 lws_libev_destroyloop(context, n);
1034 lws_libuv_destroyloop(context, n);
1035 lws_libevent_destroyloop(context, n);
1036
1037 lws_free_set_NULL(context->pt[n].serv_buf);
1038 if (pt->ah_pool)
1039 lws_free(pt->ah_pool);
1040 if (pt->http_header_data)
1041 lws_free(pt->http_header_data);
This is the macro running lws_realloc(xxx, 0) ./private-libwebsockets.h:#define lws_free_set_NULL(P) do { lws_realloc(P, 0); (P) = NULL; } while(0)
, all that we need to find is the value of context->pt[n].serv_buf
.
1) n = 0 (from bt full) 2) context is the pointer to struct XXXX (you know better than me) 3) we need pt[n-th] element (actually pt[0]) 4) find value of serv_buf-member. (we know that final P=0x25810d0)
Finally context->pt[n].serv_buf = ptr 0x25810d0
GDB can easily provides with memory dump (below is just an example from another core-dump, I lost the original):
(gdb) x /10xw 0x2298c30
0x2298c30: 0x00000000 0x00000000 0x00000000 0x00000000
0x2298c40: 0xffffffff 0x00000000 0x00000000 0x00000000
0x2298c50: 0x00000000 0x00000000
Hm... it's not like I don't know how to use gdb ^^
It's that the source debug line info is misleading if you don't tell me the exact version. Going by your filepaths before
/usr/src/libwebsockets-2.2.0-20170326/lib/context.c:1037
V2.2-stable takes me to the wrong place, the end of the function.
https://github.com/warmcat/libwebsockets/blob/v2.2-stable/lib/context.c#L1037
Now you mentioned you switched to master I can look in the right place.
Anyway this is some form of heap corruption, it's not complaining about the content of serv_buf, it doesn't care. It's complaining that the malloc housekeeping data held in a linked-list just before each allocation is invalid somewhere by that point. The problem happened earlier and valgrind is very good at spotting out-of-bounds writes.
If you disable extensions in cmake, ie, no pmd, still get the problem?
my fault. I forgot to check-out v2.2-stable , but the folder name brings the confusion. Previous test had run on master-branch.
Now I rerun test on v2.2-stable
I pushed a small patch that may help (or may change nothing) on v2.0-stable, v2.1-stable. v2.2-stable and master.
ext-disabled - no crash
With "small patch" integrated-in and ext-enabled - no crash.
Yay... is that it then?
seems you are right long story done. you are amazing !
Hello lws-team,
Nice to meet you again ! This time I actually can't complain on lws-library, due to issue reproducing with no error's in log. But let me rise my concern, hope you'll give me direction where should I dig in.
Our project having very intensive exchange over WebSockets in peak hours. After working few hours it comes to stuck state and can't send/receive/close connection.
Tested libs: 1) 2.1.0 with fix from my previous issue 2) 2.2.0 with fix from my previous issue
------ debug1: last lines in debug (pay attention to draining 1 and final 0) - > stuck
-------------- DEBUG2: last lines in debug (pay attention to draining 1 and final 1) -> stuck
It is easily reproduces on production environment by waiting several hours, but the repro on test-server end up with different log. To reproduce issue on test-fraggle, I took 2 big images (nothing special in there: https://github.com/IvanKuchin/lws-quality-assurance/tree/master/image_example), convert them to blob and sending to test-fraggle. I can't paste source of html here because of the volume, you can get it https://github.com/IvanKuchin/lws-quality-assurance/tree/master/src. ---------------------- Repro step
I looked through autobahn tests and they sending even bigger files over WebSocket (20 MB). This is looks strange to me.