Memory leakage (memory usage of containers never decrease after load test or day end)

[RabiRanaEF]

Rasa version : 2.3.1

Rasa SDK version : 2.3.1

Rasa X version : 0.37.1

Python version : 3.6.8

Operating system : Linux Centos

Issue : I setup RasaX with rasa-production replicas set to 8 with docker-compose, the memory consumption of Rasa containers is increased during load test using rest channel and after 2 days of load test memory does't decrease, please check docker stats.

My Experience : If i run load test again memory will increase more and more even Rasa will consume whole RAM.

Error (including full traceback) :

Before Load Test 

After Load Test 

Command or request that led to error :

Content of configuration file (config.yml) :

language: en
 pipeline:

*   name: WhitespaceTokenizer
*   name: RegexFeaturizer
*   name: LexicalSyntacticFeaturizer
*   name: CountVectorsFeaturizer
*   name: CountVectorsFeaturizer
     analyzer: char_wb
     min_ngram: 1
     max_ngram: 4
*   name: DIETClassifier
     epochs: 100
*   name: EntitySynonymMapper
*   name: FallbackClassifier
     threshold: 0.6
     policies:
*   name: MemoizationPolicy
*   name: RulePolicy
     core_fallback_threshold: 0.6
     core_fallback_action_name: utter_hello
     enable_fallback_prediction: true
     

Content of domain file (domain.yml) (if relevant):

session_config:
 session_expiration_time: 60
 carry_over_slots_to_new_session: true
 intents:

*   buttons
*   video
*   carousel
*   hello
*   contact
*   nlu_fallback
*   image
*   filemessage
*   location
*   image1
*   url
*   endConversation
     responses:
     utter_buttons:
*   buttons:
*   BgColor: '#FF0000'
     index: 1
     payload: /hello
     title: NSW
     type: postback
*   BgColor: '#FF0000'
     index: 2
     payload: /hello
     title: VIC
     type: postback
*   BgColor: '#FF0000'
     index: 3
     payload: /hello
     title: QLD
     type: postback
*   BgColor: '#FF0000'
     index: 4
     payload: /hello
     title: SA
     type: postback
*   BgColor: '#FF0000'
     index: 5
     payload: /hello
     title: WA
     type: postback
*   BgColor: '#FF0000'
     index: 6
     payload: /hello
     title: TAS
     type: postback
*   BgColor: '#FF0000'
     index: 7
     payload: /hello
     title: NT
     text: This service is only available to those located within Australia. Please
     indicate the State or Territory where you are currently located.
     utter_carousel:
*   custom:
     carousel:
     BgColor: '#808080'
     elements:
*   buttons:
*   BgColor: '#FFA500'
     buttonHeight: 1
     payload: /button1
     title: button 1
     type: postback
*   BgColor: '#FFA500'
     buttonHeight: 1
     payload: /button2
     title: button 2
     type: postback
     defaultAction:
     type: web_url
     url: [https://expertflow.com/#video-message](https://expertflow.com/#video-message)
     descriptionButton:
     actionBody: descriptionButton
     actionType: web_url
     text: subtitle
     textHeight: 2
     imageButton:
     actionBody: /example_intent
     actionType: postback
     imageHeight: 3
     imageUrl: [https://expertflow.com/any-image1.jpg](https://expertflow.com/any-image1.jpg)
     title: title
*   buttons:
*   BgColor: '#FFA500'
     buttonHeight: 1
     payload: /button1
     title: button 1
     type: postback
*   BgColor: '#FFA500'
     buttonHeight: 1
     payload: /button2
     title: button 2
     type: postback
     defaultAction:
     type: web_url
     url: [https://expertflow.com/#video-message](https://expertflow.com/#video-message)
     descriptionButton:
     actionBody: descriptionButton
     actionType: web_url
     text: subtitle
     textHeight: 2
     imageButton:
     actionBody: /example_intent
     actionType: postback
     imageHeight: 3
     imageUrl: [https://expertflow.com/any-image1.jpg](https://expertflow.com/any-image1.jpg)
     title: title
     maxHeight: 7
     maxWidth: 6
     utter_video:
*   custom:
     video:
     media: [https://example.com/downloadFileStream?filename=Zopt4kxya-vvvv.mp4](https://example.com/downloadFileStream?filename=Zopt4kxya-vvvv.mp4)
     size: 10000
     thumbnail: [https://example.com/downloadFileStream?filename=348s1rYHz-Annotation-2019-08-07-121320.jpg](https://example.com/downloadFileStream?filename=348s1rYHz-Annotation-2019-08-07-121320.jpg)
     utter_hello:
*   channel: web
     text: Welcome on web widget
*   channel: whatsapp
     text: Welcome on whatsapp
*   channel: facebook
     text: Welcome on facebook
*   channel: viber
     text: Welcome on viber
*   channel: hybridchat
     text: Welcome on hybridchat
*   text: Welcome on default
     utter_image:
*   custom:
     image:
     media: [https://example.com/downloadFileStream?filename=Zopt4kxya-vvvv.jpg](https://example.com/downloadFileStream?filename=Zopt4kxya-vvvv.jpg)
     size: 10000
     text: null
     utter_image1:
*   image: [https://picsum.photos/200/300](https://picsum.photos/200/300)
     text: Image message text
     utter_filemessage:
*   custom:
     file:
     fileName: file.txt
     media: [http://www.example.com/file.txt](http://www.example.com/file.txt)
     size: 200
     text: null
     utter_url:
*   custom:
     url:
     media: [https://www.example.com](https://www.example.com)
     text: null
     utter_contact:
*   custom:
     contact:
     name: John
     phone_number: '+972511123123'
     text: null
     utter_location:
*   custom:
     location:
     lat: '37.7898'
     lon: '-122.3942'
     text: null
     utter_endConversation:
*   text: end

[sara-tagger]

Thanks for raising this issue, @rctatman will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[tczekajlo]

@TyDunn the issue is not related to the infra team.

Also, there ware a fix related to a memory leak - https://github.com/RasaHQ/rasa/pull/8319 The fix should be available in the > 2.5.0 version.

[TyDunn]

Okay. Thanks. I moved it to the Fabric board for them to look into it

[TyDunn]

@RabiRanaEF Are you still facing this issue when using Rasa Open Source 2.5.0 or greater?

[RabiRanaEF]

@TyDunn, @tczekajlo thanks for your effort and response I will check it with latest Rasa 2.x version and will update you

[MahdiEsrafili]

@RabiRanaEF Hello. Could you resolve the memory issue?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jonra1993]

Hello, @RabiRanaEF were you able to solve this issue? I am facing something similar on Rasa 2.8.21 I am using Rasa API to update a model but each time I load a new model from local storage RAM increases on my container.

[alexweidauer]

Thank you for raising this issue about Rasa X. We decided to stop supporting the Community Edition (free version) of ‘Rasa X’ (see more info here: https://rasa.com/blog/rasa-x-community-edition-changes/). That’s why we’ve closed your issue. We suggest that Rasa Enterprise customers raise the issue with our customer support team if they haven’t done so already.