mfekadu
commented
4 years ago Additional Problems
-
the memory limits actually results in incorrect results on the
/askendpoint- e.g. getting and answer of
"I don't know"when given an explicit question like"What is Foaad Khosmood's email?"and there exists a row in the Professor's table with the data(Foaad, Khosmood, foaad@calpoly.edu) -
screenshots
heroku (notice memory quota && error R14 Memory Quota Exceeded
correct answer before scaling heroku
incorrect answer before scaling heroku and question was POSTed subsequent to the above screenshot
evidence of data in the database
correct answer after scaling heroku dyno
correct answer after scaling heroku dyno
correct answer after scaling heroku dyno
correct answer after scaling heroku dyno
NOT correct answer after scaling heroku dyno
NOT correct answer after scaling heroku dyno
- In the screenshot above you'll see the memory quota increased to 1 GB using the Standard-2x dyno and things seem fine for now.
- e.g. getting and answer of
For a moment, when the answers were incorrect, I hypothesized that the wrong DATABASE_NAME was set in the environment variables, but dev_feb_28_2020 is correct as of right now.
$ heroku run --app calpoly-csai-nimbus "env | grep DATABASE_NAME"
Running env | grep DATABASE_NAME on ⬢ calpoly-csai-nimbus... up, run.7815 (Standard-1X)
DATABASE_NAME=dev_feb_28_2020
$- also the command-line-output above says
Standard-1Xbut it likely takes a few minutes to propagate changes to the shell because the heroku dashboard shows me thatStandard-2Xis running.
Bug Description
The Heroku Standard 1x Dyno has a 512 MB memory quota and we have hit the limit after just
/askendpointScreenshots
screenshots
Proposed solutions
1. switch to GCP and choose an appropriately sized virtual machine
.github/workflows/*.ymlto handle GCP deployment2. scale the heroku dyno to standard-2x or something else
This would get expensive, quickly...
3. identify memory usage & performance improvements for our code
maybe avoid the use of
pandasinQAand useimport csvinsteadonly
read_csvis importedhttps://github.com/calpoly-csai/api/blob/48336b06f95e3676c5322d6760c34968b2d145d5/QA.py#L10
reference @zpdeng 's usage of
csvmodulehttps://github.com/calpoly-csai/api/blob/48336b06f95e3676c5322d6760c34968b2d145d5/database_wrapper.py#L426-L433
it may be a good idea to actually test if
pandasis truly hogging memorymaybe
spacy?maybe
nltk?maybe
SQLAlchemy?maybe
Flask?maybe
gcloud?maybe
another_package_expected_to_be_large?consider this output of
python3 -m memory_profiler flask_api.pymemory profile
``` {'entity': 'Dr. Khosmood', 'tag': 'PROF', 'normalized entity': 'Khosmood', 'input question': "What is Dr. Khosmood's phone number?", 'normalized question': "What is [PROF]'s phone number?", 'question class': "What is [PROF]'s phone number?"} Filename: /Users/mfekadu/GitHub/api/QA.py Line # Mem usage Increment Line Contents ================================================ 61 218.5 MiB 218.5 MiB @profile 62 def answer(self, extracted_vars): 63 218.5 MiB 0.0 MiB db_data = self._get_data_from_db(extracted_vars) 64 218.5 MiB 0.0 MiB return self._format_answer(extracted_vars, db_data) Filename: /Users/mfekadu/GitHub/api/nimbus.py Line # Mem usage Increment Line Contents ================================================ 22 215.0 MiB 215.0 MiB @profile 23 def answer_question(self, question): 24 218.5 MiB 3.5 MiB ans_dict = NIMBUS_NLP.predict_question(question) 25 218.5 MiB 0.0 MiB print(ans_dict) 26 218.5 MiB 0.0 MiB try: 27 218.5 MiB 0.0 MiB qa = self.qa_dict[ans_dict["question class"]] 28 except KeyError: 29 return "I'm sorry, I don't understand. Please try another question." 30 else: 31 218.5 MiB 0.0 MiB answer = qa.answer(ans_dict) 32 218.5 MiB 0.0 MiB if answer is None: 33 return("I'm sorry, I understand your question but was unable to find an answer. " 34 "Please try another question.") 35 else: 36 218.5 MiB 0.0 MiB return answer 127.0.0.1 - - [02/Mar/2020 01:28:41] "POST /ask HTTP/1.1" 200 - ```
git diff
```diff diff --git a/Pipfile b/Pipfile index 6069200..0f8e7ef 100644 --- a/Pipfile +++ b/Pipfile @@ -22,7 +22,7 @@ pytest = "==5.3.4" pyre-check = "==0.0.41" ## like the Unix `make` but better invoke = "==1.4.1" - +memory-profiler = "*" [packages] # REST API diff --git a/Pipfile.lock b/Pipfile.lock index 6575e89..e55e8ff 100644 --- a/Pipfile.lock +++ b/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "fb30d39142d3cc83d8909d9f4f4648a60ac33d4ec3a5a94d8dac7b90ef727a24" + "sha256": "c1f663e58339a2e67ba7d26a44722711969699a9998316437dcaef26cbbe8b80" }, "pipfile-spec": 6, "requires": { @@ -826,6 +826,13 @@ ], "version": "==0.6.1" }, + "memory-profiler": { + "hashes": [ + "sha256:23b196f91ea9ac9996e30bfab1e82fecc30a4a1d24870e81d1e81625f786a2c3" + ], + "index": "pypi", + "version": "==0.57.0" + }, "more-itertools": { "hashes": [ "sha256:5dd8bcf33e5f9513ffa06d5ad33d78f31e1931ac9a18f33d37e77a180d393a7c", diff --git a/QA.py b/QA.py index 250b065..06ecf0b 100644 --- a/QA.py +++ b/QA.py @@ -9,6 +9,8 @@ from Entity.Sections import Sections from database_wrapper import NimbusMySQLAlchemy from pandas import read_csv +from memory_profiler import profile + Extracted_Vars = Dict[str, Any] DB_Data = Dict[str, Any] DB_Query = Callable[[Extracted_Vars], DB_Data] @@ -33,6 +35,7 @@ class QA: A class for wrapping functions used to answer a question. """ + @profile def __init__(self, q_format, db_query, format_answer): """ Args: @@ -55,6 +58,7 @@ class QA: def _format_answer(self, extracted_vars, db_data): return self.format_answer(extracted_vars, db_data) + @profile def answer(self, extracted_vars): db_data = self._get_data_from_db(extracted_vars) return self._format_answer(extracted_vars, db_data) @@ -66,6 +70,7 @@ class QA: return hash(self.q_format) +@profile def create_qa_mapping(qa_list): """ Creates a dictionary whose values are QA objects and keys are the question @@ -186,6 +191,7 @@ def yes_no(a_format, pred=None): return functools.partial(_yes_no, a_format, pred) +@profile def generate_fact_QA(csv): df = read_csv(csv) text_in_brackets = r'\[[^\[\]]*\]' diff --git a/flask_api.py b/flask_api.py index d6478c5..34fe17b 100755 --- a/flask_api.py +++ b/flask_api.py @@ -18,6 +18,8 @@ from modules.validators import WakeWordValidator, WakeWordValidatorError from nimbus import Nimbus +from memory_profiler import profile + BAD_REQUEST = 400 SUCCESS = 200 @@ -44,6 +46,7 @@ def generate_session_token() -> str: return "SOME_NEW_TOKEN" +@profile @app.route('/ask', methods=['POST']) def handle_question(): """ diff --git a/nimbus.py b/nimbus.py index 7d6bdfc..729cf62 100644 --- a/nimbus.py +++ b/nimbus.py @@ -9,14 +9,17 @@ from werkzeug.exceptions import BadRequestKeyError from QA import create_qa_mapping, generate_fact_QA from nimbus_nlp.NIMBUS_NLP import NIMBUS_NLP +from memory_profiler import profile class Nimbus: + @profile def __init__(self): self.qa_dict = create_qa_mapping( generate_fact_QA("q_a_pairs.csv") ) + @profile def answer_question(self, question): ans_dict = NIMBUS_NLP.predict_question(question) print(ans_dict) (END) ```