search

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.49k stars 1.9k forks source link

ValidationError when running Rest API with PromptNode and PromptTemplate #4364

Closed TuanaCelik closed 1 year ago

TuanaCelik commented 1 year ago

There is a bug while running the Rest API with a pipeline that includes a PromptNode and PromptTemplate that was noticed by @rolandtannous and @recrudesce An example YAML for reproducing is below. The result we get is sometimes a response where the answers=[] is empty. However, whether we get a response or anything at all, the logs of the rest api actually contain the answer, followed by an exception. See below an example response followed by an exception. As you can see, it includes the answer it should return: "answers": ["The Biglan study included two universities."]

But then it fails.

03/09/2023 11:02:19 AM {"request": "query='How many universities did the Biglan study include?' params=None debug=False", "response": {"answers": ["The Biglan study included two universities."], "invocation_context": {"query": "How many universities did the Biglan study include?", "documents": ["database is needed, however. The learning-style\ndata came from a single occupation, and in the case of some academic areas, sample\nsizes were small. Biglan\u2019s study, on the other hand, was limited to two universities, and\ndifferences here could be attributed to the specific characteristics of these academic\ndepartments.\nIn search of a more extensive and representative sample, data collected in the Carnegie\nCommission on Higher Education\u2019s 1969 study of representative American colleges and\nuniversities were examined. These data consisted of 32,963 questionnaires from graduate\nstudents in 158 institutions and 60,028 questionnaires from faculty in 303 institutions.\nUsing tabulations of these data reported in Feldman (1974), ad hoc indices were created\nof the abstract/concrete and active/reflective dimensions for the 45 academic fields iden-\ntified in the study. The abstract/concrete index was based on graduate student responses\nto two questions asking how important an undergraduate background in mathematics or\nhumanities was for their fields. The mathematics and humanities questions were highly\n180\nExperiential Learning\nnegatively correlated (\u2013.78). The index was computed using the percentage of graduate-\nstudent respondents who strongly agreed that either humanities or mathematics was very\nimportant:\n% Math important + (100 - % Humanities important)\n\n2\nThus, high index", "academic areas at Illinois are mapped on\nthis two-dimensional space (Figure 5.3), we see a great similarity between the pattern of\nBiglan\u2019s data and the structure of knowledge and learning described in Figure 5.2. Busi-\nness (assumed equivalent to accounting and finance) is accommodative in learning style\nand contextualist in knowledge structure. Engineering fits with convergent learning and\nformist knowledge. Physics, mathematics, and chemistry are related to assimilative learn-\ning and mechanistic knowledge, and the humanistic fields\u2014history, political science,\nEnglish, and psychology\u2014fall in the divergent, organistic quadrant. Foreign languages,\neconomics, and sociology were divergent in Biglan\u2019s study rather than assimilative as in\nFigure 4.4. Biglan also reported that the pattern of academic-area relationships in the\nsmall-college data was very similar to that in the Illinois data.\nThese two studies suggest that the two basic dimensions of experiential learning theory,\nabstract/concrete and active/reflective, are major dimensions of differentiation among\nacademic disciplines. A more extensive database is needed, however. The learning-style\ndata came from a single occupation, and in the case of some academic areas, sample\nsizes were small. Biglan\u2019s study, on the other hand, was limited to two universities, and\ndifferences here could be attributed to the", "examine how others perceive the differences between academic disciplines and\nwhether these perceptions are congruent with the structure of knowledge and learn-\ning. Anthony Biglan (1973a) used a method well suited to answer these questions in\nhis  studies of faculty members at the University of Illinois and a small western college.\nUsing the technique of multidimensional scaling, he analyzed the underlying structures\nof scholars\u2019 judgments about the similarities of subject matter in different academic dis-\nciplines. The procedure required faculty members to group subject areas on the basis of\nsimilarity without any labeling of the groupings. Through a kind of factor analysis, the\nsimilarity groupings are then mapped onto an n-dimensional space where n is deter-\nmined by goodness of fit and interpretability of the dimensions. The two dimensions\naccounting for the most variance in the University of Illinois data were interpreted by\nBiglan to be hard-soft and pure-applied. When academic areas at Illinois are mapped on\nthis two-dimensional space (Figure 5.3), we see a great similarity between the pattern of\nBiglan\u2019s data and the structure of knowledge and learning described in Figure 5.2. Busi-\nness (assumed equivalent to accounting and finance) is", "knowledge\nstructure of academic disciplines, let us now examine a case study of undergraduate stu-\ndents in a well-known technological university (TECH).2\nData for the study (except for cumulative grade averages, which were obtained from\nthe registrar\u2019s office) were collected by means of a questionnaire that was sent to the\n720 TECH seniors two months before graduation. Four hundred and seven students\n(57 percent) responded to the questionnaire. Of these responses, 342 (43 percent) were\ncomplete enough to test the hypotheses in this study. The questionnaire included the\n2. For a detailed report, see Kolb and Goldman, 1973.\n245\nChapter 7 Learning and Development in Higher Education\nLearning Style Inventory, two scales measuring political alienation and anomie, ques-\ntions about plans for next year, career choice, degree of commitment to that career,\nundergraduate major, perception of academic workload, and involvement with peers.\nThese variables will be described in detail as the results are presented.\nFigure 7.1 shows the LSI scores of students with different departmental majors in those\ndepartments with ten or more students. Analysis of variance for the six learning-style\ndimensions by departmental major shows that reflective observation, active experimen-\ntation, and the combination score active-reflective all", ".  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 97\nThe Scientific Study of Individuality  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .", ".  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 87\nExperiential Learning and the Brain  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  . 87\nJames Zull and the Link between the Learning"], "answers": ["The Biglan study included two universities."]}, "_debug": {"Prompt": {"runtime": {"prompts_used": ["Given the context please generate a comprehensive answer. Context: database is needed, however. The learning-style\ndata came from a single occupation, and in the case of some academic areas, sample\nsizes were small. Biglan\u2019s study, on the other hand, was limited to two universities, and\ndifferences here could be attributed to the specific characteristics of these academic\ndepartments.\nIn search of a more extensive and representative sample, data collected in the Carnegie\nCommission on Higher Education\u2019s 1969 study of representative American colleges and\nuniversities were examined. These data consisted of 32,963 questionnaires from graduate\nstudents in 158 institutions and 60,028 questionnaires from faculty in 303 institutions.\nUsing tabulations of these data reported in Feldman (1974), ad hoc indices were created\nof the abstract/concrete and active/reflective dimensions for the 45 academic fields iden-\ntified in the study. The abstract/concrete index was based on graduate student responses\nto two questions asking how important an undergraduate background in mathematics or\nhumanities was for their fields. The mathematics and humanities questions were highly\n180\nExperiential Learning\nnegatively correlated (\u2013.78). The index was computed using the percentage of graduate-\nstudent respondents who strongly agreed that either humanities or mathematics was very\nimportant:\n% Math important + (100 - % Humanities important)\n\n2\nThus, high index; Question: How many universities did the Biglan study include?; Answer:"]}, "exec_time_ms": 959.26}}, "documents": ["<Document: id=5c89ba4410d08a889c9e3cafcfc4c7c4, content='database is needed, however. The learning-style\ndata came from a single occupation, and in the case ...'>", "<Document: id=b75bf4f5e28dc4aadce75da875d334dd, content='academic areas at Illinois are mapped on\nthis two-dimensional space (Figure 5.3), we see a great sim...'>", "<Document: id=cb0f6dbead5b310e07d21f385203e318, content='examine how others perceive the differences between academic disciplines and\nwhether these perceptio...'>", "<Document: id=f976859e44da320640c15ea1b233a766, content='knowledge\nstructure of academic disciplines, let us now examine a case study of undergraduate stu-\nd...'>", "<Document: id=db7f10a223bcaf588deb7e964ad5d03a, content='.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  ....'>", "<Document: id=3b2b8ee6ec0fd31fe53123468b09cef8, content='.  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  ....'>"], "root_node": "Query", "params": {"Query": {}, "Retriever": {}, "Prompt": {}}, "query": "How many universities did the Biglan study include?", "node_id": "Prompt"}, "time": "2.19"}
[2023-03-09 11:02:19 +0100] [49793] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/fastapi/applications.py", line 274, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/fastapi/routing.py", line 256, in app
    content = await serialize_response(
  File "/Users/tuanacelik/opt/anaconda3/lib/python3.9/site-packages/fastapi/routing.py", line 142, in serialize_response
    raise ValidationError(errors, field.type_)
pydantic.error_wrappers.ValidationError: 1 validation error for QueryResponse
response -> answers -> 0
  value is not a valid dict (type=type_error.dict)

An example YAML to reproduce this:

version: ignore

components:    # define all the building-blocks for Pipeline
  - name: DocumentStore
    type: ElasticsearchDocumentStore
    params:
      host: localhost
      embedding_dim: 1536
      recreate_index: True
  - name: Retriever
    type: EmbeddingRetriever
    params:
      api_key: API-KEY
      document_store: DocumentStore
      embedding_model: text-embedding-ada-002
      top_k: 6
  - name: lfqa_template
    params:
      name: lfqa
      prompt_text: "Given the context please generate a comprehensive, summarized answer. Context: $documents; Question: $query; Answer:"
    type: PromptTemplate
  - name: Prompt
    type: PromptNode
    params:
      api_key: API-KEY
      model_name_or_path: text-davinci-003
      default_prompt_template: lfqa_template
      output_variable: answers
  - name: FileTypeClassifier
    type: FileTypeClassifier
  - name: TextFileConverter
    type: TextConverter
  - name: Preprocessor
    type: PreProcessor
    params:
      split_by: word
      split_length: 180
      split_overlap: 40
      split_respect_sentence_boundary: False

pipelines:
  - name: query    # a sample extractive-qa Pipeline
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: Prompt
        inputs: [Retriever]
  - name: indexing
    nodes:
      - name: FileTypeClassifier
        inputs: [File]
      - name: TextFileConverter
        inputs: [FileTypeClassifier.output_1]
      - name: Preprocessor
        inputs: [TextFileConverter]
      - name: Retriever
        inputs: [Preprocessor]
      - name: DocumentStore
        inputs: [Retriever]

I also noticed that this type of template where there is more than one argument being provided is not tested in the unit tests.

bogdankostic commented 1 year ago

Hey @TuanaCelik, just had a closer look at this and the issue seems to be that there's no output_parser defined for PromptTemplate in the YAML. Therefore, PromptNode returns a list of plain strings while the REST API expects a list of Answer objects. output_parser should be set to AnswerParser in this case. There are two ways of doing this in YAML.

Either:

- name: lfqa_template
    params:
      name: lfqa
      prompt_text: "Given the context please generate a comprehensive, summarized answer. Context: $documents; Question: $query; Answer:"
      output_parser:
        type: AnswerParser
    type: PromptTemplate

Or:


- name: answer_parser
    type: AnswerParser
- name: lfqa_template
    params:
      name: lfqa
      prompt_text: "Given the context please generate a comprehensive, summarized answer. Context: $documents; Question: $query; Answer:"
      output_parser: answer_parser
    type: PromptTemplate
TuanaCelik commented 1 year ago

Do I understand correctly that with the introduction of AnswerParser this problem basically got resolved? We just have to make sure to add the output_parser?

bogdankostic commented 1 year ago

Yes, exactly.

bogdankostic commented 1 year ago

Closing this issue as it seems to be resolved with the introduction of AnswerParser.