Publish app to more channels such as Microsoft Teams, and extend citation

[mhalomari]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

NA

Any log messages given by the failure

NA

Expected/desired behavior

NA

OS and Version?

Windows 10.

azd version?

azd version 1.1.0

Versions

NA

Mention any other details that might be useful

I deployed the demo and it is amazing working on our data within Azure and would love to:

1. publish the app to more channels, similar to Power Virtual Agents, for example as a Microsoft Teams app, It would be great and highly appreciated if you advise on how to do this, please.
2. When a citation is shown after an answer, it links to a single page of one of the sources. Is it possible to show both the page and the full document? sometimes the answer is partially shown from the bottom of a page and the remaining text is missing because it exists on the next page which is not shown.

Many thanks

---

Thanks! We'll be in touch soon.

[pamelafox]

1. Nice idea, I've asked a member of the Teams team to brainstorm how to publish the app that way, they'll share their ideas on this thread.

2. Yes, that should be possible, with some code modification. Here is the relevant code:

   In prepdocs.py, upload_blobs splits PDFs into pages and uploads each page: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/scripts/prepdocs.py#L31

   Also in prepdocs.py, create_sections stores the "sourcepage" field with the filename and page number: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/scripts/prepdocs.py#L204

   The content_file route looks up a blob and returns it: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/app/backend/app.py#L85

   So, you need to add some code to prepdocs.py to upload the whole PDF. Then you'd need to make sure the backend content_file route could retrieve that, and finally decide how you wanted to show it in the frontend code. Let us know if you go down that route and run into specific issues.

[vrajroutu]

you can actually make backend api requests to

https://app-backend-*******.azurewebsites.net/chat

BODY

{ "history": [ { "user": "How to create a project in jarvis?" } ], "approach": "rrr", "overrides": { "retrieval_mode": "hybrid", "semantic_ranker": true, "semantic_captions": true, "top": 5, "temperature": 0, "prompt_template": "example_template", "prompt_template_prefix": "example_prefix", "prompt_template_suffix": "example_suffix", "exclude_category": "example_category", "suggest_followup_questions": true } }

I have created a bot for this demo-app. I will submit a PR soon.

[vrajroutu]

@mhalomari I added a sample code for bot services integration #476 476

[mhalomari]

1. Nice idea, I've asked a member of the Teams team to brainstorm how to publish the app that way, they'll share their ideas on this thread.

Thank you very much. I can't wait to add this to my deployment. I was trying to integrate the app with a power virtual agent bot but it didn't work yet.

2. Yes, that should be possible, with some code modification. Here is the relevant code: Let us know if you go down that route and run into specific issues.

Thanks @pamelafox, I will try this now.

[mhalomari]

@mhalomari I added a sample code for bot services integration #455

Thank you very much @vrajroutu, I will work on this now and will let you know how it goes.

[pamelafox]

I've spoken with some Teams colleagues. They said that PR #455 would run in a bot emulator but not necessarily in Teams. They are now working on a prototype based on this repo that would use a combination of a Chat Tab and a Static Tab for citations. Stay tuned!

[mhalomari]

I've spoken with some Teams colleagues. They said that PR #455 would run in a bot emulator but not necessarily in Teams. They are now working on a prototype based on this repo that would use a combination of a Chat Tab and a Static Tab for citations. Stay tuned!

Thanks @pamelafox for working on this. Getting the app working in Teams would drive huge value and enables a better user experience. Your time is highly appreciated.

[garrytrinder]

Thank you for raising this issue @mhalomari, as previously mentioned by @pamelafox I have been building a sample app for Microsoft Teams that builds on top of this sample to provide an alternative to the web frontend.

You can find the sample app on my personal GitHub profile, https://github.com/garrytrinder/msteams-azure-search-openai-demo, I'd love to know your thoughts. Feel free to raise issues on the sample repo.

[mhalomari]

Thanks @garrytrinder, your time and efforts are highly appreciated. I am excited to try this now. Will let you know how it works with our current deployment.

[ganesh25]

1. Nice idea, I've asked a member of the Teams team to brainstorm how to publish the app that way, they'll share their ideas on this thread.

2. Yes, that should be possible, with some code modification. Here is the relevant code: In prepdocs.py, upload_blobs splits PDFs into pages and uploads each page: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/scripts/prepdocs.py#L31

   Also in prepdocs.py, create_sections stores the "sourcepage" field with the filename and page number: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/scripts/prepdocs.py#L204

   The content_file route looks up a blob and returns it: https://github.com/Azure-Samples/azure-search-openai-demo/blob/eb087331441dfb5fce8cf5b3ecc8a371707f676f/app/backend/app.py#L85

   So, you need to add some code to prepdocs.py to upload the whole PDF. Then you'd need to make sure the backend content_file route could retrieve that, and finally decide how you wanted to show it in the frontend code. Let us know if you go down that route and run into specific issues.

My org prefers entire pdfs, so I implemented this with your suggestions @pamelafox . The problem I have now is, the citations always start with the first page. I want it to point to the exact page of the document from where the answer was retrieved, when I click on the citation link. Not sure where to make changes for that, any help would be great.

[pamelafox]

The backend code serves up the whole PDF, and lets the browser do the rendering: https://github.com/Azure-Samples/azure-search-openai-demo/blob/60f768551d23eb0f40da7bfd85c23ea5765f74a0/app/backend/app.py#L62

I think that if you want to both show the whole PDF but link to the page, you would need to use some sort of PDFViewer JavaScript library and tell it to point at that particular page. For example, pdf.js: https://github.com/mozilla/pdf.js/

A simpler approach is to make sure the citation in the chat message has the correct page number, and write in the chat message "Scroll to page N".

[zhongshuai-cao]

The scrolling method is spot-on! By adopting this strategy, we can directly upload the file instead of iterating through its pages. This simplifies the reindexing and data refresh processes. I hope this can be proposed as a PR to revise the original method.

On a side note, I'm using react-doc-viewer to render multiple file types to accommodate various formats. I'll be investigating if the page scroll feature is accessible within the React framework.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed.