✨ Assistant in desktop client & mobile apps

jancborchardt commented 7 months ago

It would be nice to have the functionality of the Assistant modal also available on the desktop and on the go in the mobile apps.

Mobile apps: Could be a navigation entry "Assistant"
Desktop client: Possibly an icon in the header bar of the desktop client tray menu

jancborchardt commented 7 months ago

This needs discussion on how to technically integrate, e.g. which APIs are there and/or which we need. cc @julien-nc @tobiasKaminsky :)

tobiasKaminsky commented 7 months ago

Quick idea

results are notifications
with new screen filter all results by app "assistant", so that only relevant notifications are shown

tobiasKaminsky commented 7 months ago

Mobile

add as first item in "talk, notes, more"
for branded clients
- Android: in drawer, below "notifications"
- iOS: as first item in "more"

Desktop:

move "talk" to three dot menu
add assistant
have a prompt, see result, same like on Mobile

For all:

have first type selected
result is in notification id
should be extracted and shown directly
show history
- see integration issue
- retrieve task list for specific task
- show list, last 5, then "load more"
- possible to delete

julien-nc commented 7 months ago

Endpoints available

get available task types: GET /ocs/v2.php/textprocessing/tasktypes This returns the types that have at least one provider.
schedule a task: POST /ocs/v2.php/textprocessing/schedule with params:
- string input
- string type, the task type id
- string appId, use "assistant" there
- string identifier = '', something to identify a task later, leave it empty if not needed (used by text to know in which document a task was created)
get a task: GET /ocs/v2.php/textprocessing/task/{id} This returns a serialized task with the output and the status.
delete a task: DELETE /ocs/v2.php/textprocessing/task/{id}
list tasks by app: GET /ocs/v2.php/textprocessing/tasks/app/{appId} You only want the assistant tasks.

Task statuses

STATUS_FAILED = 4
STATUS_SUCCESSFUL = 3
STATUS_RUNNING = 2
STATUS_SCHEDULED = 1
STATUS_UNKNOWN = 0

Capability

I will add a capability to tell the clients if the assistant is enabled for the current user. It will probably be assistant.enabled: boolean.

Handling result notifications

Result notifications have one action button pointing to /apps/assistant/t/{taskId}. This is a page displaying the task result. This page can only be accessed by authenticated users. Instead of allowing client users to browse this page, you could extract the task ID from the button target URL and get the task with the /ocs/v2.php/textprocessing/task/{id} OCS endpoint to show the result directly in the client.

@tobiasKaminsky Anything else you need to know?

tobiasKaminsky commented 7 months ago

Looks great so far :+1:

alperozturk96 commented 6 months ago

Design suggestion @jancborchardt

jancborchardt commented 6 months ago

@alperozturk96 the design should best follow the web interface (of course with Material Design and Android components), since otherwise we have different designs for the same thing. :)

Or are you saying we should also improve this in the web interface? cc @julien-nc @nimishavijay

alperozturk96 commented 6 months ago

@alperozturk96 the design should best follow the web interface (of course with Material Design and Android components), since otherwise we have different designs for the same thing. :)

Or are you saying we should also improve this in the web interface? cc @julien-nc @nimishavijay

We should have same design language but we should't do exactly same thing on mobile. Mobile devices have small screen UX/UI needs to change according to that. BottomSheet, AlertDialog etc all these things are helping user to tackle limitation of screen size.

For example are web pages have bottom sheet? No. In web we are showing buttons next to each other because we have huge wide screen. So that you can display Free prompt, Summarize etc next to each other but on mobile devices its impossible, we must use Dropdown menu.

On the other hand Input section should't visible to the user all the time because years of user experience will resist this. Almost all apps have same pattern (Apple Human Interface Guideline, Android Design) show empty list then only have one add button then filter list etc.

In short we must not fight against native iOS and Android nature.

mpivchev commented 5 months ago

For iOS: "add as first item in "talk, notes, more"" I don't think this is a good place for this, it's way too hidden. Should be either as a separate icon on the top navigation bar, or at least in the dropdown menu. @marinofaggiana what do you think?

marinofaggiana commented 5 months ago

mmm I honestly don't know how many will use it ... for me better in "add as first item in "talk, notes, more""

camilasan commented 4 months ago

have a prompt, see result, same like on Mobile

I don't understand what you mean @tobiasKaminsky.

@jancborchardt wdyt?

If only Talk is available then it will be displayed as it is now.

@julien-nc is it possible to have a direct link to open the assistant in the browser?

camilasan commented 4 months ago

The desktop PR: https://github.com/nextcloud/desktop/pull/6637

julien-nc commented 4 months ago

is it possible to have a direct link to open the assistant in the browser?

@camilasan There is a page to see the results of a task: apps/assistant/task/view/TASK_ID but there is nothing to just open it. It's a nice idea, we'll add that soon.

mpivchev commented 4 months ago

iOS PR: https://github.com/nextcloud/ios/pull/2881

camilasan commented 4 months ago

@camilasan There is a page to see the results of a task: apps/assistant/task/view/TASK_ID but there is nothing to just open it. It's a nice idea, we'll add that soon.

how soon?

julien-nc commented 4 months ago

@camilasan It's there but not released https://github.com/nextcloud/assistant/pull/72 If you want to know if this page is available, you can check the 'assistant.version' capability (exposed by the assistant) is >= 1.0.9.

nextcloud / files-clients