Implement Upload Files From The Chat

[GiovanniSmokes]

Add the option to upload image using natural language. The user should just write "upload file" and the file upload option would appear to him

[vporton]

I read at https://ts2.space/en/how-to-use-images-with-gpt-4-api/ that to use images (I assume it pertains to all kinds of files), need to add the file in base64 encoding and instruct the GPT that it is in base64 format.

So several questions arise:

• Insert files in the beginning or the end of the prompt?

• What should be the prompt addition? Like <FILENAME> is base64 file XXX<ENTER><ENTER>? base64 or Base64?

[apssouza22]

@vporton No. We don't want to send the image to openai. we just to have the option to upload image using natural language. The user should just write "upload file" and the file upload option would appear to him. You saw this input option before. To see it again we can just uncomment this

It is basically a Frontend + prompt engineering

[vporton]

I don't understand how this works (I am not at all an expert in prompt engineering):

For example, first inputs:

Upload the file, that I will give, to Google drive.

Then inputs:

Here is the file. (and attaches it)

How (and in which format, JSON?) the information is transferred from the first prompt handling to the second prompt handling?

I definitely can understand it by reading the source, but please give a clue where to start (about transferring information between several prompts).

[vporton]

@apssouza22 Could you explain this?

[apssouza22]

I don't think I will be able to explain it. You will need to learn how the core part of app works before I gave you more information than I have already provided

[vporton]

@apssouza22 @GiovanniSmokes Should we have the option to upload multiple files in a single prompt?

[apssouza22]

We should start with single file to make things easier, but multiple would not be a problem

[vporton]

Should we support only multipart/form-data or also other ways of uploading file, such as base64 in JSON?

[vporton]

@apssouza22 Consider this workflow:

We will work with the following file: [upload]

Upload the file that I gave you to my Google drive.

That is the file may be reasonably held in JavaScript memory for an indefinite amount of time until the assistant receives a command to do something with the file.

Moreover, the next command may be:

Upload the same file to my Web hosting, folder /uploads.

If so, we should not discard the file after we did something with it.

So, we have a problem: uploading many big files may overflow JS memory.

What solution of the problem will we choose?

1. Discard a file after something is done with it or another file is uploaded (bad from user's viewpoint).
2. Keep several (e.g. 5) files in memory (why 5? the number would be too arbitrary).
3. Show to the user a "panel" listing current files in consideration, with the option to delete a file by clicking cross sign (seems perfect but is hard to implement).

The above is about the end result, when we complete adding features. But what quick and dirty (but not with a too big technical debt) solution will we choose for now?

[apssouza22]

Done as part of this MR