search

arc53 / DocsGPT

Chatbot for documentation, that allows you to chat with your data. Privately deployable, provides AI knowledge sharing and integrates knowledge into your AI workflow
https://app.docsgpt.cloud/
MIT License
14.91k stars 1.58k forks source link

Uploads fixes #186

Closed dartpain closed 1 year ago

dartpain commented 1 year ago
Thayalan21 commented 1 year ago

Hello which file type should be uploaded to the drop down. Because I've tried pdf and rst both aren't working

ajaythapliyal commented 1 year ago

Hello which file type should be uploaded to the drop down. Because I've tried pdf and rst both aren't working

Hey @Thayalan21 can you tell us more about the issue you are facing? What do you see?

Thayalan21 commented 1 year ago

@ajaythapliyal image

This is the error i am facing . In the front end i've tried to upload the pdf and also rst file format for training my documents.But it just shows this pop message and nothing happened

Thayalan21 commented 1 year ago

[image: image.png]

On Wed, 22 Mar 2023 at 14:30, Ajay Thapliyal @.***> wrote:

Hello which file type should be uploaded to the drop down. Because I've tried pdf and rst both aren't working

Hey @Thayalan21 https://github.com/Thayalan21 can you tell us more about the issue you are facing? What do you see?

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1478985873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTKCHIX3U4IF6OIWEJ3W5KMCFANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

Thayalan21 commented 1 year ago

This is the error i am facing.I tried to upload documents in the upload section but it pops file uploading but nothing happens afterwards

On Wed, 22 Mar 2023 at 15:06, thayalan ramasamy @.***> wrote:

[image: image.png]

On Wed, 22 Mar 2023 at 14:30, Ajay Thapliyal @.***> wrote:

Hello which file type should be uploaded to the drop down. Because I've tried pdf and rst both aren't working

Hey @Thayalan21 https://github.com/Thayalan21 can you tell us more about the issue you are facing? What do you see?

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1478985873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTKCHIX3U4IF6OIWEJ3W5KMCFANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

dartpain commented 1 year ago

Did you launch worker? Please describe how you launched docsGPT and what steps did you go through

Thayalan21 commented 1 year ago

I've gone through all the steps you guys mentioned from the flask app to the front end. I installed all the stuff at the requirement txt and I put my api key on it. But both in flask app and front end the upload function didn't work. It says connection between redis and the mongo server is not successful. However I've already installed all the dependencies in order this could work.

Thayalan21 commented 1 year ago

Also which file type should we actually upload a pdf or a rst?

Thayalan21 commented 1 year ago

image This is the error i am facing

Thayalan21 commented 1 year ago

image

Thayalan21 commented 1 year ago

I did all the steps you guys provided starting from the flask and also the frontend part let me share you the problems i am facing.The first picture is on the frontend part of uploading section.And the second picture is error in uploading part in flask app.

[image: image.png][image: image.png]

On Wed, 22 Mar 2023 at 17:48, Alex @.***> wrote:

Did you launch worker? Please describe how you launched docsGPT and what steps did you go through

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1479230896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTKC6IXLSL2VMJ6ICODW5LDF3ANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

Thayalan21 commented 1 year ago

These are the steps i did

1.Clone the file 2.In application folder i installed requirement.txt 3.I create env file with my api key 4.And i ran app.py

For the front end.

1.In the front-end folder I ran npm install

  1. VITE_API_HOST=http://localhost:5001 i changed this in env development 3.And then i ran npm run dev But it still not working

On Thu, 23 Mar 2023 at 15:13, thayalan ramasamy @.***> wrote:

I did all the steps you guys provided starting from the flask and also the frontend part let me share you the problems i am facing.The first picture is on the frontend part of uploading section.And the second picture is error in uploading part in flask app.

[image: image.png][image: image.png]

On Wed, 22 Mar 2023 at 17:48, Alex @.***> wrote:

Did you launch worker? Please describe how you launched docsGPT and what steps did you go through

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1479230896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTKC6IXLSL2VMJ6ICODW5LDF3ANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

Thayalan21 commented 1 year ago

Also i find error in html parser in application folder

[image: image.png]

So i change the code accordingly

[image: image.png]

On Thu, 23 Mar 2023 at 15:34, thayalan ramasamy @.***> wrote:

These are the steps i did

1.Clone the file 2.In application folder i installed requirement.txt 3.I create env file with my api key 4.And i ran app.py

For the front end.

1.In the front-end folder I ran npm install

  1. VITE_API_HOST=http://localhost:5001 i changed this in env development 3.And then i ran npm run dev But it still not working

On Thu, 23 Mar 2023 at 15:13, thayalan ramasamy @.***> wrote:

I did all the steps you guys provided starting from the flask and also the frontend part let me share you the problems i am facing.The first picture is on the frontend part of uploading section.And the second picture is error in uploading part in flask app.

[image: image.png][image: image.png]

On Wed, 22 Mar 2023 at 17:48, Alex @.***> wrote:

Did you launch worker? Please describe how you launched docsGPT and what steps did you go through

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1479230896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTKC6IXLSL2VMJ6ICODW5LDF3ANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

dartpain commented 1 year ago

We are updating our wiki, but to have upload functionality you need to launch with docker-compose up

Thayalan21 commented 1 year ago

This error shows up after i ran docker-compose up

/usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, ERROR: Version in "./docker-compose.yaml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1. For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/

dartpain commented 1 year ago

Try docker --version

and docker-compose --version.

and submit your response

I think you may need to update them. Also changing version in your docker-compose.yaml file might help on the top of the file substitute version: "3.9" to version: "3.3"

Thayalan21 commented 1 year ago

/usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, docker-compose version 1.25.0, build unknown/usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, docker-compose version 1.25.0, build unknown

This is what i get after these steps

Thayalan21 commented 1 year ago

ima_574634b

I tried to solve the problem with docker but i get this error in console when i tried to upload a pdf file.So still the upload function is not working. Is it just me or everybody have this issue?

Thayalan21 commented 1 year ago

image I am still receiving the errors. I managed to run this on docker but the the upload issue is not solving

Thayalan21 commented 1 year ago

Hello people. I just wanna ask why this error pop up when I tried the uploading function. I've already launched this code in docker but still facing issues with the upload function

image

Thayalan21 commented 1 year ago

@dartpain

Thayalan21 commented 1 year ago

I've posted some issues in git hub.Do you mind checking sir?

On Thu, 23 Mar 2023 at 23:44, Alex @.***> wrote:

Try docker --version

and docker-compose --version.

and submit your response

I think you may need to update them. Also changing version in your docker-compose.yaml file might help on the top of the file substitute version: "3.9" to version: "3.3"

— Reply to this email directly, view it on GitHub https://github.com/arc53/DocsGPT/issues/186#issuecomment-1481428828, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY5IWTK35UQFRGDHNAUOSUDW5RVWXANCNFSM6AAAAAAWBFZ5GI . You are receiving this because you were mentioned.Message ID: @.***>

blindcoding9 commented 1 year ago

I`m facing the same error, running in a cloud server.

http://somemachine.devpublic.gru.oraclevcn.com:5173/

image

Thayalan21 commented 1 year ago

Is it because i am running these code on ssh proxmox server???

Thayalan21 commented 1 year ago

Is it because i am running these code on ssh proxmox server???

dartpain commented 1 year ago

Hmmm, please can you show me docker log for your worker container and backend?

Thayalan21 commented 1 year ago

image image

Thayalan21 commented 1 year ago

image

dartpain commented 1 year ago

Ok looks like its an issue with Mongo container connection

dartpain commented 1 year ago

Can you try connecting to it locally or checking if its working well?

Thayalan21 commented 1 year ago

U mean without ssh?

Thayalan21 commented 1 year ago

Nope no difference

dartpain commented 1 year ago

https://www.mongodb.com/products/compass This tool for example

Thayalan21 commented 1 year ago

image I am still receiving this error

Thayalan21 commented 1 year ago

worker_1 | [2023-03-28 03:21:14,812: INFO/MainProcess] Task app.ingest[b4ffd974-eb01-4dce-8201-998af1483073] received worker_1 | [2023-03-28 03:21:14,834: WARNING/ForkPoolWorker-4] Grouping small documents worker_1 | [2023-03-28 03:21:14,841: WARNING/ForkPoolWorker-4] Separating large documents worker_1 | [2023-03-28 03:21:14,845: WARNING/ForkPoolWorker-4] Grouping failed, try running without token_check Embedding 🦖: | | Time Left: ? WARNING/ForkPoolWorker-4] Embedding 🦖: | | Time Left: ? WARNING/ForkPoolWorker-4] backend_1 | [2023-03-28 03:21:34 +0000] [6] [ERROR] Error handling request /api/combine backend_1 | Traceback (most recent call last): backend_1 | File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 136, in handle backend_1 | self.handle_request(listener, req, client, addr) backend_1 | File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 179, in handle_request backend_1 | respiter = self.wsgi(environ, resp.start_response) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2551, in call backend_1 | return self.wsgi_app(environ, start_response) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2531, in wsgi_app backend_1 | response = self.handle_exception(e) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app backend_1 | response = self.full_dispatch_request() backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request backend_1 | rv = self.handle_user_exception(e) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request backend_1 | rv = self.dispatch_request() backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request backend_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) backend_1 | File "/app/app.py", line 301, in combined_json backend_1 | for index in vectors_collection.find({'user': user}): backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/cursor.py", line 1248, in next backend_1 | if len(self.data) or self._refresh(): backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/cursor.py", line 1139, in _refresh backend_1 | self.session = self.collection.database.client._ensure_session() backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1740, in _ensure_session backend_1 | return self.start_session(True, causal_consistency=False) backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1685, in start_session backend_1 | self._topology._check_implicit_session_support() backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 538, in _check_implicit_session_support backend_1 | self._check_session_support() backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 554, in _check_session_support backend_1 | self._select_servers_loop( backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 238, in _select_servers_loop backend_1 | raise ServerSelectionTimeoutError( backend_1 | pymongo.errors.ServerSelectionTimeoutError: mongo:27018: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64225cb0716e4782aad2cafe, topology_type: Unknown, servers: [<ServerDescription ('mongo', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('mongo:27018: [Errno -3] Temporary failure in name resolution')>]> backend_1 | [2023-03-28 03:21:45 +0000] [7] [ERROR] Error handling request /api/upload_index backend_1 | Traceback (most recent call last): backend_1 | File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 136, in handle backend_1 | self.handle_request(listener, req, client, addr) backend_1 | File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 179, in handle_request backend_1 | respiter = self.wsgi(environ, resp.start_response) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2551, in call__ backend_1 | return self.wsgi_app(environ, start_response) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2531, in wsgi_app backend_1 | response = self.handle_exception(e) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app backend_1 | response = self.full_dispatch_request() backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request backend_1 | rv = self.handle_user_exception(e) backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request backend_1 | rv = self.dispatch_request() backend_1 | File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request backend_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) backend_1 | File "/app/app.py", line 395, in upload_index_files backend_1 | vectors_collection.insert_one({ backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 628, in insert_one backend_1 | self._insert_one( backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 569, in _insert_one backend_1 | self.database.client._retryable_write(acknowledged, _insert_command, session) backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1475, in _retryable_write backend_1 | with self._tmp_session(session) as s: backend_1 | File "/usr/local/lib/python3.10/contextlib.py", line 135, in enter__ backend_1 | return next(self.gen) backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1757, in _tmp_session backend_1 | s = self._ensure_session(session) backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1740, in _ensure_session backend_1 | return self.start_session(True, causal_consistency=False) backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1685, in start_session backend_1 | self._topology._check_implicit_session_support() backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 538, in _check_implicit_session_support backend_1 | self._check_session_support() backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 554, in _check_session_support backend_1 | self._select_servers_loop( backend_1 | File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 238, in _select_servers_loop backend_1 | raise ServerSelectionTimeoutError( backend_1 | pymongo.errors.ServerSelectionTimeoutError: mongo:27018: [Errno -3] Temporary failure in name resolution, Timeout: 30s, Topology Description: <TopologyDescription id: 64225cb0f674f18d0522572d, topology_type: Unknown, servers: [<ServerDescription ('mongo', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('mongo:27018: [Errno -3] Temporary failure in name resolution')>]> worker_1 | [2023-03-28 03:21:45,957: INFO/ForkPoolWorker-4] Task app.ingest[b4ffd974-eb01-4dce-8201-998af1483073] succeeded in 31.138961516000563s: {'directory': 'temp', 'formats': ['.rst', '.md', '.pdf', '.txt'], 'name_job': 'sample-essay-1_1.rst', 'filename': 'sample-essay-1_1.rst', 'user': 'local', 'limited': False}

Thayalan21 commented 1 year ago

This happens in the log

dartpain commented 1 year ago

Please check if there are any logs for mongo_1 container. Send me docker-compose.yaml file (remove the api keys from there too please)

Thayalan21 commented 1 year ago

This is the yaml file

version: "3.3"

services: frontend: build: ./frontend environment:

volumes: mongodb_data_container:

dartpain commented 1 year ago

Do you lanuch it with docker-compose up? Do you have any container logs for mongo?

Thayalan21 commented 1 year ago

Yes i did launch it with docker-compose up. But the mongodb seem to not work. I've also tried to use external momgodb but still it does not work

dartpain commented 1 year ago

Interesting, seems like it can't start a container for mongodb then. Try docker --version

Thayalan21 commented 1 year ago

yeah docker is installed

Thayalan21 commented 1 year ago

Sir i managed to do it Thank you. But is the token issue is still there right? Because i use document it print nothing.I attach the file here

Thayalan21 commented 1 year ago

vertopal.com_Never-Split-the-Difference (1).zip

Thayalan21 commented 1 year ago

Issue resolved thanks

Thayalan21 commented 1 year ago

Is it possible if we could upload multiple files?

dartpain commented 1 year ago

As a zip you can, im closing this issue