Closed Deathproof76 closed 6 months ago
B-urb
commented
6 months ago Should not be the latest commit, the updating didn't change. Unfortunately paperless error logging is not really helpful at all. There should be a file in the paperless volume somewhere that contains the debug logs. Maybe those contain more info. The fields in the prompt should match the custom fields, when modifying the base prompt.
Deathproof76
commented
6 months ago I tried setting paperless to debug via env. But the logs don't show anything that explains the bad request. I removed the custom baseprompt from doclytics. I also completely removed all custom fields and readded the custom field that align with the baseprompt. Also tried removing those so that only "tagged" was left and retried multiple times each iteration. The error stays the same:
paperless-webserver-1 | [2024-05-13 23:20:00,020] [INFO] [celery.app.trace] Task paperless_mail.tasks.process_mail_accounts[7f59495d-3deb-4bc8-86b6-cd58a1f2b6f5] succeeded in 0.01535832998342812s: 'No new documents were added.'
paperless-webserver-1 | Bad Request: /api/documents/259/
paperless-webserver-1 | [2024-05-13 23:23:53,163] [WARNING] [django.request] Bad Request: /api/documents/259/
paperless-webserver-1 | Bad Request: /api/documents/259/
paperless-webserver-1 | [2024-05-13 23:24:12,291] [WARNING] [django.request] Bad Request: /api/documents/259/
paperless-webserver-1 | [2024-05-13 23:25:00,004] [DEBUG] [celery.beat] beat: Synchronizing schedule...doclytics | May 13 21:23:18.446 INFO Application started !
doclytics | May 13 21:23:18.471 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 13 21:23:18.502 INFO Fields: [Field { id: 21, name: "category", data_type: "string" }, Field { id: 20, name: "date_received", data_type: "string" }, Field { id: 19, name: "urgency", data_type: "string" }, Field { id: 18, name: "recipient", data_type: "string" }, Field { id: 17, name: "sender", data_type: "string" }, Field { id: 16, name: "topic", data_type: "string" }, Field { id: 15, name: "title", data_type: "string" }, Field { id: 14, name: "tagged", data_type: "boolean" }]
doclytics | May 13 21:23:18.502 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 13 21:23:18.674 INFO Successfully retrieved 25 Documents
doclytics | May 13 21:23:18.674 INFO Generate Response with LLM model
doclytics | May 13 21:23:53.108 INFO Updating document with ID: 259
doclytics | May 13 21:23:53.164 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/259/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/259/", query: None, fragment: None } }
doclytics exited with code 0version: "3"
services:
doclytics:
image: bjoern5urban/doclytics:development
container_name: doclytics
environment:
PAPERLESS_BASE_URL: http://192.168.0.208:17959
PAPERLESS_TOKEN: ebb5500af85834a1f7xxxxxxxxxxx2bc0134f7aa6bb1a
LOG_LEVEL: INFO
OLLAMA_HOST: 192.168.0.208
OLLAMA_PORT: "11434"
OLLAMA_MODEL: mixtral-paperless:latest
volumes:
- /mnt/Dockerspace/doclytics:/app/data
restart: unless-stoppedThe only thing I can see in the django admin panel in paperless is the last time it worked, but no errors, for example:
<html>
<body>
<!--StartFragment-->
May 13, 2024, 4:11 p.m. | documents \| custom field instance - Kategorie : Rechnung | create | 5 changes: id, field, created, document, value_text
-- | -- | -- | --
<!--EndFragment-->
</body>
</html>[May 13, 2024, 4:11 p.m.](https://archive.xxxxxxxxxxxxx/admin/auditlog/logentry/3180/change/?_changelist_filters=action__exact%3D0%26resource_type%3D36) [documents | custom field instance - Kategorie : Rechnung](https://archive.xxxxxxxxxxxxx/admin/documents/customfieldinstance/1474/change/) create 5 changes: id, field, created, document, value_textI don't have the faintest idea how to debug further. The log files within paperless were equally useless. I'm on Paperless-ngx 2.8.4 btw. also tried downgrading to 2.7
B-urb
commented
6 months ago I will try later to recreate this error with paperless 2.8. I published another released candidate with logging improvements. You could try pulling that and run with LOG_LEVEL=DEBUG. This should log the payload for the update now, so maybe we can see the issue then.
Deathproof76
commented
6 months ago I upgraded my paperless postgresql from 13 to 16, dumped and restored. Then saw your new comment, pulled the newest development with LOG_LEVEL: DEBUG in the env. And well it works now😅 :
doclytics | May 14 10:33:05.365 INFO Application started !
doclytics | May 14 10:33:05.391 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 10:33:05.391 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:33:05.391 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:33:05.391 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:33:05.403 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:33:05.403 INFO Fields: [Field { id: 21, name: "category", data_type: "string" }, Field { id: 20, name: "date_received", data_type: "string" }, Field { id: 19, name: "urgency", data_type: "string" }, Field { id: 18, name: "recipient", data_type: "string" }, Field { id: 17, name: "sender", data_type: "string" }, Field { id: 16, name: "topic", data_type: "string" }, Field { id: 15, name: "title", data_type: "string" }, Field { id: 14, name: "tagged", data_type: "boolean" }]
doclytics | May 14 10:33:05.403 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 10:33:05.403 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:33:05.669 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:33:05.670 INFO Successfully retrieved 25 Documents
doclytics | May 14 10:33:05.670 INFO Generate Response with LLM model
doclytics | May 14 10:33:05.670 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 10:33:05.670 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 10:33:05.670 DEBG connected to 192.168.0.208:11434
doclytics | May 14 10:33:30.324 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:33:30.324 INFO Updating document with ID: 220
doclytics | May 14 10:33:30.324 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:33:30.324 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:33:30.324 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:33:30.435 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:33:30.435 INFO Document with ID: 220 successfully updated
doclytics | May 14 10:33:30.435 INFO Generate Response with LLM model
doclytics | May 14 10:33:30.435 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:33:57.508 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:33:57.509 INFO Updating document with ID: 213
doclytics | May 14 10:33:57.509 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:33:57.509 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:33:57.509 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:33:57.611 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:33:57.611 INFO Document with ID: 213 successfully updated
doclytics | May 14 10:33:57.611 INFO Generate Response with LLM model
doclytics | May 14 10:33:57.611 DEBG reuse idle connection for ("http", 192.168.0.208:11434)could be that my database had a problem and I fixed it by upgrading and restoring it.
btw: for everyone with an old db image wanting to upgrade postgres these links helped https://github.com/paperless-ngx/paperless-ngx/discussions/3939#discussioncomment-6696701 https://www.reddit.com/r/Paperlessngx/comments/16olzcm/comment/kpm8xyd especially regarding the "User "paperless" does not have a valid SCRAM secret." error.
update1: Tried with my custom prompt and custom fields again, error:
doclytics | May 14 10:50:00.068 INFO Application started !
doclytics | May 14 10:50:00.095 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 10:50:00.095 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:50:00.095 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:50:00.096 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:50:00.109 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:50:00.109 INFO Fields: [Field { id: 28, name: "Kategorie", data_type: "string" }, Field { id: 27, name: "Datum_Empfang", data_type: "string" }, Field { id: 26, name: "Absender", data_type: "string" }, Field { id: 25, name: "Zusammenfassung", data_type: "string" }, Field { id: 24, name: "Thema", data_type: "string" }, Field { id: 23, name: "Titel", data_type: "string" }, Field { id: 22, name: "tagged", data_type: "boolean" }]
doclytics | May 14 10:50:00.109 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 10:50:00.109 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:50:00.371 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:50:00.371 INFO Successfully retrieved 25 Documents
doclytics | May 14 10:50:00.371 INFO Generate Response with LLM model
doclytics | May 14 10:50:00.371 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 10:50:00.371 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 10:50:00.371 DEBG connected to 192.168.0.208:11434
doclytics | May 14 10:50:59.188 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:50:59.188 INFO Updating document with ID: 245
doclytics | May 14 10:50:59.188 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:50:59.189 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:50:59.189 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:50:59.216 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:50:59.216 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/245/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/245/", query: None, fragment: None } }
doclytics exited with code 0update2: deleted custom prompt and custom fields, readded "base" field from example prompt: works again:
doclytics | May 14 10:56:21.982 INFO Application started !
doclytics | May 14 10:56:22.008 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 10:56:22.008 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:56:22.008 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:56:22.008 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:56:22.020 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:56:22.020 INFO Fields: [Field { id: 36, name: "category", data_type: "string" }, Field { id: 35, name: "date_received", data_type: "string" }, Field { id: 34, name: "urgency", data_type: "string" }, Field { id: 33, name: "recipient", data_type: "string" }, Field { id: 32, name: "sender", data_type: "string" }, Field { id: 31, name: "topic", data_type: "string" }, Field { id: 30, name: "title", data_type: "string" }, Field { id: 29, name: "tagged", data_type: "boolean" }]
doclytics | May 14 10:56:22.020 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 10:56:22.020 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:56:22.271 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:56:22.271 INFO Successfully retrieved 25 Documents
doclytics | May 14 10:56:22.272 INFO Generate Response with LLM model
doclytics | May 14 10:56:22.272 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 10:56:22.272 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 10:56:22.272 DEBG connected to 192.168.0.208:11434
doclytics | May 14 10:56:49.074 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:56:49.074 INFO Updating document with ID: 245
doclytics | May 14 10:56:49.074 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:56:49.074 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:56:49.074 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:56:49.185 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:56:49.185 INFO Document with ID: 245 successfully updated
doclytics | May 14 10:56:49.185 INFO Generate Response with LLM model
doclytics | May 14 10:56:49.185 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:13.770 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:13.770 INFO Updating document with ID: 211
doclytics | May 14 10:57:13.770 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:57:13.770 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:57:13.770 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:57:13.861 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:57:13.861 INFO Document with ID: 211 successfully updated
doclytics | May 14 10:57:13.861 INFO Generate Response with LLM model
doclytics | May 14 10:57:13.861 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:39.406 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:39.406 INFO Updating document with ID: 212
doclytics | May 14 10:57:39.406 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:57:39.406 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:57:39.406 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:57:39.512 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:57:39.512 INFO Document with ID: 212 successfully updated
doclytics | May 14 10:57:39.512 INFO Generate Response with LLM model
doclytics | May 14 10:57:39.512 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:57.150 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 10:57:57.150 INFO Updating document with ID: 205
doclytics | May 14 10:57:57.150 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 10:57:57.150 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 10:57:57.150 DEBG connected to 192.168.0.208:17959
doclytics | May 14 10:57:57.232 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 10:57:57.232 INFO Document with ID: 205 successfully updated
doclytics | May 14 10:57:57.232 INFO Generate Response with LLM model
doclytics | May 14 10:57:57.232 DEBG reuse idle connection for ("http", 192.168.0.208:11434)so it's most likely a problem with my custom prompt :
Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück.
Die Felder, die ich benötige, sind:
Titel,Zusammenfassung,Absender,Datum_Empfang,Kategorie.
Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als
JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort.
Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden.and custom fields:
INFO Fields: [Field { id: 28, name: "Kategorie", data_type: "string" }, Field { id: 27, name: "Datum_Empfang", data_type: "string" }, Field { id: 26, name: "Absender", data_type: "string" }, Field { id: 25, name: "Zusammenfassung", data_type: "string" }, Field { id: 24, name: "Thema", data_type: "string" }, Field { id: 23, name: "Titel", data_type: "string" }, Field { id: 22, name: "tagged", data_type: "boolean" }]
Deathproof76
commented
6 months ago tried the prompt and custom fields with lower case:
BASE_PROMPT: "Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder titel,zusammenfassung,kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden."
LOG_LEVEL: DEBUGsame error:
doclytics | May 14 11:09:27.179 INFO Application started !
doclytics | May 14 11:09:27.203 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 11:09:27.203 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 11:09:27.203 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 11:09:27.203 DEBG connected to 192.168.0.208:17959
doclytics | May 14 11:09:27.215 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:09:27.215 INFO Fields: [Field { id: 41, name: "kategorie", data_type: "string" }, Field { id: 40, name: "titel", data_type: "string" }, Field { id: 39, name: "zusammenfassung", data_type: "string" }, Field { id: 37, name: "tagged", data_type: "boolean" }]
doclytics | May 14 11:09:27.215 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 11:09:27.215 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:09:27.485 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:09:27.485 INFO Successfully retrieved 25 Documents
doclytics | May 14 11:09:27.485 INFO Generate Response with LLM model
doclytics | May 14 11:09:27.485 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 11:09:27.485 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 11:09:27.485 DEBG connected to 192.168.0.208:11434
doclytics | May 14 11:10:07.120 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 11:10:07.120 INFO Updating document with ID: 196
doclytics | May 14 11:10:07.120 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 11:10:07.120 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 11:10:07.121 DEBG connected to 192.168.0.208:17959
doclytics | May 14 11:10:07.147 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:10:07.147 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/196/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/196/", query: None, fragment: None } }
doclytics exited with code 0update:
it works with a slightly modified baseprompt:
BASE_PROMPT: Please extract metadata from the provided document and return it in JSON format. The fields I need are title,topic,category. Analyze the document to find the values for these fields and format the response as a JSON object. Use the most likely answer for each field. The response should contain only JSON data where the key and values are all in simple string format(no nested object) for direct parsing by another program. So now additional text or explanation, no introtext, the answer should start and end with curly brackets delimiting the json objectand
Deathproof76
commented
6 months ago tried keeping the baseprompt english while requesting a german answer:
BASE_PROMPT: Please extract metadata from the provided document and return it in JSON format. The fields I need are titel,absender,zusammenfassung. Analyze the document to find the values for these fields and format the response as a JSON object. Use the most likely answer for each field. The answers need to be in german language. The response should contain only JSON data where the key and values are all in simple string format(no nested object) for direct parsing by another program. So now additional text or explanation, no introtext, the answer should start and end with curly brackets delimiting the json object
again error:
doclytics | May 14 11:24:16.401 INFO Application started !
doclytics | May 14 11:24:16.425 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 11:24:16.425 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 11:24:16.426 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 11:24:16.426 DEBG connected to 192.168.0.208:17959
doclytics | May 14 11:24:16.437 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:24:16.437 INFO Fields: [Field { id: 49, name: "absender", data_type: "string" }, Field { id: 48, name: "titel", data_type: "string" }, Field { id: 47, name: "zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 14 11:24:16.437 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 11:24:16.438 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:24:16.684 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:24:16.685 INFO Successfully retrieved 25 Documents
doclytics | May 14 11:24:16.685 INFO Generate Response with LLM model
doclytics | May 14 11:24:16.685 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 11:24:16.685 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 11:24:16.685 DEBG connected to 192.168.0.208:11434
doclytics | May 14 11:24:59.584 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 11:24:59.584 INFO Updating document with ID: 188
doclytics | May 14 11:24:59.585 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 11:24:59.585 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 11:24:59.585 DEBG connected to 192.168.0.208:17959
doclytics | May 14 11:24:59.602 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 11:24:59.602 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/188/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/188/", query: None, fragment: None } }
doclytics exited with code 0so yeah, I don't know. So far it looks like there might be a problem with german? Or maybe custom fields that diverge from the default base_prompt? But it seemed to work at one time... Maybe a consistency problem with the json output of the model? (using mixtral-instruct q4_k_M).
The thing that's really confusing me is that it should have already worked here https://github.com/B-urb/doclytics/issues/44#issuecomment-2108857266 when everything was at default
B-urb
commented
6 months ago Okay, just now i am wondering why you don't see the payload debug log at: https://github.com/B-urb/doclytics/blob/1470dcace5c3b5f15c3ec93bad8b939e4cc7734e/src/paperless.rs#L151
It should log the body that is used for updating the fields. This way, we could maybe narrow down where the error is. German shouldn't be a problem. The only problematic thing could be could be creating title as a custom field in english, because that is a default metadata field that is used by paperless and title does not get added to the custom fields part of the update body, but updates the metadata field of paperless not any custom fields called title.
Also weird output with bad json in the llm output can cause a problem. I just pushed a new commit to dev where with debug logging, the llm response should be logged as well.
Deathproof76
commented
6 months ago short update: pulled bjoern5urban/doclytics:v1.1.4-rc.4 , tried german again, still no llm response:
doclytics | May 14 12:54:27.802 INFO Application started !
doclytics | May 14 12:54:27.829 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 12:54:27.829 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 12:54:27.829 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 12:54:27.829 DEBG connected to 192.168.0.208:17959
doclytics | May 14 12:54:27.840 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 12:54:27.840 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 14 12:54:27.840 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 12:54:27.840 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 12:54:28.090 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 12:54:28.090 INFO Successfully retrieved 25 Documents
doclytics | May 14 12:54:28.090 INFO Generate Response with LLM model
doclytics | May 14 12:54:28.090 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 12:54:28.090 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 12:54:28.090 DEBG connected to 192.168.0.208:11434
doclytics | May 14 12:54:55.051 INFO Updating document with ID: 197
doclytics | May 14 12:54:55.051 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 12:54:55.051 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 12:54:55.051 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 12:54:55.051 DEBG connected to 192.168.0.208:17959
doclytics | May 14 12:54:55.077 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 12:54:55.077 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/197/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/197/", query: None, fragment: None } }
doclytics exited with code 0will try different model, could also be that the response is only logged when it succeeds, will try later when I have the time.
ollama log!:
ollama | [GIN] 2024/05/14 - 13:05:47 | 200 | 3.839616075s | 172.16.47.1 | POST "/api/generate"
ollama | time=2024-05-14T13:05:49.916Z level=WARN source=types.go:382 msg="invalid option provided" option=""
ollama | [GIN] 2024/05/14 - 13:05:53 | 200 | 3.647991603s | 172.16.47.1 | POST "/api/generate"
ollama | time=2024-05-14T13:05:57.293Z level=WARN source=types.go:382 msg="invalid option provided" option=""
ollama | [GIN] 2024/05/14 - 13:06:00 | 200 | 3.139295839s | 172.16.47.1 | POST "/api/generate"
ollama | time=2024-05-14T13:06:58.569Z level=WARN source=types.go:382 msg="invalid option provided" option=""
ollama | [GIN] 2024/05/14 - 13:07:01 | 200 | 3.048830111s | 172.16.47.1 | POST "/api/generate"
ollama | time=2024-05-14T13:07:02.402Z level=WARN source=types.go:382 msg="invalid option provided" option=""
ollama | [GIN] 2024/05/14 - 13:07:06 | 200 | 4.102707797s | 172.16.47.1 | POST "/api/generate"
ollama | time=2024-05-14T13:07:07.249Z level=WARN source=types.go:382 msg="invalid option provided" option=""
ollama | [GIN] 2024/05/14 - 13:07:10 | 200 | 3.495262471s | 172.16.47.1 | POST "/api/generate"I made a mistake. Seems like when the json is empty the log is skipped. I fixed this in the latest development. Maybe this clarifies it, though the warning in ollama is a bit weird.
Deathproof76
commented
6 months ago Pulled the latest Doclytics. The Ollama error coincided with a model change to vicuna:13b-v1.5-16k-q4_K_M. So, just a fluke most likely. Once I switched back to mixtral-8x7b-instruct:latest it worked again from ollamas side.
ollama | INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="19" port="41501" tid="126807170072576" timestamp=1715697145
ollama | llama_model_loader: loaded meta data with 26 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256-89cfc3c0f6670319b01ff116e3a57807ffbd1b8371ddf45afdfedf42972d10e7 (version GGUF V3 (latest))
ollama | llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
ollama | llama_model_loader: - kv 0: general.architecture str = llama
ollama | llama_model_loader: - kv 1: general.name str = hf
ollama | llama_model_loader: - kv 2: llama.context_length u32 = 32768
ollama | llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
ollama | llama_model_loader: - kv 4: llama.block_count u32 = 32
ollama | llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
ollama | llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
ollama | llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
ollama | llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
ollama | llama_model_loader: - kv 9: llama.expert_count u32 = 8
ollama | llama_model_loader: - kv 10: llama.expert_used_count u32 = 2
ollama | llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
ollama | llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000
ollama | llama_model_loader: - kv 13: general.file_type u32 = 15
ollama | llama_model_loader: - kv 14: tokenizer.ggml.model str = llama
ollama | llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
ollama | llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
ollama | llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
ollama | llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,58980] = ["▁ t", "i n", "e r", "▁ a", "h e...
ollama | llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
ollama | llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 2
ollama | llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
ollama | llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true
ollama | llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false
ollama | llama_model_loader: - kv 24: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
ollama | llama_model_loader: - kv 25: general.quantization_version u32 = 2
ollama | llama_model_loader: - type f32: 65 tensors
ollama | llama_model_loader: - type f16: 32 tensors
ollama | llama_model_loader: - type q8_0: 64 tensors
ollama | llama_model_loader: - type q4_K: 673 tensors
ollama | llama_model_loader: - type q5_K: 32 tensors
ollama | llama_model_loader: - type q6_K: 129 tensors
ollama | llm_load_vocab: special tokens definition check successful ( 259/32000 ).
ollama | llm_load_print_meta: format = GGUF V3 (latest)
ollama | llm_load_print_meta: arch = llama
ollama | llm_load_print_meta: vocab type = SPM
ollama | llm_load_print_meta: n_vocab = 32000
ollama | llm_load_print_meta: n_merges = 0
ollama | llm_load_print_meta: n_ctx_train = 32768
ollama | llm_load_print_meta: n_embd = 4096
ollama | llm_load_print_meta: n_head = 32
ollama | llm_load_print_meta: n_head_kv = 8
ollama | llm_load_print_meta: n_layer = 32
ollama | llm_load_print_meta: n_rot = 128
ollama | llm_load_print_meta: n_embd_head_k = 128
ollama | llm_load_print_meta: n_embd_head_v = 128
ollama | llm_load_print_meta: n_gqa = 4
ollama | llm_load_print_meta: n_embd_k_gqa = 1024
ollama | llm_load_print_meta: n_embd_v_gqa = 1024
ollama | llm_load_print_meta: f_norm_eps = 0.0e+00
ollama | llm_load_print_meta: f_norm_rms_eps = 1.0e-05
ollama | llm_load_print_meta: f_clamp_kqv = 0.0e+00
ollama | llm_load_print_meta: f_max_alibi_bias = 0.0e+00
ollama | llm_load_print_meta: f_logit_scale = 0.0e+00
ollama | llm_load_print_meta: n_ff = 14336
ollama | llm_load_print_meta: n_expert = 8
ollama | llm_load_print_meta: n_expert_used = 2
ollama | llm_load_print_meta: causal attn = 1
ollama | llm_load_print_meta: pooling type = 0
ollama | llm_load_print_meta: rope type = 0
ollama | llm_load_print_meta: rope scaling = linear
ollama | llm_load_print_meta: freq_base_train = 1000000.0
ollama | llm_load_print_meta: freq_scale_train = 1
ollama | llm_load_print_meta: n_yarn_orig_ctx = 32768
ollama | llm_load_print_meta: rope_finetuned = unknown
ollama | llm_load_print_meta: ssm_d_conv = 0
ollama | llm_load_print_meta: ssm_d_inner = 0
ollama | llm_load_print_meta: ssm_d_state = 0
ollama | llm_load_print_meta: ssm_dt_rank = 0
ollama | llm_load_print_meta: model type = 8x7B
ollama | llm_load_print_meta: model ftype = Q4_K - Medium
ollama | llm_load_print_meta: model params = 46.70 B
ollama | llm_load_print_meta: model size = 26.49 GiB (4.87 BPW)
ollama | llm_load_print_meta: general.name = hf
ollama | llm_load_print_meta: BOS token = 1 '<s>'
ollama | llm_load_print_meta: EOS token = 2 '</s>'
ollama | llm_load_print_meta: UNK token = 0 '<unk>'
ollama | llm_load_print_meta: LF token = 13 '<0x0A>'
ollama | ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
ollama | ggml_cuda_init: CUDA_USE_TENSOR_CORES: no
ollama | ggml_cuda_init: found 1 CUDA devices:
ollama | Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
ollama | llm_load_tensors: ggml ctx size = 0.83 MiB
ollama | time=2024-05-14T14:32:25.604Z level=INFO source=server.go:524 msg="waiting for server to become available" status="llm server not responding"
ollama | time=2024-05-14T14:32:27.733Z level=INFO source=server.go:524 msg="waiting for server to become available" status="llm server loading model"
ollama | llm_load_tensors: offloading 19 repeating layers to GPU
ollama | llm_load_tensors: offloaded 19/33 layers to GPU
ollama | llm_load_tensors: CUDA_Host buffer size = 11181.09 MiB
ollama | llm_load_tensors: CUDA0 buffer size = 15946.78 MiB
ollama | ....................................................................................................
ollama | llama_new_context_with_model: n_ctx = 8224
ollama | llama_new_context_with_model: n_batch = 512
ollama | llama_new_context_with_model: n_ubatch = 512
ollama | llama_new_context_with_model: freq_base = 1000000.0
ollama | llama_new_context_with_model: freq_scale = 1
ollama | llama_kv_cache_init: CUDA_Host KV buffer size = 417.62 MiB
ollama | llama_kv_cache_init: CUDA0 KV buffer size = 610.38 MiB
ollama | llama_new_context_with_model: KV self size = 1028.00 MiB, K (f16): 514.00 MiB, V (f16): 514.00 MiB
ollama | llama_new_context_with_model: CUDA_Host output buffer size = 0.14 MiB
ollama | llama_new_context_with_model: CUDA0 compute buffer size = 732.63 MiB
ollama | llama_new_context_with_model: CUDA_Host compute buffer size = 24.07 MiB
ollama | llama_new_context_with_model: graph nodes = 1510
ollama | llama_new_context_with_model: graph splits = 160
ollama | INFO [main] model loaded | tid="126807170072576" timestamp=1715697150
ollama | time=2024-05-14T14:32:30.744Z level=INFO source=server.go:529 msg="llama runner started in 5.59 seconds"
ollama | [GIN] 2024/05/14 - 14:32:52 | 200 | 27.925300268s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:33:09 | 200 | 17.307316378s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:33:33 | 200 | 23.84345206s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:33:52 | 200 | 19.311706404s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:34:22 | 500 | 29.378447079s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:35:32 | 200 | 14.198592073s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:36:14 | 200 | 42.173229353s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/14 - 14:36:40 | 200 | 25.886390771s | 172.16.47.1 | POST "/api/generate"The debug output from doclytics still doesn't show the full response but there are hints that do suggest a json problem like : doclytics | May 14 14:35:32.126 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 26
Sometimes it even seems to work somewhat: doclytics | May 14 14:37:10.992 INFO Updating document with ID: 220
doclytics | May 14 14:35:17.606 INFO Application started !
doclytics | May 14 14:35:17.630 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 14:35:17.630 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 14:35:17.630 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 14:35:17.630 DEBG connected to 192.168.0.208:17959
doclytics | May 14 14:35:17.640 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:35:17.640 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 14 14:35:17.640 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 14:35:17.640 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:35:17.926 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:35:17.927 INFO Successfully retrieved 25 Documents
doclytics | May 14 14:35:17.927 INFO Generate Response with LLM model
doclytics | May 14 14:35:17.927 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 14:35:17.927 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 14:35:17.927 DEBG connected to 192.168.0.208:11434
doclytics | May 14 14:35:32.126 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:35:32.126 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 26
doclytics | May 14 14:35:32.126 INFO Generate Response with LLM model
doclytics | May 14 14:35:32.126 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:36:14.300 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:36:14.300 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 30
doclytics | May 14 14:36:14.300 INFO Generate Response with LLM model
doclytics | May 14 14:36:14.300 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:36:40.186 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:36:40.187 ERRO Error parsing llm response json EOF while parsing an object at line 6 column 0
doclytics | May 14 14:36:40.187 INFO Generate Response with LLM model
doclytics | May 14 14:36:40.187 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:01.310 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:01.310 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 37
doclytics | May 14 14:37:01.310 INFO Generate Response with LLM model
doclytics | May 14 14:37:01.310 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:10.992 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:10.992 INFO Updating document with ID: 220
doclytics | May 14 14:37:10.992 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 14:37:10.992 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 14:37:10.992 DEBG connected to 192.168.0.208:17959
doclytics | May 14 14:37:11.075 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:37:11.075 INFO Document with ID: 220 successfully updated
doclytics | May 14 14:37:11.075 INFO Generate Response with LLM model
doclytics | May 14 14:37:11.075 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:37.077 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:37.077 ERRO Error parsing llm response json EOF while parsing an object at line 5 column 35
doclytics | May 14 14:37:37.077 INFO Generate Response with LLM model
doclytics | May 14 14:37:37.077 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:50.076 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:37:50.076 INFO Updating document with ID: 245
doclytics | May 14 14:37:50.076 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 14:37:50.076 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 14:37:50.076 DEBG connected to 192.168.0.208:17959
doclytics | May 14 14:37:50.161 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:37:50.161 INFO Document with ID: 245 successfully updated
doclytics | May 14 14:37:50.161 INFO Generate Response with LLM model
doclytics | May 14 14:37:50.161 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:06.344 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:06.344 ERRO Error parsing llm response json EOF while parsing an object at line 5 column 41
doclytics | May 14 14:38:06.344 INFO Generate Response with LLM model
doclytics | May 14 14:38:06.345 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:24.512 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:24.512 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 39
doclytics | May 14 14:38:24.512 INFO Generate Response with LLM model
doclytics | May 14 14:38:24.512 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:37.124 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:37.125 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 31
doclytics | May 14 14:38:37.125 INFO Generate Response with LLM model
doclytics | May 14 14:38:37.125 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:45.728 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:38:45.728 INFO Updating document with ID: 203
doclytics | May 14 14:38:45.728 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 14:38:45.728 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 14:38:45.728 DEBG connected to 192.168.0.208:17959
doclytics | May 14 14:38:45.892 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 14:38:45.892 INFO Document with ID: 203 successfully updated
doclytics | May 14 14:38:45.892 INFO Generate Response with LLM model
doclytics | May 14 14:38:45.892 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:39:14.556 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:39:14.556 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 34
doclytics | May 14 14:39:14.556 INFO Generate Response with LLM model
doclytics | May 14 14:39:14.556 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:39:36.388 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:39:36.388 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 39
doclytics | May 14 14:39:36.388 INFO Generate Response with LLM model
doclytics | May 14 14:39:36.388 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:40:10.305 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:40:10.305 ERRO Error parsing llm response json EOF while parsing an object at line 5 column 27
doclytics | May 14 14:40:10.305 INFO Generate Response with LLM model
doclytics | May 14 14:40:10.305 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:40:44.716 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:40:44.716 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 32
doclytics | May 14 14:40:44.716 INFO Generate Response with LLM model
doclytics | May 14 14:40:44.716 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:41:05.708 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:41:05.708 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 27
doclytics | May 14 14:41:05.708 INFO Generate Response with LLM model
doclytics | May 14 14:41:05.708 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:41:31.541 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 14:41:31.541 ERRO Error parsing llm response json EOF while parsing a string at line 5 column 24
doclytics | May 14 14:41:31.541 INFO Generate Response with LLM model
doclytics | May 14 14:41:31.541 DEBG reuse idle connection for ("http", 192.168.0.208:11434)One other change I've tried was using a env_file: /mnt/Dockerspace/doclytics/doc.env with
BASE_PROMPT=Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und endenI also tried my baseprompt with mixtral manually via open webui. Just copied the custom baseprompt plus the untouched paperless ocr output. Which should give an approximation of the output:
Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden
Bericht zum Fortführungsantrag einer Verhaltenstherapie
1.
Darstellung
des
bisherigen
Behandlungsverlaufs
seit
letztem Bericht unter
Angabe
der
Veränderungen
in
der
Symptomatik
und
der
Zielerreichungen.............and the models answer looked mostly okay, as far as I can tell:
{
"Datum_Empfang": "2021-07-30",
"Titel": "Bericht zum Fortführungsantrag einer Verhaltenstherapie",
"Zusammenfassung": "Der Patient hat gute Fortschritte in der Behandlung seiner somatoformen Beschwerden gemacht, jedoch gibt es weiterhin Unsicherheiten hinsichtlich seiner epileptischen Anfälle. Diese konnten im Rahmen einer psychogenen nichtepileptischen Erklärung mit sozialphobischer Komponente erklärt werden. Die initiale Diagnose F45.0 ist weitestgehend remittiert, die Stimmung des Patienten ist aufgelockert und weniger eingeengt auf somatische Beschwerden. Es sind weitere Therapiesitzungen geplant, um die sozialphobischen dysfunktionalen Kognitionen und Schemata im Rahmen der pseudoepileptischen Anfälle zu modifizieren.",
"Kategorie": "Psychotherapie Bericht"
}This is a bug I unfortunately introduced with the latest change for the output parsing function. Thank you for helping with the debugging. I appreciate it! The latest update should fix that parsing bug and also further improve the logging.
Deathproof76
commented
6 months ago And thank you for such a cool open project! Will pull the update and report back, as soon as possible, most likely in a few hours though 👍 another thing I noticed is that django doesn't seem to be vexed anymore. at least there are no indications in the paperless logs regarding bad requests or otherwise
Deathproof76
commented
6 months ago Could pull v1.1.4-rc.6 faster than anticipated. Sadly Django seems angry again:
paperless-webserver-1 | [2024-05-14 17:41:06,789] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:41:37,203] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:42:50,304] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:43:14,443] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:43:41,809] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:44:15,117] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-paperdb-1 | 2024-05-14 15:44:15.924 UTC [26] LOG: checkpoint starting: time
paperless-paperdb-1 | 2024-05-14 15:44:16.950 UTC [26] LOG: checkpoint complete: wrote 11 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.003 s, sync=0.012 s, total=1.027 s; sync files=11, longest=0.011 s, average=0.002 s; distance=52 kB, estimate=481 kB; lsn=0/25956B8, redo lsn=0/2595680
paperless-webserver-1 | [2024-05-14 17:44:51,298] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-14 17:45:24,831] [WARNING] [django.request] Bad Request: /api/documents/261/due to this the logs are most likely more quiet than before:
doclytics | May 14 15:40:25.849 INFO Application started !
doclytics | May 14 15:40:25.876 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 15:40:25.876 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 15:40:25.876 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 15:40:25.876 DEBG connected to 192.168.0.208:17959
doclytics | May 14 15:40:25.889 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:40:25.889 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 14 15:40:25.889 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 15:40:25.889 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:40:26.210 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:40:26.210 INFO Successfully retrieved 25 Documents
doclytics | May 14 15:40:26.210 INFO Generate Response with LLM model
doclytics | May 14 15:40:26.210 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 15:40:26.210 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 15:40:26.210 DEBG connected to 192.168.0.208:11434
doclytics | May 14 15:41:06.769 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 15:41:06.769 INFO Updating document with ID: 261
doclytics | May 14 15:41:06.769 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 15:41:06.769 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 15:41:06.769 DEBG connected to 192.168.0.208:17959
doclytics | May 14 15:41:06.789 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:41:06.789 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/261/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/261/", query: None, fragment: None } }
doclytics exited with code 0
doclytics | May 14 15:42:13.879 INFO Application started !
doclytics | May 14 15:42:13.904 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 14 15:42:13.904 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 15:42:13.904 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 15:42:13.904 DEBG connected to 192.168.0.208:17959
doclytics | May 14 15:42:13.914 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:42:13.914 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 14 15:42:13.914 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 14 15:42:13.914 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:42:14.239 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:42:14.239 INFO Successfully retrieved 25 Documents
doclytics | May 14 15:42:14.239 INFO Generate Response with LLM model
doclytics | May 14 15:42:14.239 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 14 15:42:14.239 DEBG connecting to 192.168.0.208:11434
doclytics | May 14 15:42:14.239 DEBG connected to 192.168.0.208:11434
doclytics | May 14 15:42:50.280 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 14 15:42:50.280 INFO Updating document with ID: 261
doclytics | May 14 15:42:50.280 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 14 15:42:50.280 DEBG connecting to 192.168.0.208:17959
doclytics | May 14 15:42:50.280 DEBG connected to 192.168.0.208:17959
doclytics | May 14 15:42:50.304 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 14 15:42:50.304 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/261/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/261/", query: None, fragment: None } }
doclytics exited with code 0tested shortly going back to v1.1.4-rc.5 and django is appeased again like https://github.com/B-urb/doclytics/issues/44#issuecomment-2110453693
tural-ali
commented
6 months ago I'm experiencing exactly the same issue: here is my docker compose:
version: "3"
services:
doclytics:
image: bjoern5urban/doclytics:development
container_name: doclytics
environment:
PAPERLESS_BASE_URL: http://192.168.68.59:8001
PAPERLESS_TOKEN: xyz
LOG_LEVEL: DEBUG
OLLAMA_HOST: 192.168.68.59
OLLAMA_PORT: "11434"
OLLAMA_MODEL: llama3
volumes:
- /volume1/docker/doclytics:/app/data
restart: unless-stoppedI double-checked the token and URL. both are actual ones
The error message I get after starting the docker is:
Error while interacting with paperless: HTTP status client error (400 Bad Request) for url (http://192.168.68.59:8001/api/documents/?query=NOT%20tagged=true)
B-urb
commented
6 months ago @tural-ali did you set the ALLOWED_HOSTS in paperless correctly ? This error during document retrieval should have a different cause: https://github.com/B-urb/doclytics/issues/33#issuecomment-2095535993
B-urb
commented
6 months ago @Deathproof76 Okay, i am really kinda baffled why there is no debug logs for the llm response in anyway. I will try to debug this a bit more.
tural-ali
commented
6 months ago @tural-ali did you set the ALLOWED_HOSTS in paperless correctly ? This error during document retrieval should have a different cause: https://github.com/B-urb/doclytics/issues/33#issuecomment-2095535993
I'm running both of them as Docker containers in Synology NAS. Both of them use "Bridge" Networking. As far as I know, the default value for ALOWED_HOSTS is all possible IPs, no? If no, what value should it be for Doculytics' container?
Deathproof76
commented
6 months ago Pulled v1.1.4-rc.8 and tried again, still bad requests. Same settings/baseprompt/custom fields etc that worked somewhat with v1.1.4-rc.5 (where I got ERRO Error parsing llm response json EOF while parsing a string at line 5 column 27 in doclytics but no bad request from django in paperless).
v1.1.4-rc.8 logs:
paperless:
paperless-webserver-1 | [2024-05-15 20:01:24,976] [WARNING] [django.request] Bad Request: /api/documents/261/
paperless-webserver-1 | [2024-05-15 20:02:02,738] [WARNING] [django.request] Bad Request: /api/documents/261/ollama:
ollama | time=2024-05-15T18:00:38.961Z level=INFO source=server.go:529 msg="llama runner started in 6.35 seconds"
ollama | [GIN] 2024/05/15 - 18:01:24 | 200 | 52.919666811s | 172.16.47.1 | POST "/api/generate"
ollama | [GIN] 2024/05/15 - 18:02:02 | 200 | 37.10154551s | 172.16.47.1 | POST "/api/generate"doclytics:
doclytics | May 15 18:00:31.839 INFO Application started !
doclytics | May 15 18:00:31.868 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 15 18:00:31.868 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 15 18:00:31.868 DEBG connecting to 192.168.0.208:17959
doclytics | May 15 18:00:31.868 DEBG connected to 192.168.0.208:17959
doclytics | May 15 18:00:31.879 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 15 18:00:31.879 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 15 18:00:31.879 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 15 18:00:31.879 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 15 18:00:32.035 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 15 18:00:32.036 INFO Successfully retrieved 25 Documents
doclytics | May 15 18:00:32.036 INFO Generate Response with LLM model
doclytics | May 15 18:00:32.036 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 15 18:00:32.036 DEBG connecting to 192.168.0.208:11434
doclytics | May 15 18:00:32.036 DEBG connected to 192.168.0.208:11434
doclytics | May 15 18:01:24.956 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 15 18:01:24.956 INFO Updating document with ID: 261
doclytics | May 15 18:01:24.956 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 15 18:01:24.956 DEBG connecting to 192.168.0.208:17959
doclytics | May 15 18:01:24.957 DEBG connected to 192.168.0.208:17959
doclytics | May 15 18:01:24.976 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 15 18:01:24.976 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/261/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/261/", query: None, fragment: None } }
doclytics exited with code 0Maybe there's a way to improve logging from the paperless-django side to determine why the request are bad?
Just spitballing: Could it be a docker problem? (Or part of the problem) Like how the envs are parsed? Regarding bad request after retrieval - while updating document fields it must be something that happened between v1.1.4-rc.5 and v1.1.4-rc.6 which I think is https://github.com/B-urb/doclytics/commit/b3ea09ae8150a2927f405eea6010c696147377e0
Also maybe this helps with debugging (not a programmer just used ai🤷♂️):
use ollama_rs::generation::completion::GenerationResponse;
use ollama_rs::generation::completion::request::GenerationRequest;
use ollama_rs::Ollama;
use crate::Document;
pub async fn generate_response(
ollama: &Ollama,
model: &String,
prompt_base: &String,
document: &Document,
) -> std::result::Result<GenerationResponse, Box<dyn std::error::Error>> {
let prompt = format!("{} {}", document.content, prompt_base);
let res = ollama
.generate(GenerationRequest::new(model.clone(), prompt))
.await;
match res {
Ok(res) => {
slog_scope::debug!("Before calling slog_scope::debug! in generate_response");
slog_scope::debug!("Response from ollama: {}", res.response);
slog_scope::debug!("After calling slog_scope::debug! in generate_response");
Ok(res)
},
Err(e) => {
slog_scope::error!("{}", e);
Err(e.into())
}
}
}@tural-ali did you set the ALLOWED_HOSTS in paperless correctly ? This error during document retrieval should have a different cause: #33 (comment)
I'm running both of them as Docker containers in Synology NAS. Both of them use "Bridge" Networking. As far as I know, the default value for ALOWED_HOSTS is all possible IPs, no? If no, what value should it be for Doculytics' container?
@tural-ali most likely this but for the paperless webserver container
environment:
PAPERLESS_URL: http://192.168.68.59:8001
PAPERLESS_ALLOWED_HOSTS: http://192.168.68.59:8001,localhost,127.0.0.1,192.168.68.59:8001
PAPERLESS_CORS_ALLOWED_HOSTS: http://192.168.68.59:8001
PAPERLESS_CSRF_TRUSTED_ORIGINS: http://192.168.68.59:8001you can also put it in the .env if you specified one but then it should be
PAPERLESS_URL=http://192.168.68.59:8001
PAPERLESS_ALLOWED_HOSTS=http://192.168.68.59:8001,localhost,127.0.0.1,192.168.68.59:8001
etc...I think this might be the reason https://docs.paperless-ngx.com/configuration/#PAPERLESS_CSRF_TRUSTED_ORIGINS
tural-ali
commented
6 months ago @tural-ali did you set the ALLOWED_HOSTS in paperless correctly ? This error during document retrieval should have a different cause: #33 (comment)
I'm running both of them as Docker containers in Synology NAS. Both of them use "Bridge" Networking. As far as I know, the default value for ALOWED_HOSTS is all possible IPs, no? If no, what value should it be for Doculytics' container?
@tural-ali most likely this but for the paperless webserver container
environment: PAPERLESS_URL: http://192.168.68.59:8001 PAPERLESS_ALLOWED_HOSTS: http://192.168.68.59:8001,localhost,127.0.0.1,192.168.68.59:8001 PAPERLESS_CORS_ALLOWED_HOSTS: http://192.168.68.59:8001 PAPERLESS_CSRF_TRUSTED_ORIGINS: http://192.168.68.59:8001you can also put it in the .env if you specified one but then it should be
PAPERLESS_URL=http://192.168.68.59:8001 PAPERLESS_ALLOWED_HOSTS=http://192.168.68.59:8001,localhost,127.0.0.1,192.168.68.59:8001 etc...I think this might be the reason https://docs.paperless-ngx.com/configuration/#PAPERLESS_CSRF_TRUSTED_ORIGINS
@B-urb @Deathproof76 thank you for your help. but I'm still having this issue:
Here is the tag that I created
Here is paperless log
And here is the doclytics log
Deathproof76
commented
6 months ago @tural-ali What happens when you open this url in your browser in your local network? http://192.168.68.59:8001/api/documents/?query=NOT%20tagged=true
tural-ali
commented
6 months ago @tural-ali What happens when you open this url in your browser in your local network? http://192.168.68.59:8001/api/documents/?query=NOT%20tagged=true
Error listing search results, check logs for more detail.
tural-ali
commented
6 months ago @tural-ali did you set the ALLOWED_HOSTS in paperless correctly ? This error during document retrieval should have a different cause: #33 (comment)
here is my full docker compose file
version: '3.8'
services:
broker:
image: redis
container_name: Paperless-NGX-REDIS
restart: always
volumes:
- /volume1/docker/paperless/redis:/data
doclytics:
image: bjoern5urban/doclytics:v1.1.3
container_name: doclytics
environment:
PAPERLESS_BASE_URL: http://192.168.68.59:8001
PAPERLESS_TOKEN: your-paperless-token
LOG_LEVEL: DEBUG
OLLAMA_HOST: 192.168.68.59
OLLAMA_PORT: "11434"
OLLAMA_MODEL: llama3
volumes:
- /volume1/docker/doclytics:/app/data
depends_on:
- ollama
- webserver
restart: unless-stopped
ollama:
image: ollama/ollama:latest
container_name: ollama
ports:
- 11434:11434
volumes:
- /volume1/docker/paperless/ollama:/root/.ollama
pull_policy: always
tty: true
restart: always
db:
image: postgres
container_name: Paperless-NGX-DB
restart: always
volumes:
- /volume1/docker/paperless/db:/var/lib/postgresql/data
environment:
POSTGRES_DB: paperless
POSTGRES_USER: paperless
POSTGRES_PASSWORD: your-postgres-password
webserver:
image: ghcr.io/paperless-ngx/paperless-ngx:latest
container_name: Paperless-NGX
restart: always
depends_on:
- db
- broker
ports:
- 8001:8000
volumes:
- /volume1/docker/paperless/data:/usr/src/paperless/data
- /volume1/docker/paperless/media:/usr/src/paperless/media
- /volume1/docker/paperless/export:/usr/src/paperless/export
- /volume1/docker/paperless/consume:/usr/src/paperless/consume
environment:
PAPERLESS_REDIS: redis://broker:6379
PAPERLESS_DBHOST: db
USERMAP_UID: 1026
USERMAP_GID: 100
PAPERLESS_TIME_ZONE: Europe/Berlin
PAPERLESS_SECRET_KEY: your-secret-key
PAPERLESS_ADMIN_USER: admin
PAPERLESS_ADMIN_PASSWORD: your-admin-password
PAPERLESS_DBENGINE: postgresql
PAPERLESS_DBHOST: 192.168.68.59
PAPERLESS_DBPORT: 5431
PAPERLESS_DBNAME: paperless
PAPERLESS_DBUSER: paperless
PAPERLESS_DBPASS: your-db-password
PAPERLESS_FILENAME_FORMAT: "{created_year}/{correspondent}/{title}"
PAPERLESS_OCR_USER_ARGS: '{"invalidate_digital_signatures": true}'
PAPERLESS_OCR_LANGUAGE: "ces+tur+aze+deu+eng+rus"
PAPERLESS_OCR_LANGUAGES: "ces tur aze deu eng rus"
PAPERLESS_URL: http://192.168.68.59:8001
PAPERLESS_ALLOWED_HOSTS: "http://192.168.68.59:8001,localhost,127.0.0.1,192.168.68.59:8001"
PAPERLESS_CORS_ALLOWED_HOSTS: "http://192.168.68.59:8001"
PAPERLESS_CSRF_TRUSTED_ORIGINS: "http://192.168.68.59:8001"
PAPERLESS_DEBUG: "true"@tural-ali What happens when you open this url in your browser in your local network? http://192.168.68.59:8001/api/documents/?query=NOT%20tagged=true
Error listing search results, check logs for more detail.
what did the logs say? anyway it seems like a problem not related to doclytics in this case. because doclytics just uses this request to retrieve the documents etc. If it doesn't work from your browser the problem lies with paperless. maybe a general database problem. reindexing could help https://github.com/paperless-ngx/paperless-ngx/discussions/4201 https://docs.paperless-ngx.com/administration/#index
basically open a terminal to the paperless webserver and run webserver document_index reindex at worst nothing happens, best case scenario it solves the problem.
tural-ali
commented
6 months ago @tural-ali What happens when you open this url in your browser in your local network? http://192.168.68.59:8001/api/documents/?query=NOT%20tagged=true
Error listing search results, check logs for more detail.
what did the logs say? anyway it seems like a problem not related to doclytics in this case. because doclytics just uses this request to retrieve the documents etc. If it doesn't work from your browser the problem lies with paperless. maybe a general database problem. reindexing could help paperless-ngx/paperless-ngx#4201 https://docs.paperless-ngx.com/administration/#index
basically open a terminal to the paperless webserver and run
webserver document_index reindexat worst nothing happens, best case scenario it solves the problem.
you're a life saver:) thank you very much. fixed
B-urb
commented
6 months ago @Deathproof76 I just tried your german prompt with mixtral:8x7b and i get the correct log from the llm with json included. Can you try again with rc.9 ?
Deathproof76
commented
6 months ago Omfg, I tried to see more via ollama debug logging and it looks like that part of the problem must have been my modelfile with custom prompt template. When I switched to the basic mixtral:latest and only upped the context size + latest doclytics it started working on the test documents I've prepared. Updating custom fields and everything.
doclytics | May 16 14:29:18.363 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 14:29:18.394 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 14:29:18.394 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:18.394 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:18.394 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:18.405 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.405 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 14:29:18.405 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 14:29:18.405 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.660 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.660 INFO Successfully retrieved 25 Documents
doclytics | May 16 14:29:18.660 INFO Generate Response with LLM model
doclytics | May 16 14:29:18.660 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 14:29:18.660 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 14:29:18.661 DEBG connected to 192.168.0.208:11434
doclytics | May 16 14:29:34.800 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:34.800 INFO Updating document with ID: 264
doclytics | May 16 14:29:34.800 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:34.800 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:34.800 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:34.867 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:34.867 INFO Document with ID: 264 successfully updated
doclytics | May 16 14:29:34.867 INFO Generate Response with LLM model
doclytics | May 16 14:29:34.867 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:42.956 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:42.956 INFO Updating document with ID: 263
doclytics | May 16 14:29:42.956 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:42.956 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:42.956 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:43.035 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:43.035 INFO Document with ID: 263 successfully updated
doclytics | May 16 14:29:43.035 INFO Generate Response with LLM model
doclytics | May 16 14:29:43.035 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:28.559 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:28.560 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:30:28.560 INFO Generate Response with LLM model
doclytics | May 16 14:30:28.560 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:40.660 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:40.660 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:30:40.660 INFO Generate Response with LLM model
doclytics | May 16 14:30:40.660 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:31:06.864 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:31:06.864 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:31:06.864 INFO Generate Response with LLM model
doclytics | May 16 14:31:06.864 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics exited with code 0stopped the container here
ollama:
llama_new_context_with_model: graph splits = 160
DEBUG [initialize] initializing slots | n_slots=1 tid="125100883189760" timestamp=1715869765
DEBUG [initialize] new slot | n_ctx_slot=30208 slot_id=0 tid="125100883189760" timestamp=1715869765
INFO [main] model loaded | tid="125100883189760" timestamp=1715869765
DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="125100883189760" timestamp=1715869765
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=0 tid="125100883189760" timestamp=1715869765
time=2024-05-16T14:29:25.682Z level=INFO source=server.go:545 msg="llama runner started in 6.83 seconds"
time=2024-05-16T14:29:25.682Z level=DEBUG source=sched.go:351 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8
time=2024-05-16T14:29:25.682Z level=DEBUG source=routes.go:179 msg="generate handler" prompt="Datum: 16.05.2024\n\nTestdokument\n\nIch habe hier mal einfach was geschrieben um zu testen. Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden"
time=2024-05-16T14:29:25.682Z level=DEBUG source=routes.go:180 msg="generate handler" template=" [INST] {{ .System }} {{ .Prompt }} [/INST]"
time=2024-05-16T14:29:25.682Z level=DEBUG source=routes.go:181 msg="generate handler" system=""
time=2024-05-16T14:29:25.682Z level=DEBUG source=routes.go:212 msg="generate handler" prompt=" [INST] Datum: 16.05.2024\n\nTestdokument\n\nIch habe hier mal einfach was geschrieben um zu testen. Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden [/INST]"
time=2024-05-16T14:29:25.682Z level=DEBUG source=server.go:632 msg="setting token limit to 10x num_ctx" num_ctx=30196 num_predict=301960
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=1 tid="125100883189760" timestamp=1715869765
DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=2 tid="125100883189760" timestamp=1715869765
DEBUG [update_slots] slot progression | ga_i=0 n_past=0 n_past_se=0 n_prompt_tokens_processed=248 slot_id=0 task_id=2 tid="125100883189760" timestamp=1715869765
DEBUG [update_slots] kv cache rm [p0, end) | p0=0 slot_id=0 task_id=2 tid="125100883189760" timestamp=1715869765
DEBUG [print_timings] prompt eval time = 995.00 ms / 248 tokens ( 4.01 ms per token, 249.25 tokens per second) | n_prompt_tokens_processed=248 n_tokens_second=249.24748365569678 slot_id=0 t_prompt_processing=994.995 t_token=4.0120766129032255 task_id=2 tid="125100883189760" timestamp=1715869774
DEBUG [print_timings] generation eval time = 8038.14 ms / 75 runs ( 107.18 ms per token, 9.33 tokens per second) | n_decoded=75 n_tokens_second=9.330517922121027 slot_id=0 t_token=107.17518666666666 t_token_generation=8038.139 task_id=2 tid="125100883189760" timestamp=1715869774
DEBUG [print_timings] total time = 9033.13 ms | slot_id=0 t_prompt_processing=994.995 t_token_generation=8038.139 t_total=9033.134 task_id=2 tid="125100883189760" timestamp=1715869774
DEBUG [update_slots] slot released | n_cache_tokens=323 n_ctx=30208 n_past=322 n_system_tokens=0 slot_id=0 task_id=2 tid="125100883189760" timestamp=1715869774 truncated=false
DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=50090 status=200 tid="125100273696768" timestamp=1715869774
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=80 tid="125100883189760" timestamp=1715869774
DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=38132 status=200 tid="125100263211008" timestamp=1715869774
[GIN] 2024/05/16 - 14:29:34 | 200 | 16.139219451s | 172.16.167.1 | POST "/api/generate"
time=2024-05-16T14:29:34.800Z level=DEBUG source=sched.go:355 msg="context for request finished"
time=2024-05-16T14:29:34.800Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8 duration=5m0s
time=2024-05-16T14:29:34.800Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8 refCount=0
time=2024-05-16T14:29:34.868Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=81 tid="125100883189760" timestamp=1715869774
time=2024-05-16T14:29:34.868Z level=DEBUG source=routes.go:179 msg="generate handler" prompt="Datum: 16.05.2024 \nTestdokument\nIch habe hier mal einfach was geschrieben um zu testen. Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden"
time=2024-05-16T14:29:34.868Z level=DEBUG source=routes.go:180 msg="generate handler" template=" [INST] {{ .System }} {{ .Prompt }} [/INST]"
time=2024-05-16T14:29:34.868Z level=DEBUG source=routes.go:181 msg="generate handler" system=""
time=2024-05-16T14:29:34.868Z level=DEBUG source=routes.go:212 msg="generate handler" prompt=" [INST] Datum: 16.05.2024 \nTestdokument\nIch habe hier mal einfach was geschrieben um zu testen. Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden [/INST]"
time=2024-05-16T14:29:34.868Z level=DEBUG source=server.go:632 msg="setting token limit to 10x num_ctx" num_ctx=30196 num_predict=301960
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=82 tid="125100883189760" timestamp=1715869774
DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=83 tid="125100883189760" timestamp=1715869774
DEBUG [update_slots] slot progression | ga_i=0 n_past=20 n_past_se=0 n_prompt_tokens_processed=227 slot_id=0 task_id=83 tid="125100883189760" timestamp=1715869774
DEBUG [update_slots] kv cache rm [p0, end) | p0=20 slot_id=0 task_id=83 tid="125100883189760" timestamp=1715869774
DEBUG [print_timings] prompt eval time = 972.25 ms / 227 tokens ( 4.28 ms per token, 233.48 tokens per second) | n_prompt_tokens_processed=227 n_tokens_second=233.47976388716035 slot_id=0 t_prompt_processing=972.247 t_token=4.283026431718062 task_id=83 tid="125100883189760" timestamp=1715869782
DEBUG [print_timings] generation eval time = 7026.03 ms / 70 runs ( 100.37 ms per token, 9.96 tokens per second) | n_decoded=70 n_tokens_second=9.962944961133841 slot_id=0 t_token=100.37192857142857 t_token_generation=7026.035 task_id=83 tid="125100883189760" timestamp=1715869782
DEBUG [print_timings] total time = 7998.28 ms | slot_id=0 t_prompt_processing=972.247 t_token_generation=7026.035 t_total=7998.282 task_id=83 tid="125100883189760" timestamp=1715869782
DEBUG [update_slots] slot released | n_cache_tokens=317 n_ctx=30208 n_past=316 n_system_tokens=0 slot_id=0 task_id=83 tid="125100883189760" timestamp=1715869782 truncated=false
DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=38132 status=200 tid="125100263211008" timestamp=1715869782
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=156 tid="125100883189760" timestamp=1715869782
DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=37266 status=200 tid="125100252725248" timestamp=1715869782
[GIN] 2024/05/16 - 14:29:42 | 200 | 8.088376399s | 172.16.167.1 | POST "/api/generate"
time=2024-05-16T14:29:42.956Z level=DEBUG source=sched.go:304 msg="context for request finished"
time=2024-05-16T14:29:42.956Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8 duration=5m0s
time=2024-05-16T14:29:42.956Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8 refCount=0
time=2024-05-16T14:29:43.035Z level=DEBUG source=sched.go:447 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=157 tid="125100883189760" timestamp=1715869783
time=2024-05-16T14:29:43.035Z level=DEBUG source=routes.go:179 msg="generate handler" prompt="Fortführungsantrag Beispiel – Verlängerung von 60 auf 80 Stunden\n\n\n1. Darstellung des bisherigen Behandlungsverlaufs seit dem letzten Bericht, der Veränderung der\nSymptomatik und des Behandlungsergebnisses in Bezug auf die Erreichung bzw. Nichterreichung der\nTherapieziele\nWährend Frau P. zu Beginn der Therapie (Februar 20xx) arbeitslos war, konnte sie schnell motiviert werden,\nwieder eine berufliche Tätigkeit aufzunehmen (Bekleidungsgeschäft), in der sie positive Erfahrungen sammeln\nkonnte (zufriedene Chefin, auch private Treffen mit Kolleg:innen).\nDie Emotionsregulation ist verbessert (Einsatz von Skills wie Musikhören oder Brausepulver, z.B. bei\nSpannungszuständen durch Konflikte mit Kolleg:innen) und sie hat größeres Vertrauen, dass unangenehme\nGefühle auch wieder vorbeigehen.\nPsychoedukation half Frau P., die Symptome zu verstehen und das Selbstwerterleben zu erhöhen.\nDas selbstverletzende Verhalten (Schneiden), welches vor Therapiebeginn und während der probatorischen\nSitzungen noch auftrat, hat Frau P. komplett eingestellt.\nStarke Spannungszustände, starke innere Wut, Gefühle von Gefühllosigkeit sowie Leere und „Chaos“ im Kopf\ntreten deutlich seltener auf als noch zu Therapiebeginn. Hilfreich dabei waren Übungen zur\nGefühlswahrnehmung: Frau P. kann ihre Gefühle deutlich besser erkennen, differenzieren und Ursachen\nzuordnen, was ihr zu Beginn der Therapie kaum möglich war.\nFrau P., welche am Anfang lange brauchte, um Vertrauen zu fassen und ihre Problematik eher „schönredete“\n(v.a. die Alkoholabhängigkeit des Partners, welche sie sich nun erstmal eingestanden habe) konnte sich im\nVerlauf der Behandlung gut auf die therapeutische Beziehung einlassen und berichtet zunehmend\nvertrauensvoll über Unzufriedenheit in ihrer Partnerschaft. Sie hat sich z.B. im Internet Informationen (auch\nzur Hilfe für Angehörige) gesucht und sie konnte beginnen, einzelne co-abhängige Verhaltsweisen nicht mehr\nauszuführen (z.B. Flaschen wegräumen). Ihre Ängste, den Partner zu verlieren, haben sich reduziert, sie kann\nnun ihre eigenen Gefühle und Bedürfnisse als „richtig“ anerkennen und erstmals auch den Gedanken an eine\nmögliche Trennung vom Partner zulassen, da sie sich „nicht alles gefallen lassen müsse“.\nFrau P. zeigt besseres Selbstfürsorgeverhalten als zu Beginn der Behandlung (z.B. relativ regelmäßige\nMahlzeiten).\nDas Selbstwertgefühl, welches sie zu Therapiebeginn als stark eingeschränkt erlebte (Ekel vor sich selbst,\nUnzufriedenheit mit dem eigenen Körper, Abhängigkeitsgefühl vom Partner) besserte sich deutlich, was sich\ndarin äußerte, dass sie Konflikte nicht mehr nur sich selbst zuschrieb, bei Konflikten im Beruf und in der\nPartnerschaft weniger grübelte (aus Angst, „etwas Falsches zu sagen“), sondern Konflikte offener anging.\nWährend des Therapieverlaufs traten jedoch zwei Phasen auf, in denen es für Frau P. unangenehm schien,\nsich mit ihren Schwierigkeiten auseinanderzusetzen, sie sagte dann Termine ab und meldete sich wochenlang\nnicht / war nicht erreichbar. Nachdem es Ende 20xx zu einer fast dreimonatigen Unterbrechung der Therapie\nkam, meldete sich Frau P. auf einen Brief meinerseits (Interpretation ihres Nicht-Meldens als Wunsch, die\nTherapie zu beenden) sofort, entschuldigte sich vielmals und nahm seither die Termine zuverlässig wahr. Es\nkönnte thematisiert werden, dass es Phasen gab, in denen sie generell nicht ans Telefon ging und sich dann zu\nsehr schämte, sich nach längerer Zeit wieder zu melden.\n\n2. Aktuelle Diagnose/n gemäß ICD-10 und aktueller psychischer Befund, weitere Ergebnisse\npsychodiagnostischer Testverfahren\nICD-10 F60.31 Emotional-instabile Persönlichkeitsstörung, Borderline-Typ\nFrau P. ist eine gepflegte, kräftig gebaute Frau mit lila gefärbten Haaren und dunkler Kleidung. Sie ist wach, zu\nallen Qualitäten orientiert, freundlich und sehr bemüht auf alle gestellten Fragen genaue Antworten zu geben,\nwodurch sie oft zögert bevor sie etwas sagt bzw. sehr lange nachdenkt. Aufmerksamkeit und Gedächtnis sind\nohne Befund, ihre Konzentration beschreibt sie durch eine Leere im Kopf noch manchmal als eingeschränkt.\nDiese Beschreibung deckt sich mit der Außenbeobachtung des verzögerten Antwortverhaltens der Patientin.\nSie beschreibt ausgeprägtes Grübeln, besonders über partnerschaftliche Konflikte. Zwänge liegen nicht vor.\nPhobische Ängste liegen bezüglich Spinnen und offenen Gewässern vor. Es bestehen Befürchtungen vor dem\nVerlassenwerden. Sinnestäuschungen und Ich-Störungen liegen nicht vor. Bezüglich der Affektivität berichtet\nsie, sich oft von Gefühlen überrollt zu fühlen, sie ist affektlabil und berichtet schwankende Gefühle zwischen\nEuphorie und Dysphorie sowie Ängstlichkeit, Gereiztheit und innere Unruhe. Bei Konflikten treten Gefühle der\nGefühllosigkeit auf. Der Antrieb ist regelrecht, das Befinden sei abends (besonders bei Dunkelheit) schlechter,\n\n1\nFortführungsantrag Beispiel – Verlängerung von 60 auf 80 Stunden\n\n\nbei gutem Wetter besser. Der Schlaf ist bis auf einzelne intensive Träume ohne Befund. Selbstverletzendes\nVerhalten liegt nicht mehr vor. Es bestehen keine Suizidgedanken oder Suizidpläne sowie kein Hinweis auf\nFremdgefährdung.\nFrau P. nimmt therapiebegleitend Citalopram (30mg) ein, derzeit regelmäßig, im Therapieverlauf fiel ihr die\nregelmäßige Einnahme jedoch teilweise schwer, was wiederholt angesprochen wurde. Frau P. ist Raucherin,\nnimmt keine sonstigen Drogen und trinkt keinen Alkohol.\nTestbefunde:\nDer Wert im Beck-Depressions-Inventar (BDI-II) liegt mit 11 Punkten im subklinischen Bereich. In der\nBorderline-Symptom-Liste (BSL-23) zeigt sich ein Mittelwert von 2,609 (Skala von 0 = Symptom überhaupt\nnicht vorhanden bis 4 = sehr stark), was einem Prozentrang von 68 entspricht.\n\n3. Begründung der Notwendigkeit der Fortführung der Behandlung, weitere Therapieplanung,\ngeänderte/erweiterte Behandlungsziele, geänderte Behandlungsmethoden und -techniken, Prognose, Planung\ndes Therapieabschlusses, ggf. weiterführende Maßnahmen nach Ende der Therapie\nIm Laufe der Behandlung wurde zunehmend deutlich, dass der Partner schwerer Alkoholiker ist, hiermit\nverbundene zeigten sich Konflikte (Aggressivität, tägliches Betrunkensein des Partners, kaum Unterstützung\nbei der Bewältigung des Haushalts und der Pflege der Hündin, Unzuverlässigkeit), welche daraufhin in den im\nMittelpunkt der Therapie rückten. Es fällt Frau P. weiterhin schwer, Wut rechtzeitig wahrzunehmen und diese\ndeutlich auszudrücken, v.a. gegenüber ihrem Partner.\nFrau P. möchte ihre erlernte Fähigkeit, Gefühle und Bedürfnisse zu erkennen und zu benennen beibehalten\nund weiter ausbauen. Zudem soll die bereits verbesserte Konfliktfähigkeit weiter ausgebaut werden und sie\nmöchte sich insgesamt für zukünftige schwierige Situationen stabilisierter fühlen. Diese Ziele werden aus\ntherapeutischer Sicht unterstützt, zudem soll das Thema „Co-Abhängigkeit“ weiterhin bearbeitet, co-\nabhängiges Verhalten soll abgebaut werden, da dies / die Partnerschaftskonflikte als aufrechterhaltend für die\nSymptomatik angesehen werden. Außerdem kann das Verhalten, das Frau P. teilweise von ihrem Partner\n„erduldet“ (Anbrüllen, Nicht-Ernstnehmen ihrer Bedürfnisse, ständige Trunkenheit, Unzuverlässigkeit,\nmangelnde Unterstützung) als verdeckt selbstschädigendes Verhalten interpretiert werden.\nFolgende Strategien sollen zur Erreichung der Ziele eingesetzt werden:\n\uf0b7 Verfestigung und weiterer Ausbau des Selbstfürsorgeverhaltens (regelmäßiges Essen, Einplanen von\nangenehmen, entspannenden Aktivitäten)\n\uf0b7 Ausbau des Freizeitverhaltens (auch unabhängig vom Partner), z.B. Unternehmungen mit\nFreundinnen, Wiederaufnahme der Kontakte zu älteren Freunden, Umsetzung des geplanten Eintritts\nin einen Sportverein\n\uf0b7 Ausbau von Gefühlswahrnehmung, Gefühlsausdruck und Konfliktfähigkeit, z.B. mit Gefühlsprotokollen\nsowie übenden Rollenspielen\n\uf0b7 Förderung des Selbstwerts und Stabilisierung durch Installieren eines „positiven Begleiters“ (nach\nPotreck-Rose, 2010)\n\uf0b7 Weitere Reduktion des co-abhängigen Verhaltens durch Thematisieren und Abbau dazugehöriger\nVerhaltensweisen\n\uf0b7 Bearbeiten von Wünschen und Konflikten in der Partnerschaft z.B. durch Exploration der Wünsche\nund Bedürfnisse sowie Validieren angemessener Wünsche (z.B. nach einem respektvollen Umgang mit\nihrer Person)\n\uf0b7 Ausführliche Rückfallprophylaxe mit besonderem Fokus auf zukünftigen antizipierten Belastungen\n\uf0b7 Langsames Ausschleichen der Therapie, um der Patientin über einen längeren Zeitraum Halt zu bieten\nAufgrund der bisherigen Behandlungserfolge sowie der guten therapeutischen Beziehung wird die Prognose\nals günstig beurteilt. Besonderen Wert muss allerdings auf die ausreichende Verfestigung der erlernten\nStrategien sowie auf die Rückfallprophylaxe gelegt werden, damit die Patientin für zukünftige Belastungen\n(z.B. eventuelle Trennung vom alkoholkranken Partner, eventueller Arbeitsplatzverlust) stabilisiert ist.\nEs wird um eine Therapieverlängerung von 20 Stunden gebeten. Die Therapiefrequenz wird zunächst noch\nwöchentlich sein, dann vierzehntägig bzw. monatlich. Danach soll die Therapie ihren Abschluss finden, sofern\nkeine akuten Krisen (wie z.B. Trennung vom Partner) auftreten.\n\n2 Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden"
time=2024-05-16T14:29:43.035Z level=DEBUG source=routes.go:180 msg="generate handler" template=" [INST] {{ .System }} {{ .Prompt }} [/INST]"
time=2024-05-16T14:29:43.035Z level=DEBUG source=routes.go:181 msg="generate handler" system=""
time=2024-05-16T14:29:43.035Z level=DEBUG source=routes.go:212 msg="generate handler" prompt=" [INST] Fortführungsantrag Beispiel – Verlängerung von 60 auf 80 Stunden\n\n\n1. Darstellung des bisherigen Behandlungsverlaufs seit dem letzten Bericht, der Veränderung der\nSymptomatik und des Behandlungsergebnisses in Bezug auf die Erreichung bzw. Nichterreichung der\nTherapieziele\nWährend Frau P. zu Beginn der Therapie (Februar 20xx) arbeitslos war, konnte sie schnell motiviert werden,\nwieder eine berufliche Tätigkeit aufzunehmen (Bekleidungsgeschäft), in der sie positive Erfahrungen sammeln\nkonnte (zufriedene Chefin, auch private Treffen mit Kolleg:innen).\nDie Emotionsregulation ist verbessert (Einsatz von Skills wie Musikhören oder Brausepulver, z.B. bei\nSpannungszuständen durch Konflikte mit Kolleg:innen) und sie hat größeres Vertrauen, dass unangenehme\nGefühle auch wieder vorbeigehen.\nPsychoedukation half Frau P., die Symptome zu verstehen und das Selbstwerterleben zu erhöhen.\nDas selbstverletzende Verhalten (Schneiden), welches vor Therapiebeginn und während der probatorischen\nSitzungen noch auftrat, hat Frau P. komplett eingestellt.\nStarke Spannungszustände, starke innere Wut, Gefühle von Gefühllosigkeit sowie Leere und „Chaos“ im Kopf\ntreten deutlich seltener auf als noch zu Therapiebeginn. Hilfreich dabei waren Übungen zur\nGefühlswahrnehmung: Frau P. kann ihre Gefühle deutlich besser erkennen, differenzieren und Ursachen\nzuordnen, was ihr zu Beginn der Therapie kaum möglich war.\nFrau P., welche am Anfang lange brauchte, um Vertrauen zu fassen und ihre Problematik eher „schönredete“\n(v.a. die Alkoholabhängigkeit des Partners, welche sie sich nun erstmal eingestanden habe) konnte sich im\nVerlauf der Behandlung gut auf die therapeutische Beziehung einlassen und berichtet zunehmend\nvertrauensvoll über Unzufriedenheit in ihrer Partnerschaft. Sie hat sich z.B. im Internet Informationen (auch\nzur Hilfe für Angehörige) gesucht und sie konnte beginnen, einzelne co-abhängige Verhaltsweisen nicht mehr\nauszuführen (z.B. Flaschen wegräumen). Ihre Ängste, den Partner zu verlieren, haben sich reduziert, sie kann\nnun ihre eigenen Gefühle und Bedürfnisse als „richtig“ anerkennen und erstmals auch den Gedanken an eine\nmögliche Trennung vom Partner zulassen, da sie sich „nicht alles gefallen lassen müsse“.\nFrau P. zeigt besseres Selbstfürsorgeverhalten als zu Beginn der Behandlung (z.B. relativ regelmäßige\nMahlzeiten).\nDas Selbstwertgefühl, welches sie zu Therapiebeginn als stark eingeschränkt erlebte (Ekel vor sich selbst,\nUnzufriedenheit mit dem eigenen Körper, Abhängigkeitsgefühl vom Partner) besserte sich deutlich, was sich\ndarin äußerte, dass sie Konflikte nicht mehr nur sich selbst zuschrieb, bei Konflikten im Beruf und in der\nPartnerschaft weniger grübelte (aus Angst, „etwas Falsches zu sagen“), sondern Konflikte offener anging.\nWährend des Therapieverlaufs traten jedoch zwei Phasen auf, in denen es für Frau P. unangenehm schien,\nsich mit ihren Schwierigkeiten auseinanderzusetzen, sie sagte dann Termine ab und meldete sich wochenlang\nnicht / war nicht erreichbar. Nachdem es Ende 20xx zu einer fast dreimonatigen Unterbrechung der Therapie\nkam, meldete sich Frau P. auf einen Brief meinerseits (Interpretation ihres Nicht-Meldens als Wunsch, die\nTherapie zu beenden) sofort, entschuldigte sich vielmals und nahm seither die Termine zuverlässig wahr. Es\nkönnte thematisiert werden, dass es Phasen gab, in denen sie generell nicht ans Telefon ging und sich dann zu\nsehr schämte, sich nach längerer Zeit wieder zu melden.\n\n2. Aktuelle Diagnose/n gemäß ICD-10 und aktueller psychischer Befund, weitere Ergebnisse\npsychodiagnostischer Testverfahren\nICD-10 F60.31 Emotional-instabile Persönlichkeitsstörung, Borderline-Typ\nFrau P. ist eine gepflegte, kräftig gebaute Frau mit lila gefärbten Haaren und dunkler Kleidung. Sie ist wach, zu\nallen Qualitäten orientiert, freundlich und sehr bemüht auf alle gestellten Fragen genaue Antworten zu geben,\nwodurch sie oft zögert bevor sie etwas sagt bzw. sehr lange nachdenkt. Aufmerksamkeit und Gedächtnis sind\nohne Befund, ihre Konzentration beschreibt sie durch eine Leere im Kopf noch manchmal als eingeschränkt.\nDiese Beschreibung deckt sich mit der Außenbeobachtung des verzögerten Antwortverhaltens der Patientin.\nSie beschreibt ausgeprägtes Grübeln, besonders über partnerschaftliche Konflikte. Zwänge liegen nicht vor.\nPhobische Ängste liegen bezüglich Spinnen und offenen Gewässern vor. Es bestehen Befürchtungen vor dem\nVerlassenwerden. Sinnestäuschungen und Ich-Störungen liegen nicht vor. Bezüglich der Affektivität berichtet\nsie, sich oft von Gefühlen überrollt zu fühlen, sie ist affektlabil und berichtet schwankende Gefühle zwischen\nEuphorie und Dysphorie sowie Ängstlichkeit, Gereiztheit und innere Unruhe. Bei Konflikten treten Gefühle der\nGefühllosigkeit auf. Der Antrieb ist regelrecht, das Befinden sei abends (besonders bei Dunkelheit) schlechter,\n\n1\nFortführungsantrag Beispiel – Verlängerung von 60 auf 80 Stunden\n\n\nbei gutem Wetter besser. Der Schlaf ist bis auf einzelne intensive Träume ohne Befund. Selbstverletzendes\nVerhalten liegt nicht mehr vor. Es bestehen keine Suizidgedanken oder Suizidpläne sowie kein Hinweis auf\nFremdgefährdung.\nFrau P. nimmt therapiebegleitend Citalopram (30mg) ein, derzeit regelmäßig, im Therapieverlauf fiel ihr die\nregelmäßige Einnahme jedoch teilweise schwer, was wiederholt angesprochen wurde. Frau P. ist Raucherin,\nnimmt keine sonstigen Drogen und trinkt keinen Alkohol.\nTestbefunde:\nDer Wert im Beck-Depressions-Inventar (BDI-II) liegt mit 11 Punkten im subklinischen Bereich. In der\nBorderline-Symptom-Liste (BSL-23) zeigt sich ein Mittelwert von 2,609 (Skala von 0 = Symptom überhaupt\nnicht vorhanden bis 4 = sehr stark), was einem Prozentrang von 68 entspricht.\n\n3. Begründung der Notwendigkeit der Fortführung der Behandlung, weitere Therapieplanung,\ngeänderte/erweiterte Behandlungsziele, geänderte Behandlungsmethoden und -techniken, Prognose, Planung\ndes Therapieabschlusses, ggf. weiterführende Maßnahmen nach Ende der Therapie\nIm Laufe der Behandlung wurde zunehmend deutlich, dass der Partner schwerer Alkoholiker ist, hiermit\nverbundene zeigten sich Konflikte (Aggressivität, tägliches Betrunkensein des Partners, kaum Unterstützung\nbei der Bewältigung des Haushalts und der Pflege der Hündin, Unzuverlässigkeit), welche daraufhin in den im\nMittelpunkt der Therapie rückten. Es fällt Frau P. weiterhin schwer, Wut rechtzeitig wahrzunehmen und diese\ndeutlich auszudrücken, v.a. gegenüber ihrem Partner.\nFrau P. möchte ihre erlernte Fähigkeit, Gefühle und Bedürfnisse zu erkennen und zu benennen beibehalten\nund weiter ausbauen. Zudem soll die bereits verbesserte Konfliktfähigkeit weiter ausgebaut werden und sie\nmöchte sich insgesamt für zukünftige schwierige Situationen stabilisierter fühlen. Diese Ziele werden aus\ntherapeutischer Sicht unterstützt, zudem soll das Thema „Co-Abhängigkeit“ weiterhin bearbeitet, co-\nabhängiges Verhalten soll abgebaut werden, da dies / die Partnerschaftskonflikte als aufrechterhaltend für die\nSymptomatik angesehen werden. Außerdem kann das Verhalten, das Frau P. teilweise von ihrem Partner\n„erduldet“ (Anbrüllen, Nicht-Ernstnehmen ihrer Bedürfnisse, ständige Trunkenheit, Unzuverlässigkeit,\nmangelnde Unterstützung) als verdeckt selbstschädigendes Verhalten interpretiert werden.\nFolgende Strategien sollen zur Erreichung der Ziele eingesetzt werden:\n\uf0b7 Verfestigung und weiterer Ausbau des Selbstfürsorgeverhaltens (regelmäßiges Essen, Einplanen von\nangenehmen, entspannenden Aktivitäten)\n\uf0b7 Ausbau des Freizeitverhaltens (auch unabhängig vom Partner), z.B. Unternehmungen mit\nFreundinnen, Wiederaufnahme der Kontakte zu älteren Freunden, Umsetzung des geplanten Eintritts\nin einen Sportverein\n\uf0b7 Ausbau von Gefühlswahrnehmung, Gefühlsausdruck und Konfliktfähigkeit, z.B. mit Gefühlsprotokollen\nsowie übenden Rollenspielen\n\uf0b7 Förderung des Selbstwerts und Stabilisierung durch Installieren eines „positiven Begleiters“ (nach\nPotreck-Rose, 2010)\n\uf0b7 Weitere Reduktion des co-abhängigen Verhaltens durch Thematisieren und Abbau dazugehöriger\nVerhaltensweisen\n\uf0b7 Bearbeiten von Wünschen und Konflikten in der Partnerschaft z.B. durch Exploration der Wünsche\nund Bedürfnisse sowie Validieren angemessener Wünsche (z.B. nach einem respektvollen Umgang mit\nihrer Person)\n\uf0b7 Ausführliche Rückfallprophylaxe mit besonderem Fokus auf zukünftigen antizipierten Belastungen\n\uf0b7 Langsames Ausschleichen der Therapie, um der Patientin über einen längeren Zeitraum Halt zu bieten\nAufgrund der bisherigen Behandlungserfolge sowie der guten therapeutischen Beziehung wird die Prognose\nals günstig beurteilt. Besonderen Wert muss allerdings auf die ausreichende Verfestigung der erlernten\nStrategien sowie auf die Rückfallprophylaxe gelegt werden, damit die Patientin für zukünftige Belastungen\n(z.B. eventuelle Trennung vom alkoholkranken Partner, eventueller Arbeitsplatzverlust) stabilisiert ist.\nEs wird um eine Therapieverlängerung von 20 Stunden gebeten. Die Therapiefrequenz wird zunächst noch\nwöchentlich sein, dann vierzehntägig bzw. monatlich. Danach soll die Therapie ihren Abschluss finden, sofern\nkeine akuten Krisen (wie z.B. Trennung vom Partner) auftreten.\n\n2 Bitte extrahiere Metadaten aus dem bereitgestellten Dokument und gebe diese im JSON-Format zurück. Ich benötige die Felder Datum_Empfang,Titel,Zusammenfassung,Kategorie. Analysiere das Dokument, um in deutscher Sprache die Werte für diese Felder zu finden, und formatiere die Antwort als JSON-Objekt. Verwende für jedes Feld die wahrscheinlichste Antwort. Die Antwort sollte nur JSON-Daten enthalten, bei denen die Schlüssel und Werte alle im einfachen String-Format (kein verschachteltes Objekt) vorliegen, damit sie von einem anderen Programm direkt geparst werden können. Die Antwort sollte mit geschweiften Klammern, die das JSON-Objekt abgrenzen, beginnen und enden [/INST]"
time=2024-05-16T14:29:43.036Z level=DEBUG source=server.go:632 msg="setting token limit to 10x num_ctx" num_ctx=30196 num_predict=301960
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=158 tid="125100883189760" timestamp=1715869783
DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=159 tid="125100883189760" timestamp=1715869783
DEBUG [update_slots] slot progression | ga_i=0 n_past=6 n_past_se=0 n_prompt_tokens_processed=3308 slot_id=0 task_id=159 tid="125100883189760" timestamp=1715869783
DEBUG [update_slots] kv cache rm [p0, end) | p0=6 slot_id=0 task_id=159 tid="125100883189760" timestamp=1715869783
DEBUG [print_timings] prompt eval time = 8451.16 ms / 3308 tokens ( 2.55 ms per token, 391.43 tokens per second) | n_prompt_tokens_processed=3308 n_tokens_second=391.4256947303192 slot_id=0 t_prompt_processing=8451.157 t_token=2.5547633010882707 task_id=159 tid="125100883189760" timestamp=1715869828
DEBUG [print_timings] generation eval time = 37026.26 ms / 249 runs ( 148.70 ms per token, 6.72 tokens per second) | n_decoded=249 n_tokens_second=6.724956827937794 slot_id=0 t_token=148.69983935742974 t_token_generation=37026.26 task_id=159 tid="125100883189760" timestamp=1715869828
DEBUG [print_timings] total time = 45477.42 ms | slot_id=0 t_prompt_processing=8451.157 t_token_generation=37026.26 t_total=45477.417 task_id=159 tid="125100883189760" timestamp=1715869828
DEBUG [update_slots] slot released | n_cache_tokens=3563 n_ctx=30208 n_past=3562 n_system_tokens=0 slot_id=0 task_id=159 tid="125100883189760" timestamp=1715869828 truncated=false
DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=37266 status=200 tid="125100252725248" timestamp=1715869828
DEBUG [process_single_task] slot data | n_idle_slots=1 n_processing_slots=0 task_id=411 tid="125100883189760" timestamp=1715869828
DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=56858 status=200 tid="125100242239488" timestamp=1715869828
[GIN] 2024/05/16 - 14:30:28 | 200 | 45.524554207s | 172.16.167.1 | POST "/api/generate"
time=2024-05-16T14:30:28.559Z level=DEBUG source=sched.go:304 msg="context for request finished"
time=2024-05-16T14:30:28.559Z level=DEBUG source=sched.go:237 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364acadcf08458f8 duration=5m0s
time=2024-05-16T14:30:28.559Z level=DEBUG source=sched.go:255 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-3a17f7cde150070bbc815645693fb93c311cc42e7deaf198364paperless
[2024-05-16 16:24:54,594] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[a42856da-7b85-490a-bbd8-0bab2a5505ab] succeeded in 0.06093939603306353s: None
[2024-05-16 16:25:16,745] [INFO] [celery.worker.strategy] Task documents.tasks.bulk_update_documents[6fca90b8-1c05-42ec-9ddb-19e5b2bf34b5] received
[2024-05-16 16:25:19,321] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[6fca90b8-1c05-42ec-9ddb-19e5b2bf34b5] succeeded in 2.5741557349683717s: None
[2024-05-16 16:25:53,117] [INFO] [celery.worker.strategy] Task documents.tasks.bulk_update_documents[b016788a-5083-4d92-9c2e-d3341d020682] received
[2024-05-16 16:25:53,197] [INFO] [celery.app.trace] Task documents.tasks.bulk_update_documents[b016788a-5083-4d92-9c2e-d3341d020682] succeeded in 0.07842971791978925s: None
[2024-05-16 16:30:00,001] [INFO] [celery.beat] Scheduler: Sending due task Check all e-mail accounts (paperless_mail.tasks.process_mail_accounts)
[2024-05-16 16:30:00,004] [INFO] [celery.worker.strategy] Task paperless_mail.tasks.process_mail_accounts[9e0890c6-1e85-44df-a462-a2869ba05b7a] receivedstill no logging for response, but at least the main problem seems isolated. will test more later and update
Deathproof76
commented
6 months ago It worked for about 30 documents and then the container shut down coinciding with [2024-05-16 16:56:33,797] [WARNING] [django.request] Bad Request: /api/documents/227/ This specific document doesn't seem to have any special properties. Of the 30 documents about 4 got their custom fiels updated with the responses, the others stayed empty.
doclytics | May 16 14:29:18.363 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 14:29:18.394 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 14:29:18.394 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:18.394 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:18.394 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:18.405 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.405 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 14:29:18.405 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 14:29:18.405 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.660 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:18.660 INFO Successfully retrieved 25 Documents
doclytics | May 16 14:29:18.660 INFO Generate Response with LLM model
doclytics | May 16 14:29:18.660 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 14:29:18.660 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 14:29:18.661 DEBG connected to 192.168.0.208:11434
doclytics | May 16 14:29:34.800 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:34.800 INFO Updating document with ID: 264
doclytics | May 16 14:29:34.800 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:34.800 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:34.800 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:34.867 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:34.867 INFO Document with ID: 264 successfully updated
doclytics | May 16 14:29:34.867 INFO Generate Response with LLM model
doclytics | May 16 14:29:34.867 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:42.956 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:29:42.956 INFO Updating document with ID: 263
doclytics | May 16 14:29:42.956 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:29:42.956 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:29:42.956 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:29:43.035 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:29:43.035 INFO Document with ID: 263 successfully updated
doclytics | May 16 14:29:43.035 INFO Generate Response with LLM model
doclytics | May 16 14:29:43.035 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:28.559 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:28.560 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:30:28.560 INFO Generate Response with LLM model
doclytics | May 16 14:30:28.560 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:40.660 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:30:40.660 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:30:40.660 INFO Generate Response with LLM model
doclytics | May 16 14:30:40.660 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:31:06.864 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:31:06.864 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:31:06.864 INFO Generate Response with LLM model
doclytics | May 16 14:31:06.864 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics exited with code 0
doclytics | May 16 14:47:45.224 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 14:47:45.250 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 14:47:45.250 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:47:45.251 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:47:45.251 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:47:45.264 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:47:45.264 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 14:47:45.264 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 14:47:45.264 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:47:45.423 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:47:45.423 INFO Successfully retrieved 25 Documents
doclytics | May 16 14:47:45.423 INFO Generate Response with LLM model
doclytics | May 16 14:47:45.423 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 14:47:45.423 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 14:47:45.423 DEBG connected to 192.168.0.208:11434
doclytics | May 16 14:48:02.608 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:48:02.609 INFO Updating document with ID: 263
doclytics | May 16 14:48:02.609 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:48:02.609 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:48:02.609 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:48:02.706 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:48:02.706 INFO Document with ID: 263 successfully updated
doclytics | May 16 14:48:02.706 INFO Generate Response with LLM model
doclytics | May 16 14:48:02.706 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:48:13.688 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:48:13.688 INFO Updating document with ID: 264
doclytics | May 16 14:48:13.688 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:48:13.688 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:48:13.688 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:48:13.798 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:48:13.798 INFO Document with ID: 264 successfully updated
doclytics | May 16 14:48:13.798 INFO Generate Response with LLM model
doclytics | May 16 14:48:13.798 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:48:27.212 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:48:27.212 INFO Updating document with ID: 262
doclytics | May 16 14:48:27.212 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:48:27.212 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:48:27.212 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:48:27.298 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:48:27.298 INFO Document with ID: 262 successfully updated
doclytics | May 16 14:48:27.298 INFO Generate Response with LLM model
doclytics | May 16 14:48:27.298 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:49:13.659 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:49:13.659 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:49:13.660 INFO Generate Response with LLM model
doclytics | May 16 14:49:13.660 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:49:44.644 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:49:44.644 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:49:44.644 INFO Generate Response with LLM model
doclytics | May 16 14:49:44.644 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:04.132 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:04.132 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:50:04.132 INFO Generate Response with LLM model
doclytics | May 16 14:50:04.132 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:13.672 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:13.672 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:50:13.672 INFO Generate Response with LLM model
doclytics | May 16 14:50:13.672 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:49.556 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:50:49.556 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:50:49.556 INFO Generate Response with LLM model
doclytics | May 16 14:50:49.556 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:04.660 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:04.660 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:51:04.660 INFO Generate Response with LLM model
doclytics | May 16 14:51:04.660 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:30.564 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:30.564 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:51:30.564 INFO Generate Response with LLM model
doclytics | May 16 14:51:30.564 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:46.280 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:51:46.280 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:51:46.280 INFO Generate Response with LLM model
doclytics | May 16 14:51:46.280 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:01.660 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:01.660 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:52:01.660 INFO Generate Response with LLM model
doclytics | May 16 14:52:01.660 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:23.656 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:23.656 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:52:23.656 INFO Generate Response with LLM model
doclytics | May 16 14:52:23.656 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:50.510 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:52:50.510 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:52:50.510 INFO Generate Response with LLM model
doclytics | May 16 14:52:50.510 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:53:13.272 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:53:13.272 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:53:13.272 INFO Generate Response with LLM model
doclytics | May 16 14:53:13.272 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:06.201 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:06.201 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:54:06.201 INFO Generate Response with LLM model
doclytics | May 16 14:54:06.201 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:44.188 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:44.189 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:54:44.189 INFO Generate Response with LLM model
doclytics | May 16 14:54:44.189 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:59.772 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:54:59.772 INFO Updating document with ID: 222
doclytics | May 16 14:54:59.772 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:54:59.772 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:54:59.772 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:54:59.851 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:54:59.851 INFO Document with ID: 222 successfully updated
doclytics | May 16 14:54:59.851 INFO Generate Response with LLM model
doclytics | May 16 14:54:59.851 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:55:19.008 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:55:19.008 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:55:19.008 INFO Generate Response with LLM model
doclytics | May 16 14:55:19.008 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:55:54.033 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:55:54.033 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:55:54.033 INFO Generate Response with LLM model
doclytics | May 16 14:55:54.033 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:56:15.904 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:56:15.904 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 14:56:15.904 INFO Generate Response with LLM model
doclytics | May 16 14:56:15.904 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:56:33.776 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 14:56:33.776 INFO Updating document with ID: 227
doclytics | May 16 14:56:33.776 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 14:56:33.776 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 14:56:33.776 DEBG connected to 192.168.0.208:17959
doclytics | May 16 14:56:33.797 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 14:56:33.797 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/227/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/227/", query: None, fragment: None } }
doclytics exited with code 0I've tried using llama-3-instruct and it got a lot further and tagged a lot more documents, albeit I had to restart a few times (because when doclytics exits it keeps running in the background, just doesn't log anymore). So I'm pretty certain that it does seem to be a model problem in the end. A lot of my documents have more than 3 pages with full text which may compound the problem.
doclytics | May 16 15:12:48.223 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 15:12:48.249 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 15:12:48.249 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:12:48.249 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:12:48.249 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:12:48.261 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:12:48.261 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 15:12:48.261 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 15:12:48.261 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:12:48.403 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:12:48.403 INFO Successfully retrieved 25 Documents
doclytics | May 16 15:12:48.403 INFO Generate Response with LLM model
doclytics | May 16 15:12:48.403 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 15:12:48.403 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 15:12:48.403 DEBG connected to 192.168.0.208:11434
doclytics | May 16 15:13:35.297 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:13:35.298 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:13:35.298 INFO Generate Response with LLM model
doclytics | May 16 15:13:35.298 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:09.769 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:09.769 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:14:09.769 INFO Generate Response with LLM model
doclytics | May 16 15:14:09.769 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:23.368 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:23.368 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:14:23.368 INFO Generate Response with LLM model
doclytics | May 16 15:14:23.368 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:35.688 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:35.688 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:14:35.688 INFO Generate Response with LLM model
doclytics | May 16 15:14:35.688 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:59.832 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:14:59.832 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:14:59.832 INFO Generate Response with LLM model
doclytics | May 16 15:14:59.832 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:15:13.701 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:15:13.701 ERRO Error parsing llm response json invalid escape at line 2 column 8
doclytics | May 16 15:15:13.701 INFO Generate Response with LLM model
doclytics | May 16 15:15:13.701 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:15:31.088 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:15:31.088 INFO Updating document with ID: 211
doclytics | May 16 15:15:31.088 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:15:31.088 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:15:31.088 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:15:31.113 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:15:31.113 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/211/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/211/", query: None, fragment: None } }
doclytics exited with code 0
doclytics | May 16 15:17:47.434 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 15:17:47.464 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 15:17:47.464 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:17:47.464 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:17:47.464 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:17:47.475 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:17:47.475 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 15:17:47.475 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 15:17:47.475 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:17:47.630 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:17:47.630 INFO Successfully retrieved 25 Documents
doclytics | May 16 15:17:47.630 INFO Generate Response with LLM model
doclytics | May 16 15:17:47.630 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 15:17:47.630 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 15:17:47.631 DEBG connected to 192.168.0.208:11434
doclytics | May 16 15:17:56.473 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:17:56.473 INFO Updating document with ID: 261
doclytics | May 16 15:17:56.473 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:17:56.473 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:17:56.473 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:17:56.492 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:17:56.492 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/261/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/261/", query: None, fragment: None } }
doclytics exited with code 0
doclytics | May 16 15:18:23.842 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 15:18:23.867 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 15:18:23.867 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:18:23.867 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:18:23.867 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:18:23.876 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:23.876 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 15:18:23.876 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 15:18:23.876 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:24.164 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:24.164 INFO Successfully retrieved 25 Documents
doclytics | May 16 15:18:24.164 INFO Generate Response with LLM model
doclytics | May 16 15:18:24.164 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 15:18:24.164 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 15:18:24.164 DEBG connected to 192.168.0.208:11434
doclytics | May 16 15:18:26.348 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:26.348 INFO Updating document with ID: 203
doclytics | May 16 15:18:26.348 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:18:26.348 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:18:26.348 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:18:26.418 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:26.418 INFO Document with ID: 203 successfully updated
doclytics | May 16 15:18:26.418 INFO Generate Response with LLM model
doclytics | May 16 15:18:26.418 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:31.337 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:31.337 INFO Updating document with ID: 202
doclytics | May 16 15:18:31.337 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:18:31.337 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:18:31.337 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:18:31.478 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:31.478 INFO Document with ID: 202 successfully updated
doclytics | May 16 15:18:31.478 INFO Generate Response with LLM model
doclytics | May 16 15:18:31.478 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:33.144 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:33.144 INFO Updating document with ID: 201
doclytics | May 16 15:18:33.144 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:33.211 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:33.211 INFO Document with ID: 201 successfully updated
doclytics | May 16 15:18:33.211 INFO Generate Response with LLM model
doclytics | May 16 15:18:33.211 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:36.653 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:36.653 INFO Updating document with ID: 200
doclytics | May 16 15:18:36.653 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:18:36.653 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:18:36.653 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:18:36.737 INFO Document with ID: 200 successfully updated
doclytics | May 16 15:18:36.737 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:36.737 INFO Generate Response with LLM model
doclytics | May 16 15:18:36.737 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:39.688 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:18:39.688 INFO Updating document with ID: 199
doclytics | May 16 15:18:39.688 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:18:39.688 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:18:39.689 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:18:39.707 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:18:39.707 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/199/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/199/", query: None, fragment: None } }
doclytics exited with code 0
doclytics | May 16 15:19:26.063 INFO Application started, version: 1.1.4-rc.9
doclytics | May 16 15:19:26.087 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
doclytics | May 16 15:19:26.088 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:19:26.088 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:19:26.088 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:19:26.098 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:26.098 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
doclytics | May 16 15:19:26.098 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
doclytics | May 16 15:19:26.098 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:26.366 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:26.367 INFO Successfully retrieved 25 Documents
doclytics | May 16 15:19:26.367 INFO Generate Response with LLM model
doclytics | May 16 15:19:26.367 DEBG starting new connection: http://192.168.0.208:11434/
doclytics | May 16 15:19:26.367 DEBG connecting to 192.168.0.208:11434
doclytics | May 16 15:19:26.367 DEBG connected to 192.168.0.208:11434
doclytics | May 16 15:19:27.852 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:27.852 INFO Updating document with ID: 204
doclytics | May 16 15:19:27.852 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:27.926 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:27.926 INFO Document with ID: 204 successfully updated
doclytics | May 16 15:19:27.926 INFO Generate Response with LLM model
doclytics | May 16 15:19:27.926 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:31.132 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:31.132 INFO Updating document with ID: 188
doclytics | May 16 15:19:31.132 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:19:31.132 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:19:31.133 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:19:31.201 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:31.201 INFO Document with ID: 188 successfully updated
doclytics | May 16 15:19:31.201 INFO Generate Response with LLM model
doclytics | May 16 15:19:31.201 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:31.949 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:31.949 ERRO No JSON object found in the response!
doclytics | May 16 15:19:31.949 INFO Generate Response with LLM model
doclytics | May 16 15:19:31.949 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:34.041 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:34.041 INFO Updating document with ID: 197
doclytics | May 16 15:19:34.041 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:19:34.041 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:19:34.041 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:19:34.229 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:34.229 INFO Document with ID: 197 successfully updated
doclytics | May 16 15:19:34.229 INFO Generate Response with LLM model
doclytics | May 16 15:19:34.229 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:35.124 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:35.124 INFO Updating document with ID: 228
doclytics | May 16 15:19:35.124 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:35.211 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:35.211 INFO Document with ID: 228 successfully updated
doclytics | May 16 15:19:35.211 INFO Generate Response with LLM model
doclytics | May 16 15:19:35.211 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:37.044 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:37.044 INFO Updating document with ID: 186
doclytics | May 16 15:19:37.044 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:37.115 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:37.115 INFO Document with ID: 186 successfully updated
doclytics | May 16 15:19:37.115 INFO Generate Response with LLM model
doclytics | May 16 15:19:37.115 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:40.852 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
doclytics | May 16 15:19:40.853 INFO Updating document with ID: 185
doclytics | May 16 15:19:40.853 DEBG starting new connection: http://192.168.0.208:17959/
doclytics | May 16 15:19:40.853 DEBG connecting to 192.168.0.208:17959
doclytics | May 16 15:19:40.853 DEBG connected to 192.168.0.208:17959
doclytics | May 16 15:19:40.873 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
doclytics | May 16 15:19:40.873 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/185/)
doclytics | Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/185/", query: None, fragment: None } }
doclytics exited with code 0update: doclytics ran through all the documents, when it exited it just restarted. the status just wasn't updated in https://github.com/louislam/dockge and the logging stopped there. Portainer showed correct logs:
May 16 15:38:19.352 INFO Document with ID: 83 successfully updated
May 16 15:38:19.352 INFO Generate Response with LLM model
May 16 15:38:19.352 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:22.620 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:22.620 INFO Updating document with ID: 178
May 16 15:38:22.620 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:22.620 DEBG connecting to 192.168.0.208:17959
May 16 15:38:22.621 DEBG connected to 192.168.0.208:17959
May 16 15:38:22.645 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:22.645 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/178/)
Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/178/", query: None, fragment: None } }
May 16 15:38:23.092 INFO Application started, version: 1.1.4-rc.9
May 16 15:38:23.116 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:38:23.116 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:23.116 DEBG connecting to 192.168.0.208:17959
May 16 15:38:23.116 DEBG connected to 192.168.0.208:17959
May 16 15:38:23.128 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:23.128 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:38:23.128 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:38:23.128 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:23.412 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:23.412 INFO Successfully retrieved 25 Documents
May 16 15:38:23.412 INFO Generate Response with LLM model
May 16 15:38:23.412 DEBG starting new connection: http://192.168.0.208:11434/
May 16 15:38:23.412 DEBG connecting to 192.168.0.208:11434
May 16 15:38:23.412 DEBG connected to 192.168.0.208:11434
May 16 15:38:26.124 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:26.124 INFO Updating document with ID: 205
May 16 15:38:26.124 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:26.124 DEBG connecting to 192.168.0.208:17959
May 16 15:38:26.124 DEBG connected to 192.168.0.208:17959
May 16 15:38:26.438 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:26.438 INFO Document with ID: 205 successfully updated
May 16 15:38:26.438 INFO Generate Response with LLM model
May 16 15:38:26.438 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:27.236 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:27.236 ERRO No JSON object found in the response!
May 16 15:38:27.236 INFO Generate Response with LLM model
May 16 15:38:27.236 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:30.560 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:30.560 INFO Updating document with ID: 178
May 16 15:38:30.560 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:30.560 DEBG connecting to 192.168.0.208:17959
May 16 15:38:30.560 DEBG connected to 192.168.0.208:17959
May 16 15:38:30.632 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:30.632 INFO Document with ID: 178 successfully updated
May 16 15:38:30.632 INFO Generate Response with LLM model
May 16 15:38:30.632 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:34.236 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:34.236 INFO Updating document with ID: 243
May 16 15:38:34.236 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:34.236 DEBG connecting to 192.168.0.208:17959
May 16 15:38:34.237 DEBG connected to 192.168.0.208:17959
May 16 15:38:34.255 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:34.255 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/243/)
Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/243/", query: None, fragment: None } }
May 16 15:38:34.652 INFO Application started, version: 1.1.4-rc.9
May 16 15:38:34.676 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:38:34.676 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:34.676 DEBG connecting to 192.168.0.208:17959
May 16 15:38:34.676 DEBG connected to 192.168.0.208:17959
May 16 15:38:34.686 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:34.686 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:38:34.686 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:38:34.686 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:34.954 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:34.954 INFO Successfully retrieved 25 Documents
May 16 15:38:34.954 INFO Generate Response with LLM model
May 16 15:38:34.954 DEBG starting new connection: http://192.168.0.208:11434/
May 16 15:38:34.954 DEBG connecting to 192.168.0.208:11434
May 16 15:38:34.954 DEBG connected to 192.168.0.208:11434
May 16 15:38:36.876 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:36.876 INFO Updating document with ID: 205
May 16 15:38:36.876 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:36.958 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:36.958 INFO Document with ID: 205 successfully updated
May 16 15:38:36.958 INFO Generate Response with LLM model
May 16 15:38:36.958 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:38.205 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:38.205 ERRO No JSON object found in the response!
May 16 15:38:38.205 INFO Generate Response with LLM model
May 16 15:38:38.205 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:41.420 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:41.421 INFO Updating document with ID: 243
May 16 15:38:41.421 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:41.421 DEBG connecting to 192.168.0.208:17959
May 16 15:38:41.421 DEBG connected to 192.168.0.208:17959
May 16 15:38:41.484 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:41.484 INFO Document with ID: 243 successfully updated
May 16 15:38:41.484 INFO Generate Response with LLM model
May 16 15:38:41.484 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:44.304 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:44.304 INFO Updating document with ID: 177
May 16 15:38:44.304 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:44.304 DEBG connecting to 192.168.0.208:17959
May 16 15:38:44.305 DEBG connected to 192.168.0.208:17959
May 16 15:38:44.399 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:44.399 INFO Document with ID: 177 successfully updated
May 16 15:38:44.399 INFO Generate Response with LLM model
May 16 15:38:44.399 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:53.007 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:53.008 INFO Updating document with ID: 255
May 16 15:38:53.008 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:53.008 DEBG connecting to 192.168.0.208:17959
May 16 15:38:53.008 DEBG connected to 192.168.0.208:17959
May 16 15:38:53.295 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:53.295 INFO Document with ID: 255 successfully updated
May 16 15:38:53.295 INFO Generate Response with LLM model
May 16 15:38:53.295 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:55.688 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:55.688 INFO Updating document with ID: 80
May 16 15:38:55.688 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:38:55.688 DEBG connecting to 192.168.0.208:17959
May 16 15:38:55.688 DEBG connected to 192.168.0.208:17959
May 16 15:38:55.757 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:55.757 INFO Document with ID: 80 successfully updated
May 16 15:38:55.757 INFO Generate Response with LLM model
May 16 15:38:55.757 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:57.684 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:57.684 INFO Updating document with ID: 76
May 16 15:38:57.684 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:57.762 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:57.762 INFO Document with ID: 76 successfully updated
May 16 15:38:57.762 INFO Generate Response with LLM model
May 16 15:38:57.762 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:59.184 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:38:59.184 INFO Updating document with ID: 75
May 16 15:38:59.184 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:59.253 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:38:59.253 INFO Document with ID: 75 successfully updated
May 16 15:38:59.253 INFO Generate Response with LLM model
May 16 15:38:59.253 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:02.124 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:02.124 INFO Updating document with ID: 74
May 16 15:39:02.124 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:02.124 DEBG connecting to 192.168.0.208:17959
May 16 15:39:02.125 DEBG connected to 192.168.0.208:17959
May 16 15:39:02.190 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:02.190 INFO Document with ID: 74 successfully updated
May 16 15:39:02.190 INFO Generate Response with LLM model
May 16 15:39:02.190 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:08.861 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:08.861 INFO Updating document with ID: 232
May 16 15:39:08.861 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:08.861 DEBG connecting to 192.168.0.208:17959
May 16 15:39:08.861 DEBG connected to 192.168.0.208:17959
May 16 15:39:09.185 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:09.185 INFO Document with ID: 232 successfully updated
May 16 15:39:09.185 INFO Generate Response with LLM model
May 16 15:39:09.185 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:11.620 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:11.620 INFO Updating document with ID: 238
May 16 15:39:11.620 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:11.620 DEBG connecting to 192.168.0.208:17959
May 16 15:39:11.621 DEBG connected to 192.168.0.208:17959
May 16 15:39:11.713 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:11.713 INFO Document with ID: 238 successfully updated
May 16 15:39:11.713 INFO Generate Response with LLM model
May 16 15:39:11.713 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:14.636 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:14.636 INFO Updating document with ID: 77
May 16 15:39:14.636 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:14.636 DEBG connecting to 192.168.0.208:17959
May 16 15:39:14.637 DEBG connected to 192.168.0.208:17959
May 16 15:39:14.733 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:14.733 INFO Document with ID: 77 successfully updated
May 16 15:39:14.733 INFO Generate Response with LLM model
May 16 15:39:14.733 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:16.528 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:16.528 INFO Updating document with ID: 236
May 16 15:39:16.528 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:16.604 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:16.604 INFO Document with ID: 236 successfully updated
May 16 15:39:16.604 INFO Generate Response with LLM model
May 16 15:39:16.604 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:18.972 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:18.972 INFO Updating document with ID: 252
May 16 15:39:18.972 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:18.972 DEBG connecting to 192.168.0.208:17959
May 16 15:39:18.973 DEBG connected to 192.168.0.208:17959
May 16 15:39:19.046 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:19.046 INFO Document with ID: 252 successfully updated
May 16 15:39:19.046 INFO Generate Response with LLM model
May 16 15:39:19.046 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:21.400 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:21.400 INFO Updating document with ID: 237
May 16 15:39:21.400 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:21.400 DEBG connecting to 192.168.0.208:17959
May 16 15:39:21.400 DEBG connected to 192.168.0.208:17959
May 16 15:39:21.704 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:21.704 INFO Document with ID: 237 successfully updated
May 16 15:39:21.704 INFO Generate Response with LLM model
May 16 15:39:21.704 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:23.824 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:23.824 INFO Updating document with ID: 239
May 16 15:39:23.824 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:23.824 DEBG connecting to 192.168.0.208:17959
May 16 15:39:23.824 DEBG connected to 192.168.0.208:17959
May 16 15:39:23.998 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:23.998 INFO Document with ID: 239 successfully updated
May 16 15:39:23.998 INFO Generate Response with LLM model
May 16 15:39:23.998 DEBG reuse idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:26.856 DEBG pooling idle connection for ("http", 192.168.0.208:11434)
May 16 15:39:26.856 INFO Updating document with ID: 215
May 16 15:39:26.856 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:26.856 DEBG connecting to 192.168.0.208:17959
May 16 15:39:26.856 DEBG connected to 192.168.0.208:17959
May 16 15:39:26.874 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:26.875 ERRO Error while updating document fields: HTTP status client error (400 Bad Request) for url (http://192.168.0.208:17959/api/documents/215/)
Error: reqwest::Error { kind: Status(400), url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(192.168.0.208)), port: Some(17959), path: "/api/documents/215/", query: None, fragment: None } }
May 16 15:39:27.267 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:27.292 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:27.292 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:27.292 DEBG connecting to 192.168.0.208:17959
May 16 15:39:27.292 DEBG connected to 192.168.0.208:17959
May 16 15:39:27.302 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:27.302 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:27.302 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:27.302 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:27.574 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:27.574 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:27.574 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:27.574 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:28.013 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:28.037 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:28.037 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:28.037 DEBG connecting to 192.168.0.208:17959
May 16 15:39:28.037 DEBG connected to 192.168.0.208:17959
May 16 15:39:28.047 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:28.047 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:28.047 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:28.047 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:28.317 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:28.317 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:28.317 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:28.317 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:28.970 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:28.994 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:28.994 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:28.994 DEBG connecting to 192.168.0.208:17959
May 16 15:39:28.994 DEBG connected to 192.168.0.208:17959
May 16 15:39:29.006 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:29.006 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:29.006 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:29.006 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:29.270 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:29.271 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:29.271 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:29.271 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:30.341 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:30.367 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:30.367 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:30.367 DEBG connecting to 192.168.0.208:17959
May 16 15:39:30.367 DEBG connected to 192.168.0.208:17959
May 16 15:39:30.379 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:30.379 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:30.379 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:30.379 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:30.644 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:30.645 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:30.645 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:30.645 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:32.470 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:32.495 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:32.495 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:32.495 DEBG connecting to 192.168.0.208:17959
May 16 15:39:32.495 DEBG connected to 192.168.0.208:17959
May 16 15:39:32.506 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:32.506 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:32.506 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:32.506 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:32.777 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:32.778 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:32.778 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:32.778 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:36.227 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:36.252 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:36.252 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:36.252 DEBG connecting to 192.168.0.208:17959
May 16 15:39:36.252 DEBG connected to 192.168.0.208:17959
May 16 15:39:36.262 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:36.262 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:36.262 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:36.262 DEBG reuse idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:36.543 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:36.543 ERRO Error while creating json of document response from paperless invalid type: null, expected a string at line 1 column 46077
May 16 15:39:36.543 ERRO Error at column 46077: null,"original_file_name":null,"archived_file_name":"2022-09
May 16 15:39:36.544 ERRO Error while interacting with paperless: invalid type: null, expected a string at line 1 column 46077
May 16 15:39:43.170 INFO Application started, version: 1.1.4-rc.9
May 16 15:39:43.196 INFO Fetching custom fields from paperless at http://192.168.0.208:17959
May 16 15:39:43.196 DEBG starting new connection: http://192.168.0.208:17959/
May 16 15:39:43.196 DEBG connecting to 192.168.0.208:17959
May 16 15:39:43.196 DEBG connected to 192.168.0.208:17959
May 16 15:39:43.206 DEBG pooling idle connection for ("http", 192.168.0.208:17959)
May 16 15:39:43.206 INFO Fields: [Field { id: 50, name: "Datum_Empfang", data_type: "string" }, Field { id: 49, name: "Absender", data_type: "string" }, Field { id: 48, name: "Titel", data_type: "string" }, Field { id: 47, name: "Zusammenfassung", data_type: "string" }, Field { id: 46, name: "tagged", data_type: "boolean" }]
May 16 15:39:43.206 INFO Retrieve Documents from paperless at: http://192.168.0.208:17959, with query: NOT tagged=true
May 16 15:39:43.206 DEBG reuse idle connection for ("http", 192.168.0.208:17959)So yeah, the logging problem is still there. But in the end, I'm confident that the main problem wasn't a bug of doclytics, but a compounded problem of model, modelfile prompt template, document complexity, most likely an actual shitty json output due to that, which offended Django and my confusion regarding the exit status of doclytics because of https://github.com/louislam/dockge just stopping the log upon a container restart not initiated by the user.
@B-urb Looked a bit in your code and wasn't sure. Not a programmer so I'm guessing you already know about this https://github.com/ollama/ollama/blob/main/docs/api.md#request-json-mode
B-urb
commented
5 months ago @Deathproof76 I actually didn't yet know about this. Great find. I will look into this.
I'm not sure why, but it stopped working again. Worked through 30 documents, I then changed the Baseprompt a little and downed, pulled the latest develop. maybe related to the newest commit?
tried to tweak and fix via PAPERLESS_ALLOWED_HOSTS PAPERLESS_CORS_ALLOWED_HOSTS PAPERLESS_CSRF_TRUSTED_ORIGINS etc because of the bad request warnings
but to no avail. I'm able to login and change the values manually in Browser via http://192.168.0.208:17959. Also tried to use the https:/paperless.lalal.com which is assigned as PAPERLESS_URL with the same problem
update: same problem with bjoern5urban/doclytics:v1.1.4-rc.1 and v1.1.4-rc.2 no idea. Maybe I messed something up
edit: this is part of my env for paperless
tried it directly and via reverse proxy with caddy
also tried removing all custom fields
maybe still related to custom fields?
Fields: [Field { id: 8, name: "Zusammenfassung", data_type: "string" }, Field { id: 7, name: "Datum_Empfang", data_type: "string" }, Field { id: 6, name: "Kategorie", data_type: "string" }, Field { id: 4, name: "Absender", data_type: "string" }, Field { id: 3, name: "Thema", data_type: "string" }, Field { id: 2, name: "Titel", data_type: "string" }, Field { id: 1, name: "tagged", data_type: "boolean" }]I removed one I didn't need and "Field { id: 5..." is missing because of that. Maybe Django gets mad because it receives input it doesn't like 🤷♂️Originally posted by @Deathproof76 in https://github.com/B-urb/doclytics/issues/36#issuecomment-2107983939