search

eikek / docspell

Assist in organizing your piles of documents, resulting from scanners, e-mails and other sources with miminal effort.
https://docspell.org
GNU Affero General Public License v3.0
1.65k stars 127 forks source link

Multi-document upload with filename(s) containing Umlauts stuck #2842

Open nekrondev opened 3 weeks ago

nekrondev commented 3 weeks ago

Environment

Docspell: v0.42.0 Joex: using docker image found at ghcr.io/docspell/joex:latest (i.e. Debian-based image with fixed Tesseract)

Issue

Today I uploaded a multi-document ZIP archive into Docspell using the manual document upload feature, but document processing got stuck.

The contents found inside the ZIP archive was:

grafik

I uploaded the archive using the following manual upload settings:

grafik

The processing of the filename containing umlauts (ü) crashes processing of the archive. The error message I got from the job queue was Malformed input or input contains unmappable characters: /tmp/docspell-zip-9930113460477770389/123456_2024_Anpassung der Ausführungsfristen bei Echtzeitüberweisungen_vom_2024.11.01_20241101101038.pdf.

grafik

(Note: Inside the screenshots I only made some private number at the beginning of the PDF document unrecognizable but you can replace that with a six digit random number)

I'm not sure if this issue may be present in earlier versions of Docspell, because it was the first time my bank send me a document containing umlauts.

Workaround

I've uploaded the plain document and this seems to be fine.

Testdata

Here is a test archive containing only one PDF filename with umlauts that had been zipped with Windows FileExplorer. However, this time Docspell tells me the error was invalid CEN header (bad entry name or comment). Zipping the file with 7Zip returns the same error message so somehow the processing of ZIP file streams with UTF-8 chars seems to be broken.

grafik

testdata.zip

nekrondev commented 3 weeks ago

Ok, found a working solution for UTF-8 encoded ZIP archive filenames not processed by the Ubuntu 24.04 based image. The docker-compose.yml file must contain an environment setting to tell java runtime that it should use UTF-8 instead of some ANSI collation.

So you need to add this to joex and restapi-server service:

environment:
      - LANG=C.utf8

Uploading the archive now does a correct processing of the filename as UTF-8. However, the CEN header error still exists if I upload my 7Zip or Window FileExplorer created archive but that might be some Windows ZIP/7Zip related issue as some other test archive I created was processed ok.