search

crawford / doxie-upload

Simple HTTP server for accepting scans from the Doxie scanner webhook
GNU Affero General Public License v3.0
2 stars 1 forks source link

Issues getting it working with Doxie Q #10

Closed ffleischer closed 3 years ago

ffleischer commented 3 years ago

Hi @crawford, I've got some issues getting it to work with my Doxie Q.

After switching to rust:1.42 due following build error

error[E0658]: use of unstable library feature 'matches_macro'
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tracing-core-0.1.18/src/event.rs:115:9
    |
115 |         matches!(self.parent, Parent::Current)
    |         ^^^^^^^
    |
    = note: for more information, see https://github.com/rust-lang/rust/issues/65721

I was able to build the image with following command docker build -t doxie-upload . The run command is docker run -v /home/fabian/git/paperless-ng/paperless-ng/consume/:/uploads -p 8080:8080/tcp doxie-upload and I'm able to see log entries if I try to connect via webserver [INFO doxie_upload] Request from 192.168.178.125:58836

But my Doxie wont connect to the server. Here is my config:

URL: http://<sever_ip>:8080
Username (optional) : not set
Password (optional) : not set
Upload New Scans via Webhook: Always
File Parameter Name: not set 
Optional Parameters:
  - tags = Doxie

I'll always receive a Gethostname Error. in the status-webhooklog ....

Is there anything I'm missing?

crawford commented 3 years ago

Make sure you are sitting down for this.

It is because the scanner interprets the port number in your URL as being part of the domain. You must use port 80 and omit it from the URL.

Here was my message to support and their response:

I'm trying to use the webhook feature but the scanner is incorrectly parsing the provided URL. It's interpreting the port number in the URL as if it were a part of the hostname. For example, if I set the URL to http://my-test-machine:8080/, the scanner says "Gethostname Error." in the logs and the upload fails (repeatedly). I can see in my nameserver's logs that the scanner is attempting to resolve my-test-machine:8080 instead of my-test-machine. This restriction makes it such that the scanner can only upload to port 80 (and I assume 443 when using HTTPS).

Is there something I'm missing or is this a firmware bug?

Webhook is not currently implemented – we ran into a few bugs we wanted to squash first. So, while we can't formally commit to the feature or the future per company policy, I can tell you that we're currently looking into this to make sure it's available in the near future.

crawford commented 3 years ago

So I don’t forget, it would be great to add a warning whenever a non-standard port is specified.

EDIT: The default port is 8080. I should add a README too.

ffleischer commented 3 years ago

Ok, that info was totally unexpected. Thanks for that.

I was able to receive a Request after changing the settings to:

URL: http://<sever_ip>
Username (optional) : not set
Password (optional) : not set
Upload New Scans via Webhook: Always
File Parameter Name: not set 
Optional Parameters:
  - tags = Doxie

and starting the container via sudo docker run -v /home/fabian/git/paperless-ng/paperless-ng/consume/:/uploads -p 80:8080/tcp doxie-upload (the sudo was required in order to take port 80).

But it looks like the request never gets handled or returns an error and the Doxie just tries it again. Here is the output:

fabian@fabian-pc:~/git/doxie-upload$ sudo docker run -v /home/fabian/git/paperless-ng/paperless-ng/consume/:/uploads -p 80:8080/tcp doxie-upload
[INFO  doxie_upload] Request from 192.168.178.115:50262
[INFO  doxie_upload] Request from 192.168.178.115:50263
[INFO  doxie_upload] Request from 192.168.178.115:50264
[INFO  doxie_upload] Request from 192.168.178.115:50265
[INFO  doxie_upload] Request from 192.168.178.115:50266
[INFO  doxie_upload] Request from 192.168.178.115:50267
[INFO  doxie_upload] Request from 192.168.178.115:50268
[INFO  doxie_upload] Request from 192.168.178.115:50269
[INFO  doxie_upload] Request from 192.168.178.115:50270
[INFO  doxie_upload] Request from 192.168.178.115:50271
[INFO  doxie_upload] Request from 192.168.178.115:50272
[INFO  doxie_upload] Request from 192.168.178.115:50273
[INFO  doxie_upload] Request from 192.168.178.115:50274
[INFO  doxie_upload] Request from 192.168.178.115:50275
[INFO  doxie_upload] Request from 192.168.178.115:50276
[INFO  doxie_upload] Request from 192.168.178.115:50277

Do you have any idea what could cause this behaviour?

I tried to change the folder permissions of the output folder on my host but the problem persits:

fabian@fabian-pc:~/git/paperless-ng/paperless-ng$ ls -al /home/fabian/git/paperless-ng/paperless-ng/consume/
total 8
drwxrwxrwx 2 root   fabian 4096 Aug  9 15:10 .
drwxrwxr-x 7 fabian fabian 4096 Aug  6 10:48 ..

Anything else I'm missing here?

PS: @crawford I think your repo is the only resource regarding the Doxie Webhook feature. Here was my support request

I'm currently trying to utilze the Webhook feature of my Doxie Q to integrate it into a self hosted paperless-ng instance. But The webhook log only shows a generic "GetHostname Error" if it tries to upload a scan. In Wireshark, I don't see any communication from the Doxie Q to my server. Is there any documentation or hints about the webhook feature and how to get detailed logs?

Thanks so much for your patience while we were looking into this – alas, at the moment, Webhooks isn't supported. We ran into a few bugs we wanted to squash first. So, while we can't formally commit to the feature or the future per company policy, I can tell you that we're currently looking into this to make sure it's available in the near future

crawford commented 3 years ago

I believe I’ve also seen this. I wish I could remember the details. Try increasing the logging verbosity (-vv or maybe even -vvv).

ffleischer commented 3 years ago

ok, here is the -vvv output. maybe it helps you to remember, because I have no real Idea how to fix it. I assume its some invalid data from the Doxie but I don't know which one.

fabian@fabian-pc:~/git/doxie-upload$ sudo docker run -v /home/fabian/git/paperless-ng/paperless-ng/consume/:/uploads -p 80:8080/tcp doxie-upload
[sudo] password for fabian: 
[TRACE mio::poll] registering with poller
[TRACE mio::poll] registering with poller
[TRACE mio::poll] registering with poller
[INFO  doxie_upload] Request from 192.168.178.115:46990
[TRACE hyper::proto::h1::conn] Conn::read_head 
[TRACE hyper::proto::h1::conn] flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy } 
[TRACE hyper::proto::h1::conn] Conn::read_head 
[DEBUG hyper::proto::h1::io] read 225 bytes 
[TRACE hyper::proto::h1::role] parse_headers
[TRACE hyper::proto::h1::role] -> parse_headers
[TRACE hyper::proto::h1::role] Request.parse([Header; 100], [u8; 225]) 
[TRACE hyper::proto::h1::role] Request.parse Complete(225) 
[TRACE hyper::proto::h1::role] <- parse_headers
[TRACE hyper::proto::h1::role] -- parse_headers
[TRACE hyper::proto::h1::conn] State::close_read() 
[DEBUG hyper::proto::h1::conn] parse error (invalid URI) with 225 bytes 
[DEBUG hyper::proto::h1::role] sending automatic response (400 Bad Request) for parse error 
[TRACE hyper::proto::h1::role] encode_headers
[TRACE hyper::proto::h1::role] -> encode_headers
[TRACE hyper::proto::h1::role] Server::encode status=400, body=None, req_method=None 
[TRACE hyper::proto::h1::role] <- encode_headers
[TRACE hyper::proto::h1::role] -- encode_headers
[DEBUG hyper::proto::h1::io] flushed 84 bytes 
[TRACE hyper::proto::h1::conn] flushed({role=server}): State { reading: Closed, writing: Closed, keep_alive: Disabled, error: hyper::Error(Parse(Uri)) } 
[TRACE hyper::proto::h1::conn] shut down IO complete 
[DEBUG hyper::server::conn::spawn_all] connection error: invalid URI 
[TRACE mio::poll] deregistering handle with poller
[TRACE mio::poll] registering with poller
crawford commented 3 years ago

I took a look at my settings and two things stand out:

  1. I’m not using custom parameters
  2. I have a trailing slash in my URL (e.g. http://scanner.domain/)

A trace from tcpdump would also be helpful. I’m sure that would reveal the structure of the malformed URL. My guess is that it’s the custom parameter that’s causing the error.

ffleischer commented 3 years ago

Hi @crawford, after adding a trailing slash and removing the custom parameters, I'm able to consume the documents provided by the webhook.

Many thanks for your support. Now i'm able to upload the documents via the webhook to your doxie-upload container which places the documents into the paperless-ng consume folder. In the pre-processing script a homeassistant automation is started which monitors if the import could be finished successfully (post-processing script needs to be executed within 10 minutes). Afterwards I receive a notification on my phone.

This is exactly what I expect from a modern EDMS setup.

Here are my final webhook settings:

URL: http://<sever_ip>/
Username (optional) : not set
Password (optional) : not set
Upload New Scans via Webhook: Always
File Parameter Name: not set 
Optional Parameters: none