Open wangxu94 opened 4 years ago
deranjer
commented
4 years ago Let me look into that. I made a few changes on the last hotfix for adding files to the database and didn't test that. I may also need the frontend log (F12 if on a chrome based browser). I'll get back to you.
deranjer
commented
4 years ago Okay, looks like I'll need the browser you were using, the frontend log when an upload occurs, and the backend log for when a file is uploaded. I'm running debian 9 and chrome and was not able to replicate the issue, although I am apparently not able to delete files from the UI, that will hopefully be a simple fix.
wangxu94
commented
4 years ago Browser: Mozilla Firefox 72.0.2 on Windows 10 1909
Frontend log when uploading and pressing "Submit":
preparing
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 0, id: "1580673064393-0", status: "preparing" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }
HomePage.js:75:69
getting_upload_params
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 0, id: "1580673064393-0", status: "getting_upload_params" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }
HomePage.js:75:69
uploading
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 0, id: "1580673064393-0", status: "uploading" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }
HomePage.js:75:69
headers_received
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 100, id: "1580673064393-0", status: "headers_received" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }
HomePage.js:75:69
done
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 100, id: "1580673064393-0", status: "done" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }
HomePage.js:75:69
Array [ {…} ]
HomePage.js:79:16
removed
Object { name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:51:04.394Z", percent: 100, id: "1580673064393-0", status: "removed" }
File { name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, webkitRelativePath: "", size: 133258, type: "application/pdf" }Backend log for the same:
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"GET","uri":"/src.f305e722.js.map","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":304,"referer":"","latency":0.089377,"latency_human":"89.377µs","bytes_in":"0","bytes_out":"0","time":"2020-02-03T08:50:15+13:00"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"GET","uri":"/documents/filesystem","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":0.679932,"latency_human":"679.932µs","bytes_in":"0","bytes_out":"703","time":"2020-02-03T08:50:15+13:00"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"GET","uri":"/documents/filesystem","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":0.649143,"latency_human":"649.143µs","bytes_in":"0","bytes_out":"703","time":"2020-02-03T08:50:15+13:00"}
{"level":"debug","time":"2020-02-03T08:51:07+13:00","message":"Working on current file: 8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T08:51:07+13:00","message":"PDF Text Result is empty, sending to OCR: 8tb_Ironwolf.pdfPDF Text Result is empty"}
{"level":"info","time":"2020-02-03T08:51:07+13:00","message":"Converting PDF To image for OCR/opt/goEDMS/documents/8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T08:51:07+13:00","message":"Creating temp image for OCR at: /opt/goEDMS/temp/8tb_Ironwolf.png"}
{"level":"error","time":"2020-02-03T08:51:07+13:00","message":"Unable to convert PDF Using Magick: /opt/goEDMS/documents/8tb_Ironwolf.pdfexit status 1"}
{"level":"error","time":"2020-02-03T08:51:07+13:00","message":"OCR Processing failed on file: /opt/goEDMS/documents/8tb_Ironwolf.pdf: exit status 1"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"POST","uri":"/document/upload","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":49.709431,"latency_human":"49.709431ms","bytes_in":"133553","bytes_out":"41","time":"2020-02-03T08:51:07+13:00"}The same issue occurs when using Chrome so I suspect it is server-side. Here are the frontend logs for Chrome v79.0.3945.130:
preparing {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}
HomePage.js:75 getting_upload_params {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}
HomePage.js:75 uploading {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}
HomePage.js:75 headers_received {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}
HomePage.js:75 done {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}
HomePage.js:79 [{…}]
HomePage.js:75 removed {name: "8tb_Ironwolf.pdf", size: 133258, type: "application/pdf", lastModifiedDate: "2020-01-29T21:19:32.315Z", uploadedDate: "2020-02-02T19:56:38.856Z", …} File {name: "8tb_Ironwolf.pdf", lastModified: 1580332772315, lastModifiedDate: Thu Jan 30 2020 10:19:32 GMT+1300 (New Zealand Daylight Time), webkitRelativePath: "", size: 133258, …}Please let me know if you need anything else. Thanks!
deranjer
commented
4 years ago Okay, I'm pretty sure I know what the issue is. When uploading from the webui it attempts to read the PDF, and if that fails it runs an OCR operation on the file. That OCR operation is failing. I need to see if there is a way for me to get more information from that error. When that error occurs when reading from the ingress folder it just leaves the file there so the user knows for sure there was a failure, however, I must not have done testing for when it is uploaded from the webUI because it is still placing that file in the documents folder which is causing all problem. Give me a few days and I'll push an update to fix that.
However, in the mean time, ensure the "convert" command runs from the command line of your Ubuntu server, and then run "which convert" and verify that the path to convert matches the one set in serverConfig.toml.
wangxu94
commented
4 years ago I have confirmed that the convert command runs, and that the path set in serverConfig.toml is correct.
I'm not sure if this will help, but I also tried placing the same file in the the ingress folder and this is what the log showed:
{"level":"-","time":"2020-02-03T15:33:08+13:00","message":"wake, now=2020-02-03T15:33:08+13:00"}
{"level":"-","time":"2020-02-03T15:33:08+13:00","message":"run, now=2020-02-03T15:33:08+13:00, entry=1, next=2020-02-03T15:43:08+13:00"}
{"level":"info","time":"2020-02-03T15:33:08+13:00","message":"Starting Ingress Job on folder:/opt/goEDMS/ingress"}
{"level":"debug","time":"2020-02-03T15:33:08+13:00","message":"Starting processing for file: /opt/goEDMS/ingress"}
{"level":"info","time":"2020-02-03T15:33:08+13:00","message":"Skipping Folder: /opt/goEDMS/ingress"}
{"level":"debug","time":"2020-02-03T15:33:08+13:00","message":"Starting processing for file: /opt/goEDMS/ingress/8tb_Ironwolf.pdf"}
{"level":"debug","time":"2020-02-03T15:33:08+13:00","message":"Working on current file: 8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T15:33:08+13:00","message":"PDF Text Result is empty, sending to OCR: 8tb_Ironwolf.pdfPDF Text Result is empty"}
{"level":"info","time":"2020-02-03T15:33:08+13:00","message":"Converting PDF To image for OCR/opt/goEDMS/ingress/8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T15:33:08+13:00","message":"Creating temp image for OCR at: /opt/goEDMS/temp/8tb_Ironwolf.png"}
{"level":"error","time":"2020-02-03T15:33:08+13:00","message":"Unable to convert PDF Using Magick: /opt/goEDMS/ingress/8tb_Ironwolf.pdfexit status 1"}
{"level":"error","time":"2020-02-03T15:33:08+13:00","message":"OCR Processing failed on file: /opt/goEDMS/ingress/8tb_Ironwolf.pdf: exit status 1"}UPDATE: I realised that there is an issue with my ImageMagick config, which I have resolved following the answer here: https://stackoverflow.com/questions/42928765/convertnot-authorized-aaaa-error-constitute-c-readimage-453
I am now able to manually run /usr/bin/convert 8tb_Ironwolf.pdf 8tb_Ironwolf.png, which successfully produces the png file.
However, I am still unable to add the same file to goEDMS, either through the web interface or through the ingress folder. The backend log is now as follows:
{"level":"debug","time":"2020-02-03T20:51:07+13:00","message":"Working on current file: 8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T20:51:07+13:00","message":"PDF Text Result is empty, sending to OCR: 8tb_Ironwolf.pdfPDF Text Result is empty"}
{"level":"info","time":"2020-02-03T20:51:07+13:00","message":"Converting PDF To image for OCR/opt/goEDMS/documents/8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-03T20:51:07+13:00","message":"Creating temp image for OCR at: /opt/goEDMS/temp/8tb_Ironwolf.png"}
{"level":"debug","time":"2020-02-03T20:51:12+13:00","message":"Output from pdfConvertCmd "}
{"level":"error","time":"2020-02-03T20:51:19+13:00","message":"Magick was unable to clean the image for some reason... skipping this file for now: /opt/goEDMS/documents/8tb_Ironwolf.pdfexit status 1"}
{"level":"error","time":"2020-02-03T20:51:19+13:00","message":"OCR Processing failed on file: /opt/goEDMS/documents/8tb_Ironwolf.pdf: exit status 1"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"POST","uri":"/document/upload","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":11827.89672,"latency_human":"11.82789672s","bytes_in":"133550","bytes_out":"41","time":"2020-02-03T20:51:19+13:00"}Can you give the latest version a shot? It should produce a more detailed error for me to look at and see what is going on.
wangxu94
commented
4 years ago Hi, I've just installed the latest version uploaded the same file again. Unfortunately the log doesn't seem to be more detailed:
{"level":"debug","time":"2020-02-04T11:25:53+13:00","message":"Creating path for file upload to ingress: /opt/goEDMS/ingress"}
{"level":"debug","time":"2020-02-04T11:25:53+13:00","message":"Working on current file: 8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-04T11:25:53+13:00","message":"PDF Text Result is empty, sending to OCR: 8tb_Ironwolf.pdfPDF Text Result is empty"}
{"level":"info","time":"2020-02-04T11:25:53+13:00","message":"Converting PDF To image for OCR/opt/goEDMS/ingress/8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-04T11:25:53+13:00","message":"Creating temp image for OCR at: /opt/goEDMS/temp/8tb_Ironwolf.png"}
{"level":"debug","time":"2020-02-04T11:26:01+13:00","message":"Output from magick: "}
{"level":"error","time":"2020-02-04T11:26:09+13:00","message":"Magick was unable to clean the image for some reason... skipping this file for now: /opt/goEDMS/ingress/8tb_Ironwolf.pdf exit status 1"}
{"level":"error","time":"2020-02-04T11:26:09+13:00","message":"OCR Processing failed on file: /opt/goEDMS/ingress/8tb_Ironwolf.pdf: exit status 1"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"POST","uri":"/document/upload","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":15325.699094,"latency_human":"15.325699094s","bytes_in":"133556","bytes_out":"39","time":"2020-02-04T11:26:09+13:00"}I uninstalled the old version, manually deleted /opt/goEDMS, then installed the new .deb file so I'm sure I'm running the latest version.
The good news is that the conversion/OCR failing no longer uploads the file to the /documents folder, and no longer causes the web interface to malfunction.
deranjer
commented
4 years ago Hmm, I may be able to push one more version in a few hours that might give us more info. I'll let you know.
deranjer
commented
4 years ago Made a few quick changes to allow us to see more information, give this a try. goEDMS v0.1.9-next
wangxu94
commented
4 years ago Thanks for that. The upload was successful with this version. Logs are as follows:
{"level":"debug","time":"2020-02-04T15:47:28+13:00","message":"Creating path for file upload to ingress: /opt/goEDMS/ingress"}
{"level":"debug","time":"2020-02-04T15:47:28+13:00","message":"Working on current file: 8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-04T15:47:28+13:00","message":"PDF Text Result is empty, sending to OCR: 8tb_Ironwolf.pdfPDF Text Result is empty"}
{"level":"info","time":"2020-02-04T15:47:28+13:00","message":"Converting PDF To image for OCR/opt/goEDMS/ingress/8tb_Ironwolf.pdf"}
{"level":"info","time":"2020-02-04T15:47:28+13:00","message":"Creating temp image for OCR at: /opt/goEDMS/temp/8tb_Ironwolf.png"}
{"level":"debug","time":"2020-02-04T15:47:34+13:00","message":"Output from magick: "}
{"level":"error","time":"2020-02-04T15:47:42+13:00","message":"Magick was unable to clean the image for some reason... skipping cleaning will attempt OCR anyway: /opt/goEDMS/ingress/8tb_Ironwolf.pdf "}
{"level":"debug","time":"2020-02-04T15:47:45+13:00","message":"Tesseract Command Run was: /usr/bin/tesseract /opt/goEDMS/temp/8tb_Ironwolf.png /opt/goEDMS/temp/8tb_Ironwolf"}
{"level":"info","time":"2020-02-04T15:47:45+13:00","message":"No record found, assume no duplicate hash: not found"}
{"level":"debug","time":"2020-02-04T15:47:45+13:00","message":"Adding full text for search to bleve: \n\n***i*fi*tt**t*i***t*iii*i*tt***t*i***\n1.0.1.3\n\nTax Invoice\n\nPB Tech Christchurch\n\nPB Technologies Ltd\n244 Main South Road\nHornby\n\nChristchurch 8042\nPh: 03-9262488\n\nGST No: 61280472\n\n \n\nInvoice No: SIC5569249\n\nDate: 29/01/2020\n\nCashier: Yanfei BAO\n\nSales Rep: Joseph Liang (R)\n\nAccount: NWBI34826\n\nx0 WANG\n\nCust.0rder No:\n\nProduct Qty Each Total\n\nHDDSE8181 18395.65 395.65\nSeaqste Ironwolf 8TB 256MB Cache\nSATA 6.06b/s NAS Internal Hard\n\nB==========:========================\nTotal excluding tar 395.65\nGST 15% 59.35\nTotal including tax 455.00\nEttpos 455 00\nSIC8569249\n\nUnless otherwise specified, goods\ncome with 1 year return to base\nwerranty, Ex-Lease with 3 months.\nReturns subject to a minimum 20%\nrestocking fee. Software not\ncovered under warranty, and cannot\nbe returned. Business / Trade\npurchases are not covered by the\nConsumer Guarantees Act. Company\nTerms and Conditions Apply See\nhttp://pbtech.co.nz/terms.php\nPLEASE KEEP THIS DOCKET AS PROOF\nOF PURCHASE\n\nitiiiiiitiiiiiiihiisetiiwiswi+saw*oi\n\n \n\n"}
{"level":"info","time":"2020-02-04T15:47:45+13:00","message":"Added /opt/goEDMS/ingress/8tb_Ironwolf.pdf to the database!"}
{"level":"info","remote_ip":"192.168.1.211","host":"192.168.1.7:8000","method":"POST","uri":"/document/upload","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0","status":200,"referer":"http://192.168.1.7:8000/","latency":17476.042046,"latency_human":"17.476042046s","bytes_in":"133553","bytes_out":"39","time":"2020-02-04T15:47:45+13:00"}Looks like the OCR was successful too. I think you can close this issue now, but I'm happy to help if there's anything else you would like me to test :)
deranjer
commented
4 years ago Okay, but Magick still is outputting nothing from the command when it fails. OCR works but magick isn't able to clean up the image (deskew, despeckle). Not a massive issue, but still could improve OCR performance. I'll dig some more into it, leaving it open for now.
wangxu94
commented
4 years ago Sounds good, let me know if you would like me to test anything :)
First of all thanks for your work on this, I've been looking for an alternative to MayanEDMS that is easier to install and manage.
Operating System: Ubuntu Server 16.04 goEDMS version: v0.1.8
I installed goEDMS from the .deb package following the instructions in the documentation. After uploading a file to the top level directory using the web interface, the web interface now does not show any files or folders.
This persists even after restarting goEDMS and reloading the page.
The debug log is as follows:
The file
/opt/goEDMS/documents/PBTech — Logitech G512.pdfis present on the filesystem.I was able to get the web interface to display the folders again by deleting the pdf file from the filesystem. The same error occurs when I upload a file into a folder instead of the top level.