search

nextcloud / workflow_pdf_converter

Let Nextcloud automatically convert documents to PDF
https://apps.nextcloud.com/apps/workflow_pdf_converter
GNU Affero General Public License v3.0
23 stars 8 forks source link

No PDF generated, no log #118

Open manu-p opened 4 years ago

manu-p commented 4 years ago

Hi,

I set up a PDF conversion flow as follows: When File created, File updated and File MIME type matches Office documents => Keep original, overwrite existing PDF

I'd expect that when creating a new odt file with Collabora Online Development Edition, or just uploading a new odt file, I would see its corresponding pdf appearing aside, but nothing happens.

And absolutely nothing in the nextcloug log (loglevel : 2).

libreoffice --headless --invisible --norestore --convert-to pdf file.odt works fine in a terminal.

PDF Converter app

PDF Converter app version: 1.3.1

Server configuration

Operating system: Ubuntu 18.04 Web server: nginx 1.14.0 Database: MariaDB 10.1.44 PHP version: PHP 7.4.6 Nextcloud Version: 18.0.4

LibreOffice 6.0.7

# which libreoffice
/usr/bin/libreoffice

List of activated apps: Enabled:

Disabled:

Nextcloud configuration:

'enable_previews' => true, 'preview_libreoffice_path' => '/usr/bin/libreoffice', 'preview_office_cl_parameters' => ' --headless --invisible --norestore --convert-to pdf ',

Are you using external storage, if yes which one: local/smb/sftp/... No

Are you using encryption: yes/no No

Server log (data/nextcloud.log)

Nothing

manu-p commented 4 years ago

Any help appreciated. What more information can I provide to help solve this issue?

manu-p commented 4 years ago

Is there anybody out there?

janikvonrotz commented 4 years ago

I have similar setup and got the same problem. I am not sure about the cron.php. Does flow work at all in your case?

janikvonrotz commented 4 years ago

grafik

manu-p commented 4 years ago

@janikvonrotz yes, flow's working fine for "Block access to a file". We get no other flow setup at the moment.

manu-p commented 4 years ago

Whatever the background jobs method, no PDF is generated with this workflow.

Surprising there's no reaction here, apart yours, @janikvonrotz to confirm the problem.

Is this feature still supported?

Schmuuu commented 4 years ago

Well I gave a thumbs up, to support this issue and to show that I'm also affected :) I tried automated PDF creation via flow when NC 18 came out, but I never succeeded. I gave up at some point. I believed it was maybe my mistake and didn't file an issue.

glorenzutti commented 4 years ago

Im on the same page here. This dosent seems to work.

Also, I did a post on the help forum of nextcloud.

https://help.nextcloud.com/t/how-to-debug-a-workflow/84975

manu-p commented 4 years ago

Alright, thanks @glorenzutti Hope you get more luck than us before. :-)

vinid223 commented 4 years ago

For some updates, I did comment on @glorenzutti post in the help forums to be able to figure out the problem.

Follow the forum thread in case the fix is found, but I don't really have high hopes.

manu-p commented 4 years ago

I already do that, following this help forum thread... ;-)

rcdncn commented 4 years ago

@manu-p back in March I tried to implement this and could not get any action in the log (or PDF in the folder). Following.

vinid223 commented 4 years ago

@rcdncn You need to be patient, I found that the cronjob for conversion is executed each 5 minutes not as soon as a file is edited.

I did found that I get logs each time it tries to run, but the reasons array is empty since the shell_exec command just doesn't return anything. There might be a bug in the php version used, or just the command. I am not that familiar with PHP though.

vinid223 commented 4 years ago

For the devs, here is what I did for testing and searching. It is from the post I commented on the Nextcloud Help forum.

I confirmed first that my entry for the Flow was well configured with someone from the community.

Since I was using Nextcloud in a docker, I had to install the libreoffice binary into the docker image to have it. I can use the “libreoffice” command anywhere in the image since it is configured to run in the path.

I changed a docx file a lot of time to test each changes I could have done. Even restart the docker container for nextcloud.

I’ve configures a flow to send messages to Talk to be sure that the flow cron updates did actually worked to be sure I didn’t messed with this feature. It did sent a message into a conversation each time I did something that fits the creteria of the PDF converter, which I copied into my other flow for talk message.

I was also curisous to find if it was something else I have done with my installation so I’ve looked at the source code for the extension to convert to PDF and here is what I found.

The extension code should be able to detect my libreoffice installation in this method https://github.com/nextcloud/workflow_pdf_converter/blob/6951eb123e0618b125c2aba1f15791345ae2d41e/lib/BackgroundJobs/Convert.php#L130-L147

The first step check if the configuration 'preview_libreoffice_path' => '/usr/bin/libreoffice' is documented which I have, but by default it is not. So normally, the next step check for the binary location with the command shell_exec('command -v libreoffice'); on line 136 of the Convert.php file in the BackgroundJobs folder of the lib directory.

I wanted to go further to do some more testing to prove the thing is not working. Here is what I did.

Setup:

Virtual Machine with

image

When I update a file, the talk message is well received

image (1)

But when I check to see if there is a PDF generated, nothing is happening. I looked at the logs and this is what is returned.

image (2)

When I setup the config in the config.php, the error show "Could not convert file Test.docx, reason: []". There is no error in the reason array. Nothing is happening.

After looking again at the source code, it might just be the exec that always return a non 0 return code and never continues the save process. This might be the reason the array of reason is empty. I could debug the code, but I don’t really want to go that deep since I can reproduce the error with a fresh install of Nexcloud on docker or in a VM.

So I don’t know what the issue is really. More debug should be done, but at this point, I think the maintainers could check it out. The problem seems to be on any platform. Even the leaders on the forum told me that their installation stop generating PDFs for a while.

Hope this helps

xDeerStalker commented 4 years ago

Think i solved it. I added a logline in Converter.php to print $exec and then manually run it as www-data user. The issue is not in the scope of this app nor in the scope of nextcloud.

The underyling problem is that libreoffice will be run with www-data user and it needs the directory ~/.cache/dconf/

Since the home-directory of www-data usually is /var/www which is owned by root, www-data can´t create the directory and libreoffice fails to convert.

So either:

sudo mkdir /var/www/.cache/dconf/ sudo chown -R www-data:www-data /var/www/.cache sudo chown www-data:www-data /var/www

OR

change the home directory of www-data via passwd to somewhere else and create .cache/dconf/ there.

The first method probably has some security impacts whilest the second method probably has some stability impacts for software that relies on www-data home beeing /var/www

Maybe someone else knows a better solution? At least, both solutions provided should get the App working again

manu-p commented 4 years ago

Hi, glad to read it works for you (at least). But not for me (1st method).

Still no pdf generated (15 min cron job), still nothing in log files (nextcloud and nginx).

Maybe related, or not, I sometimes get a error mail from Cron: Subject: Cron www-data@_server_ /usr/bin/php -f /var/www/nextcloud/cron.php Content: func=xmlSecCheckVersionExt:file=xmlsec.c:line=188:obj=unknown:subj=unknown:error=19:invalid version:mode=abi compatible;expected minor version=2;real minor version=2;expected subminor version=25;real subminor version=26

Seems to be related to the preview generator, but in case...

xDeerStalker commented 4 years ago

Okay, try the following:

  1. upload a docx file named ConverterTest.docx to nextcloud
  2. sudo -u www-data cp /var/www/nextcloud/data/path/to/ConverterTest.docx /tmp
  3. sudo -u www-data /usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/ConverterTest.docx'

"3" is the exact command with the same priviledges that would be used by this app. Now there are 2 possible outcomes:

After the command has finished, /tmp/ConverterTest.pdf exists. If this is the case: sudo -u www-data cp /tmp/ConverterTest.pdf /var/www/nextcloud/data/path/to/ and the .pdf will be visible in Nextcloud OR you get the error which is the cause why the app doesn´t work

Second possibilty: The command fails. You get the error which is the cause why the app doesn´t work

I think I maybe got the bug in this app which prevents printing something useful into the log: In Converter.php where the command is built (line 92):

$exec = $command . $clParameters . escapeshellarg($tmpDir) . ' ' . escapeshellarg($tmpPath);

should be

$exec = $command . $clParameters . escapeshellarg($tmpDir) . ' ' . escapeshellarg($tmpPath).' 2>1&';

That redirects STDERR to STDOUT which is saved to $out. At least that´s what I found in the web; I never worked in php before.

manu-p commented 4 years ago

Hi, thanks @xDeerStalker for taking care.

I used a .odt instead of .docx, don't think it makes a big difference, and here's what the output gives me (sudoed as root):

# sudo -u www-data /usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/test.odt'
/usr/bin/libreoffice: 54: cd: can't cd to /root

and when I cd /tmp before:

# sudo -u www-data /usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/test.odt' 
javaldx failed!
Warning: failed to read path from javaldx
X11 connection rejected because of wrong authentication.

(process:18560): dconf-CRITICAL **: 10:15:32.096: unable to create directory '/root/.cache/dconf': Permission denied.  dconf will not work properly.
X11 connection rejected because of wrong authentication.

"2>1&" is just a shell redirection to put stderr (2) to stdout (1).

xDeerStalker commented 4 years ago

Yep, that's the same error i got, just with another home directory for www-data which seems to be /root in your case.

please verify with: cat /etc/passwd | grep www-data

this should in your case return something like this: www-data:x:33:33:www-data:/root:/usr/sbin/nologin

usually the home-directory of www-data is not /root but /var/www and the output should look like this: www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

I would suggest changing the home directory, but you have to check if www-data owns something inside /root for whatever reason. To do so, type:

sudo ls -la /root | grep www-data

if the command above returns nothing you´re probably good to go.
Also, ensure root is the only user with access to /root by doing

sudo chmod 700 /root

If you didn´t revert the changes i suggested (creating /var/www/.cache/dconf and changing priviledges with chown) you can just sudo usermod -d /var/www www-data

and the bug should be solved.

In any case, don´t mess with priviledges for the /root directory and don´t create .cache there. At least that´s my advice. It can lead to serious disasters if www-data or anyone else who is not root can read .mysql_history or .bash_history of root due to a misconfiguration.

"2>1&" is just a shell redirection to put stderr (2) to stdout (1).

Yes and exec() in php seems to only print stdout which would be the reason why the exception you get when you run the command manually is not logged. A similar problem can be found here

manu-p commented 4 years ago

Yep, that's the same error i got, just with another home directory for www-data which seems to be /root in your case.

Nope!

please verify with: cat /etc/passwd | grep www-data

# grep www-data /etc/passwd
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

Exactly what you're showing too.

There must be some other issue somewhere...

# ll /var/www
total 40
drwxr-xr-x  9 www-data www-data 4096 Sep 22 14:00 ./
drwxr-xr-x 13 root     root     4096 Apr  9 11:27 ../
drwxr-xr-x  3 www-data www-data 4096 Sep 15 11:00 .cache/
. . .
# ll -R /var/www/.cache/
/var/www/.cache/:
total 12
drwxr-xr-x 3 www-data www-data 4096 Sep 15 11:00 ./
drwxr-xr-x 9 www-data www-data 4096 Sep 22 14:00 ../
drwx------ 2 www-data www-data 4096 Sep 15 11:00 dconf/

/var/www/.cache/dconf:
total 12
drwx------ 2 www-data www-data 4096 Sep 15 11:00 ./
drwxr-xr-x 3 www-data www-data 4096 Sep 15 11:00 ../
-rw------- 1 www-data www-data    2 Sep 25 13:15 user
# file /var/www/.cache/dconf/user
/var/www/.cache/dconf/user: data
# od /var/www/.cache/dconf/user
0000000 000000
0000002
xDeerStalker commented 4 years ago

Yea, i´m stupid. I could reproduce your error and it´s no miracle - you got the wrong command from me =/

Ignore this: sudo -u www-data /usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/ConverterTest.docx'

and replace that step from https://github.com/nextcloud/workflow_pdf_converter/issues/118#issuecomment-698276354 with: 3: sudo su www-data -s /bin/bash and then run:

4: usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/ConverterTest.docx'

For whatever reason in this case, sudo -u www-data ; and sudo su www-data; ; don't produce the same output. Usually, any commands that are triggered by the command behind sudo (let´s say a lot of programs in a script) indirectly are run with the same user you sudoed into. This is not the case here. I think as soon as the exception throws and libreoffice exits with code != 0, sudo is considered done.

Then, when the actual error gets printed, you are back in the account you started from (root i guess?) and all the relative paths with their nice shortcuts are replaced by absolute paths - but this pathconversion should happen inside libreoffice before it exits and not somewhere else. Probably a bug of libreoffice ; i´m able to reproduce it from multiple accounts. It´s always requesting access to ~, ~/.cache and ~/.cache./dconf where ~ is replaced by the absolute path to the user-home of the user you sudo -u from.

Okay, long story short: sudo su directly into www-data and you probably get the exception that you have no access to /var/www. That was the last step from the initial answer https://github.com/nextcloud/workflow_pdf_converter/issues/118#issuecomment-695221354 which you didn´t execute.

So grant privileges ( chown www-data:www-data /var/www ). Test the command again from a www-data shell. If this works, test in Nextcloud - this should work now.

Now it gets funny: remove all the given the privileges again which means changing /var/www back to root:root and the same for .cache and .cache/dconf

It still works.

How?

black magic of open source software ;-)

I think the permissions for www-data/libreoffice aren´t really needed to get the job done but they are an initial requirement to get it working.

phil16727 commented 4 years ago

Hey Guys, I am now totally confused sorry. I am on Ubuntu 20.04 installed Nextcloud as a snap. There was not /var/www/.cache/dconf folder as mkdir was noch able to create it. I did it by hand. So what I did: sudo mkdir /var/www/.cache/dconf/ sudo chown -R www-data:www-data /var/www/.cache sudo chown www-data:www-data /var/www sudo su www-data -s /bin/bash

NO Luck the Convert Workflow App is still not working. Is there an "how to" to get this thing running? I do get this error could not convert /admin/files/FileName, reason: []

xDeerStalker commented 4 years ago

@phil16727 after sudo su www-data -s /bin/bash you did usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/ConverterTest.docx' where /tmp/ConverterTest.docx is a valid .docx, owned by www-data and is readable by www-data?

While executing that command you don´t get any exception and after the command is finished and /tmp/ConverterTest.pdf still doesn´t exist, is that correct? However, if /tmp/ConverterTest.pdf exists, the app should work too because it does exactly the same thing.

Is there an "how to" to get this thing running?

I dont´t think so, maybe our conversation will lead to some "how to"

phil16727 commented 4 years ago

Hi, well after I sudo su www-data -s /bin/bash i went back to the Nextcloud GUI and Uploaded a file (Upload is the trigger to convert a docx into a pdf) but still got could not convert /admin/files/FileName, reason: [] in the nextcloud logger

xDeerStalker commented 4 years ago

Okay, please cp /var/www/nextcloud/data/path/to/ConverterTest.docx /tmp

after that, run usr/bin/libreoffice --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir '/tmp' '/tmp/ConverterTest.docx' from the shell where you are www-data@hostname (the one where you entered sudo su www-data -s /bin/bash). In doubt, the command whoami should return www-data.

if that finishes without an error, verify /tmp/ConverterTest.pdf exists. If the command throws an error, please post it here

phil16727 commented 4 years ago

Hey @xDeerStalker Thanks for the reply. Just to get this right, we are about to roll out the nextcloud to 25 User or so. I wont ssh every single file verytime it changes. Therefore I dont really get it, why I should convert a single docx by ssh?

xDeerStalker commented 4 years ago

because that's the only way to get the real cause of the error while the nextcloud log will always say "reason: []". You will not have to do that every time you want to generate a pdf, just now to get it debugged

phil16727 commented 4 years ago

OK i will do that, but I have no var/www/nextcloud since the installtion is though snap. Any idea?

xDeerStalker commented 4 years ago

/var/snap/nextcloud/common/, from https://github.com/nextcloud/nextcloud-snap/wiki/Finding-files

phil16727 commented 4 years ago

Ok, well I looked for the files I uploaded. They are located at /var/snap/nextcloud/common/nextcloud/data/admin/files But I cant get past common /var/snap/nextcloud/common/ bash: cd: nextcloud: Permission denied

Sorry for the newbee question how do i figure out thre Systemuser and HomeDrive of Nextcloud

xDeerStalker commented 4 years ago

The problem is snap, i never worked with it. From what i understand, it´s basically like a pre defined docker image without the virtualization.

So there may be many more problems now: where is libreoffice installed? can the user owning /var/snap/nextcloud access it? test with sudo su into his account and try the command above.

if he can access libreoffice, you basically have to do everything you've done again as the user which runs the snap / nextcloud. This user will execute the command to convert the pdf.

Even if you get all this done - from what i have read it would be probably a better choice considering to install nextcloud directly on a system of your choice or (manually) inside a dockercontainer. Here is why:

I did the same thing you try to do ( with 100GB of shared data and 20 Users) 2 weeks ago on a vps with 8 vcores and 16GB of RAM. Nextcloud had a really low performance before tuning (opcache, redis, switching from prefork to php-fpm, http2 and so on). It´s a game changer if you do all these things and edit all those configs. All in all, it was a 2 nights job including the initial data upload without any knowledge about nextcloud, php or redis at all. If you know what you're doing, i guess it can be setup and tuned within 2 hours (without data upload obviously)

All the configs, paths and settings are very well known to the internet and every error i encountered was already solved somewhere else. That may not be the case for the snap version. Check this to get a summary of why it´s a bad idea to use snap in the long run.

I´m pretty sure you can solve the problem after getting a linux and a snap guru but if you are a newbee it will take forever to figure things out that are not considered "standard". it´s like plesk or all those other admin tools. As long as it works, you´re fine. But as soon as you have to dig into it, things get really confusing because nothing is where it should be.

phil16727 commented 4 years ago

@xDeerStalker i started all over without the snap installation. It now works perfectly. thx

manu-p commented 4 years ago

Yea, i´m stupid. I could reproduce your error and it´s no miracle - you got the wrong command from me =/ [ . . . ] I think the permissions for www-data/libreoffice aren´t really needed to get the job done but they are an initial requirement to get it working.

Hi, sorry for the delay, I have now some time to confirm that:

  1. I've been able to convert to pdf my odt example file on /tmp following the steps you corrected in your comment 18 days ago.

  2. /var/www has been owned by www-data:www-data for a long time now, as of its subdirectories.

  3. I don't see any pdf automatically generated in NC (even after 15' which is the crontab interval for running cron.php.

xDeerStalker commented 4 years ago

Okay, if it works as www-data when doing it by hand, it should work by the app as well.

Just to be sure, you can sudo -u www-data mv /tmp/ConverterTest.pdf /var/www/nextcloud/data/wherever/the/file/should/be , the file is owned by www-data and readable? I guess it should be the case. If this works, is the file visible in Nextcloud?

I didn´t have any problems with nextcloud not recognizing the created pdf, but in another case i needed to run sudo -u www-data /var/www/nextcloud/occ files:scan --path="myuser/files/newInsertedFilesDirectory/" after generated and inserted a file into Nextcloud via some pythonscript. If the file is in nextcloud after running the command, i guess there's a(nother) bug with the app i don´t know about.

You definitly have to wait until cron.php runs, so the conversion is never "instant" (except for the ajax method maybe). If i understand it correctly, you are using system cron to execute cron.php ? This would by default execute every 5 minutes - not every 15 minutes. Did you change something there? Please make sure one entry of www-data's crontab ( which can be viewed by sudo -u www-data crontab -l and edited by sudo -u www-data crontab -e ) looks like this:

*/5 * * * * php -f /var/www/nextcloud/cron.php or like this is you want to stay at 15 minutes interval: */15 * * * * php -f /var/www/nextcloud/cron.php

manu-p commented 4 years ago

Thanks @xDeerStalker I'll be checking all these asap. As you could read I'd been waiting for 15' for the cron.php job to be executed.

BTW I don't know why this job is scheduled on a 15' basis, and what is the incidence (side effects for instance) if I reduce the delay to 5'. It's not the topic here, I admit. All I find is this documentation mentioning the way to reduce the delay between runs to 5' (which seems to be acceptable then) and before that:

Nextcloud apps register actions with cron.php automatically to take care of typical housekeeping operations, such as garbage collecting of temporary files or checking for newly updated files using filescan() for externally mounted file systems.

I guess (but I don't know) that if the next cron.php starts before the current job is not finished, nothing wrong is supposed to happen to the running NC instance, apart maybe a higher CPU and/or I/O load...

Thanks again, I'll be posting soon the results I see.

manu-p commented 4 years ago

Hi, nothing has happened to my odt test files, no pdf has appeared in NC aside these files. They're not in the NC data directory where the 2 odt files are stored, neither. (cron.php execution reduced to every 5')

Everything's working fine when I do the commands manually, as you explained before.

abclution commented 4 years ago

I gonna contribute a bit here because I too am struggling with this. I have gotten it work but only sometimes. So far, most of the initial actions DO NOT create the cron job and convert the file.

Right now the ONLY action I know for sure which creates the cron job properly is the When "File renamed" trigger is used as seen below.

image

If I find any other that work, I will mention it but please test the File Renamed trigger (also put the job in your user cron / or run the cron job manually after renaming a file to test.

ex. as your webuser or apache user /usr/bin/php7.4 -f /home/cloud.mynextcloud.com/www/cron.php

Triggers that do not create the cron or convert the file. File created, File updated, File accessed, File copied.

elpraga commented 3 years ago

Is there any update on this issue? I've been able to successfully convert files under the www-data user in the terminal (by changing the ownership of /var/www), but the tag based automatic PDF generation upon change is still not w working.

Is there anyone at all for whom it works?

elpraga commented 3 years ago

BTW as some people might feel uneasy about changing the ownership of /var)www, would it be possible to make setting a different home part of the app (script)? By making user of the HOME variable?

elpraga commented 3 years ago

I'd suggest to set the HOME variable to /whatever/www/yournextcloud/apps/workflow_pdf_converter/libreiffice_home This directory should be owned by www-data anyways and it's contents are under full control of the Nextcloud app.

tauceti82 commented 3 years ago

I gonna contribute a bit here because I too am struggling with this. I have gotten it work but only sometimes. So far, most of the initial actions DO NOT create the cron job and convert the file.

Right now the ONLY action I know for sure which creates the cron job properly is the When "File renamed" trigger is used as seen below.

image

If I find any other that work, I will mention it but please test the File Renamed trigger (also put the job in your user cron / or run the cron job manually after renaming a file to test.

ex. as your webuser or apache user /usr/bin/php7.4 -f /home/cloud.mynextcloud.com/www/cron.php

Triggers that do not create the cron or convert the file. File created, File updated, File accessed, File copied.

I can confirm that file renamed action works but the other actions not. PDF generation works with command line steps described here manually. Can this issue please be fixed?

tauceti82 commented 3 years ago

did some more testing: works also in admin flows and also when selecting single actions like file rename or file created...but not with multiple