unable to generate invoice pdf

[datformatie]

(manual) update to v5.10.11 => production.ERROR: Unable to generate the raw PDF {"userId":1,"exception":"[object] (App\Exceptions\FilePermissionsFailure(code: 0): Unable to generate the raw PDF at /public_html/invoiceninja/app/Jobs/Entity/CreateRawPdf.php:116) Seems like re-appearing issue?

Slack Message

[turbo124]

I believe the issue is with crashpad not being executable, i've added a PR for this upstream here:

https://github.com/beganovich/snappdf/pull/44

In the meantime you can test the fix by chmod +x crashpad

[datformatie]

No crashpad on system ..? find / -name crashpad => nofind. Where to find crashpad..?

[turbo124]

I have tagged a new version of the application with a different binary which should resolve this issue

[datformatie]

v5.10.13 installed => still not possible to use SNAPPDF to view pdf's...

[turbo124]

This is corrected in v5.10.13

[datformatie]

This issue still not resolved in our selfhosted intance of Invoiceninja after upgrade to v5.10.13: see attched pics. We repeated the upgrade (manual) and same result=> use of snappdf is not possible.

[CoryTrevor]

+1 this error still persists in 5.10.16. Snappdf works fine from the command line but not from the app.

[hennio]

Same here on 5.10.16

[CoryTrevor]

@hennio

Did it used to work for you and then stopped working after an update or are you trying to set it up for the first time?

[hennio]

For the first time.

[datformatie]

@CoryTrevor In our case it uses to work in pre v5.10.13 versions.

[CoryTrevor]

@hennio Same here, I'm setting up for the first time.

@datformatie That's interesting, I tried it with 5.9.6 and also experimented with older versions of Chromium but it always has the same error and there's nothing in the server error logs to help pinpoint what exactly causes the error.

What OS/servers are you guys using? I'm on Ubuntu 22.04, OpenLiteSpeed.

[datformatie]

@CoryTrevor (sorry forgot to dm you)

Hi there, OpenLiteSpeed 1.8.1 Almalinux 8 (also tried older chromium versions => no success) But now that I think of it: we switched server recently and also from Apache to Openlitespeed ... will check previous server again...

[datformatie]

@CoryTrevor, well just went back to OS=Almalinux 8, Apache 2.4.59 and IN v5.10.10 => SNAPPDF works from within IN too! Issue could be openlitespeed related then (maybe web user of openlitespeed related..?)

[turbo124]

This is an issue with the chrome binary that snappdf uses. I believe that integration is moving away from the chromium channel and pointing at a stable version of chrome such as ungoogled chrome.

[CoryTrevor]

Thanks @turbo124

I think the issue that @datformatie and I are having is separate from the recent snappdf/chromium issue. When using snappdf from the command line, old versions of chromium work fine for me as does the new ungoogled chrome.

I can replicate the issue that occurs when using snappdf from the command line after downloading chromium via snappdf on 5.9.6.: In Process.php line 435: The process has been signaled with signal "6". Making chrome_crashpad_handler executable resolves the issue and allows snappdf to work from the command line.

The issue that @datformatie and I are running into on OpenLIteSpeed is that even though snappdf works perfectly from the command line, it doesn't work from within the app. I've enabled OLS debugging and but nothing shows up in the OLS error logs, just this in laravel.log: production.ERROR: Unable to generate the raw PDF {"userId":119,"exception":"[object] (App\\Exceptions\\FilePermissionsFailure(code: 0): Unable to generate the raw PDF at /home/runcloud/webapps/testninja/app/Jobs/Entity/CreateRawPdf.php:114)

So it's tricky to troubleshoot without a more specific error message as to what exactly is causing the issue.

[turbo124]

The problem is the executing user does not have permission, you would need to add the litespeed server to your own group.

Just to note, ungoogled chrome does not use crashpad, so you are able to work around the issue by using the different binary.

[hennio]

I installed Invoiceninja with Softaculous . Its on my own Ubuntu Plesk server, so I have root-access. For me it's still unclear what to do to solve this.

[CoryTrevor]

@turbo124 I can use snappdf successfully from the command line when running the command as the linux user that has ownership of everything in the Invoice Ninja directory so I wouldn't have though there would be a permissions issue unless it's trying to access something outside of the Invoice Ninja directory?

[turbo124]

you need to try running it as the user of the webserver.

[datformatie]

@turbo124 if it is chromium (google or ungoogle) related: what is the best way to force snappdf / IN to use chromium google or ungoogle?

[CoryTrevor]

Thanks @turbo124, that makes sense. In the litespeed web server httpd_config.conf file it's set to run as runcloud-www user and if I run snappdf as runcloud-www rather than ninjauser that owns the files then it fails:

sudo -u runcloud-www ./vendor/bin/snappdf convert --html "Hello world" test.pdf
In Process.php line 435:                   
  The process has been signaled with signal "6".  

So I added runcloud-www user to the ninjauser user group so it's now:

id runcloud-www
uid=998(runcloud-www) gid=998(runcloud-www) groups=998(runcloud-www),1000(ninjauser)

But it still comes up with the same signal 6 error when running the command as runcloud-www. So I experimented with changing the permissions for the snappdf directory and everything inside it to 777 but still no luck. Not sure what to try next.

[turbo124]

@CoryTrevor

I believe the root issue here is that as these types of users ie www-data are not "real" users, they do not have their own .local directory. I believe chromium is attempting to write to the users local dir for crashpad (or something else) and this is the root issue that has been introduced in the latest version of chromium.

If you pivot to google chrome table, or ungoogled chrome, you will not see this particular issue.

[datformatie]

@CoryTrevor hi, did you mange to resolve this issue in the end? ps: in our case the issue was not solved in v5.10.13

[CoryTrevor]

@datformatie Not yet but making some progess. So for you does the same version work fine on Apache but not on OLS? Are you using ungoogled chrome? If you update to the latest IN version and then run: vendor/bin/snappdf download --force it should install ungoogled chrome. I believe snappdf uses whatever chrome install is specified in /snappdf/versions/revisions.txt but that should automatically update when it downloads a new version.

Thanks @turbo124, your help is much appreciated as always. When using ungoogled chrome instead of chromium and running snappdf as the webserver user rather than the website file owner, instead of the 'signal 6' error it comes up with a much more promising permissions error as the webserver user doesn't have write permissions to be able to save the pdf.

So I added the webserver user to the file owner's group, but it still wouldn't work because it turns out the server is also using Access Control List permissions. I was able to add the file owner's group to the ACL permission rules and then it works successfully to run snappdf from the command line as the webserver user:

sudo -u runcloud-www ./vendor/bin/snappdf convert --html "Hello world" test.pdf
Success! PDF saved at test.pdf

However it still runs into the same error when trying to generate the PDFs from within the app. To see if it would make a difference I tried setting ACL permissions to rwx for the runcloud-www user rather than just group access, and setting all folders and files to 777 but still no luck. If I try to set runcloud-www as the website file owner nothing works and it comes up with a 500 error and the laravel log shows a permissions issue as it can't write to /storage/framework/sessions/.

So there seems to be something that prevents the webserver user from writing and executing things when the request comes through the server rather than from the command line.

[datformatie]

@CoryTrevor migrated to v.5.10.17 => it works!

[datformatie]

UPDATE! migrated to v5.10.18 this morning (via Desktop App) and snappdf fails to work again ...

[2024-08-05 08:32:28] production.ERROR: Unable to generate the raw PDF {"userId":1,"exception":"[object] (App\Exceptions\FilePermissionsFailure(code: 0): Unable to generate the raw PDF at public_html/invoiceninja/app/Jobs/Entity/CreateRawPdf.php:116)

[datformatie]

ps going back to v.5.10.17 does not work (manually copied the .tar in IN home dir)=>snappdf not working

[CoryTrevor]

@datformatie Sorry to hear the issue persists, hopefully we can figure it out.

I've made some progress narrowing the issue on my server down to a memory limit issue. To simplify things for troubleshooting I created a php file that just checks the ungoogled chrome version with php exec():

$command = '../vendor/beganovich/snappdf/versions/ungoogled/chrome-linux/chrome --version 2>&1';
echo "Running command: " . $command;
echo '<br>';
$output = [];
$return_var = 0;
exec($command, $output, $return_var);
echo "Output: " . implode("\n", $output);
echo '<br>';
echo "Return Status: $return_var";

It works perfectly from the command line when run as the webserver user:

sudo -u runcloud-www php /home/ninjauser/webapps/testninja/public/testexec.php 
Output: Chromium 126.0.6478.182, Return Status: 0

But when the request is routed through the webserver by loading it in a browser it fails:

Output: Trace/breakpoint trap (core dumped)
Return Status: 133

The logs from running strace show that the two processes are virtually identical for the first 860 lines until the webserver initiated one terminates with:

mmap(0x2e800000000, 17179869184, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(0x2e800000000, 17179869184, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
--- SIGTRAP {si_signo=SIGTRAP, si_code=SI_KERNEL} ---
+++ killed by SIGTRAP (core dumped) +++

So I created an executable bash script that can be run from command line or called by php exec() through the server to output the resource limits using 'ulimit -a'.

When requests go through the webserver, max locked memory (kbytes) is 64 vs 1016020 from command line. And virtual memory (kbytes) is 2096128 vs unlimited from command line.

So the next step will be trying to increase those limits, but unfortunately it doesn't seem to be straightforward.

[CoryTrevor]

@datformatie Success! Finally got it working. It would have saved me a lot of time if I'd found this Litespeed forum thread earlier - https://www.litespeedtech.com/support/forum/threads/chrome-headless-crashes-with-cannot-allocate-memory-error.21545/

The solution was to set memSoftLimit and memHardlimit to 80G in the litespeed config.

[datformatie]

@CoryTrevor Great news (for you...) we are at the verge of giving up. Tried the settings memSoftLimit and memHardlimit to 80G in OLS => snappdf works in cli but not from within IN. Tried to replace OLS wth Apache to create a carbon copy of the IN instance => snappdf works in cli but not from within IN. Maybe if we had internal knowledge of the logic how snappdf is used from within IN, we could be succesful in solving this. It takes too much time however and we consider to use Hosted_pdf (if our client agrees).

[CoryTrevor]

@datformatie Sorry to hear the 80G trick didn't work for you. If you try creating a testexec.php file in the public directory like in my post above then the error output of that page should provide clues as to where it's going wrong. Just make sure to update the $command file path to the correct path for your server.

[datformatie]

@CoryTrevor ah good one thnx, will try that an keep you posted.

[datformatie]

@CoryTrevor quick question: how do we check log using strace when testexec.php is executed in browser?

[CoryTrevor]

@datformatie To run strace and output in browser you can use:

// Use command with strace and capture output directly
$command = 'strace -e trace=all ../vendor/beganovich/snappdf/versions/ungoogled/chrome-linux/chrome --version 2>&1';

echo "Running command: " . htmlspecialchars($command);
echo '<br>';

// Execute the command and capture the output
$output = [];
$return_var = 0;
exec($command, $output, $return_var);

// Display the output
echo "Output:<br>";
echo nl2br(htmlspecialchars(implode("\n", $output)));
echo '<br>';
echo "Return Status: $return_var";

[datformatie]

@CoryTrevor Hi there, well script runs in cli mode but not when invoked via Apache webserver.... => Running command: strace ... is prompted together with error 500 ... Any clues why this is happening (tried on two systems, same result)

[CoryTrevor]

@datformatie That's weird, sounds like it might be a server config issue. What output does it give when you visit the original testexec.php command that doesn't have strace in browser?

[datformatie]

@CoryTrevor same: in cli I see => Running command: ../vendor/beganovich/snappdf/versions/ungoogled/chrome-linux/chrome --version 2>&1
Output: Chromium 126.0.6478.182
Return Status: 0 when trying thru websever =>Running command: ../vendor/beganovich/snappdf/versions/ungoogled/chrome-linux/chrome --version 2>&1 and error 500 ps: tried on two different systems with similar results.

[CoryTrevor]

@datformatie Will it let you access anything in the public directory without causing a 500 error? What if you create an info.php file with: <?php phpinfo();

[datformatie]

@CoryTrevor tried that and I am able to execute other .php script without problems. so <?php phpinfo(); .php script in in/public dir works...

[datformatie]

@CoryTrevor looks like its in the script ... when I execute <?php phpinfo(); .php from with in your script it fails with error 500

[CoryTrevor]

@datformatie Have you got the opening php tag in the testexec.php file?

What happens if you cd into the public directory and run: php testexec.php

[CoryTrevor]

@datformatie Have you checked that strace is installed by running: which strace

Other thing to check is if exec is in the disable_functions section on the phpinfo page.

[datformatie]

Have you got the opening php tag in the testexec.php file? Yep What happens if you cd into the public directory and run: php testexec.php it runs perfectly Have you checked that strace is installed by running: which strace /usr/bin/strace Other thing to check is if exec is in the disable_functions section on the phpinfo page. disable_functions = system,passthruc,proc_close,proc_open,dl,popen,show_source,posix_kill,posix_mkfifo,posix_getpwuid,posix_setpgid,posix_setsid,posix_setuid,posix_setgid,posix_seteuid,posix_setegid,posix_uname

[CoryTrevor]

Well that all looks in order. What happens if you change the command to $command = 'echo "Hello World"';

[CoryTrevor]

Actually, make sure proc_open isn't disabled as that's listed as a requirement here - https://invoiceninja.github.io/en/self-host-installation/ Might be worth enabling proc_close as well.

[datformatie]

@CoryTrevor wait... just noticed something odd...

[datformatie]

@CoryTrevor $command = 'echo "Hello World"'; does not work either. I suspect PHP related issue however so need to investigate in that direction. Will keep you posted (if you don't mind that is.)

[CoryTrevor]

@datformatie For sure, keep me posted. Sounds like there's something preventing the php exec() function from running.

[datformatie]

@CoryTrevor Okay here is the latest: After getting no results and spendig too much time we decided to take a more drastic approach.

1. All but one version of PHP were removed from the VPS,
2. PHP was upgraded to 8.2.22 version,
3. PHP extensions ionCube PHP Loader v13.3.1 and Zend OPcache v8.2.22 were enabled,
4. PHP functions proc_close,proc_open were enabled,
5. PHP memory_limit was set to 2048M,
6. not sure if it was related but server hostname was reconfigured to resolve to ipv4 and ipv6 addresses,
7. /vendor/beganovich/snappdf/versions/revision.txt contains "ungoogled" value,
8. .env contains "PDF_GENERATOR=snappdf" value,
9. we decided to keep using Apache instead of OLS.

We did not test the script any further as in this configuration IN was able to produce .pdf views of the invoices again! Bottom line is that we did not get a finger on what really caused the issue (which we do not tend to like) however given the time and effort spend to resolve this issue, we deem acceptable. Thanks for your time and support!