Attempting to convert large file to PDF causes 4D Server to hang

mhpetersaz commented 10 years ago

I'm trying to use HTML Convert to PDF to convert a large HTML file (sorry, the file has already been deleted to I can't provide the size) to a PDF. Doing so causes 4D server to hang and go "Not Responding" while consuming lots of CPU. Nothing else can take place on 4D Server and it must be Force Quit.

miyako commented 10 years ago

Hello,

the plugin is only calling an external library, but my guess is that issue is not the size of the HTML, but the fact that is has many (possibly recursive) references to other resources. (images, CSS, etc)

how is the HTML obtained? is a single file, is it complete for displaying locally?

miyako

From: mhpetersaz notifications@github.com<mailto:notifications@github.com> Reply-To: miyako/4d-plugin-html-converter To: miyako/4d-plugin-html-converter 4d-plugin-html-converter@noreply.github.com<mailto:4d-plugin-html-converter@noreply.github.com> Subject: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

I'm trying to use HTML Convert to PDF to convert a large HTML file (sorry, the file has already been deleted to I can't provide the size) to a PDF. Doing so causes 4D server to hang and go "Not Responding" while consuming lots of CPU. Nothing else can take place on 4D Server and it must be Force Quit.

[cid:20a3f4.png@99c353c4.429a36f1] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

mhpetersaz commented 10 years ago

Thanks for your quick reply.

The HTML is generated on the fly within a 4D Server method. This method is generating invoices for our customers and does contain their logo for each initial page of an invoice (which can be nultiple pages). This method frequently outputs the HTML string stored in a variable to a BLOB and to a file (the file is used for HTML Convert to PDF). If the user has selected HTML for output, we send the blob via HTTP SEND BLOB otherwise I call HTML Convert to PDF.

In the case that fails, the customer is trying to create a PDF for 3500 multiple page invoices. If we break that up into smaller chunks everything works fine.

Mike

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Saturday, November 15, 2014 2:31 PM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

Hello,

the plugin is only calling an external library, but my guess is that issue is not the size of the HTML, but the fact that is has many (possibly recursive) references to other resources. (images, CSS, etc)

how is the HTML obtained? is a single file, is it complete for displaying locally?

miyako

From: mhpetersaz notifications@github.com<mailto:notifications@github.com> Reply-To: miyako/4d-plugin-html-converter To: miyako/4d-plugin-html-converter 4d-plugin-html-converter@noreply.github.com<mailto:4d-plugin-html-converter@noreply.github.com> Subject: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

I'm trying to use HTML Convert to PDF to convert a large HTML file (sorry, the file has already been deleted to I can't provide the size) to a PDF. Doing so causes 4D server to hang and go "Not Responding" while consuming lots of CPU. Nothing else can take place on 4D Server and it must be Force Quit.

[cid:20a3f4.png@99c353c4.429a36f1] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63193437.

mhpetersaz commented 10 years ago

I did some more digging and it appears the problem is in wkhtmltopdf. I changed the code to stop using your plugin and to call wkhtmltopdf via LAUNCH EXTERNAL PROCESS. What I observed was that at some point, wkhtmltopdf stopped taking CPU time and just went dormant. While this occurred, 4D Server still ran at 100% CPU. If I killed the wkhtmltopdf process, control would return to 4D Server and the process would finish.

Any insight you can provide would be greatly appreciated.

Mike

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Saturday, November 15, 2014 2:31 PM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

Hello,

the plugin is only calling an external library, but my guess is that issue is not the size of the HTML, but the fact that is has many (possibly recursive) references to other resources. (images, CSS, etc)

how is the HTML obtained? is a single file, is it complete for displaying locally?

miyako

From: mhpetersaz notifications@github.com<mailto:notifications@github.com> Reply-To: miyako/4d-plugin-html-converter To: miyako/4d-plugin-html-converter 4d-plugin-html-converter@noreply.github.com<mailto:4d-plugin-html-converter@noreply.github.com> Subject: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

I'm trying to use HTML Convert to PDF to convert a large HTML file (sorry, the file has already been deleted to I can't provide the size) to a PDF. Doing so causes 4D server to hang and go "Not Responding" while consuming lots of CPU. Nothing else can take place on 4D Server and it must be Force Quit.

[cid:20a3f4.png@99c353c4.429a36f1] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63193437.

miyako commented 10 years ago

Hello,

in that case, I suggest you "break that up into smaller chunks". you must understand, creating a 3500 page PDF using this method is an extremely bad idea.

have you tried to open that 3500 page HTML with a browser? how long did it take to load? how much memory did it consume?

wkhtml2pdf is like a web kit off screen area, it first "opens" the HTML in memory, and then sends the print() command.

you probably know that a 4D plugin command is like a subroutine, it is loaded in the stack. so a process that uses a plugin needs a lot of stack (new process, execute on server).

in 4D you have the command GET MEMORY STATISTICS, so you should have first measured the memory required to produce a single page PDF, then multiply that by something like 5000, to be safe.

you should have measured how much time it takes to produce a single page PDF, and use SET DATABASE PARAMETER to extend the timeout because you know that the plugin command will not complete for several minutes, if not hours. (if it takes 2 second for 1 page, then 3500 will take 1 hour 56 minutes.) and because this is a cooperative process, nothing can be done on the server for that duration. but then, I would imagine that such process will simply blow up the wkhtml2pdf program, as I imagine the author did not anticipate so many pages to be processed at once, can you imagine a web page that is 3500 pages long?

this is why I say, creating a 3500 page PDF using this method is an extremely bad idea.

miyako

From: mhpetersaz notifications@github.com<mailto:notifications@github.com> Reply-To: miyako/4d-plugin-html-converter reply@reply.github.com<mailto:reply@reply.github.com> Date: 2014年11月15日土曜日 23:08 To: miyako/4d-plugin-html-converter 4d-plugin-html-converter@noreply.github.com<mailto:4d-plugin-html-converter@noreply.github.com> Cc: miyako keisuke keisuke.miyako@4d.com<mailto:keisuke.miyako@4d.com> Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

Thanks for your quick reply.

The HTML is generated on the fly within a 4D Server method. This method is generating invoices for our customers and does contain their logo for each initial page of an invoice (which can be nultiple pages). This method frequently outputs the HTML string stored in a variable to a BLOB and to a file (the file is used for HTML Convert to PDF). If the user has selected HTML for output, we send the blob via HTTP SEND BLOB otherwise I call HTML Convert to PDF.

In the case that fails, the customer is trying to create a PDF for 3500 multiple page invoices. If we break that up into smaller chunks everything works fine.

[cid:9a6747.png@209c09c1.42a57c56] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

mhpetersaz commented 10 years ago

Well, I overstated the size based on what a colleague told me was going on. It’s only 250 pages. The HTML opens fine. If I manually run the command line version of wkhtmltopdf it will actually convert the document. If I run it from 4D Server via LAUNCH EXTERNAL PROCESS wkhtmltopdf never completes. It just goes to 0% CPU.

I’ve seen other cases of this when I searched Google. I opened an issue with GitHub for the authors.

Thanks,

Mike

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Monday, November 17, 2014 12:55 AM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

Hello,

in that case, I suggest you "break that up into smaller chunks". you must understand, creating a 3500 page PDF using this method is an extremely bad idea.

have you tried to open that 3500 page HTML with a browser? how long did it take to load? how much memory did it consume?

wkhtml2pdf is like a web kit off screen area, it first "opens" the HTML in memory, and then sends the print() command.

you probably know that a 4D plugin command is like a subroutine, it is loaded in the stack. so a process that uses a plugin needs a lot of stack (new process, execute on server).

in 4D you have the command GET MEMORY STATISTICS, so you should have first measured the memory required to produce a single page PDF, then multiply that by something like 5000, to be safe.

you should have measured how much time it takes to produce a single page PDF, and use SET DATABASE PARAMETER to extend the timeout because you know that the plugin command will not complete for several minutes, if not hours. (if it takes 2 second for 1 page, then 3500 will take 1 hour 56 minutes.) and because this is a cooperative process, nothing can be done on the server for that duration. but then, I would imagine that such process will simply blow up the wkhtml2pdf program, as I imagine the author did not anticipate so many pages to be processed at once, can you imagine a web page that is 3500 pages long?

this is why I say, creating a 3500 page PDF using this method is an extremely bad idea.

miyako

From: mhpetersaz notifications@github.com<mailto:notifications@github.com> Reply-To: miyako/4d-plugin-html-converter reply@reply.github.com<mailto:reply@reply.github.com> Date: 2014年11月15日土曜日 23:08 To: miyako/4d-plugin-html-converter 4d-plugin-html-converter@noreply.github.com<mailto:4d-plugin-html-converter@noreply.github.com> Cc: miyako keisuke keisuke.miyako@4d.com<mailto:keisuke.miyako@4d.com> Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

Thanks for your quick reply.

The HTML is generated on the fly within a 4D Server method. This method is generating invoices for our customers and does contain their logo for each initial page of an invoice (which can be nultiple pages). This method frequently outputs the HTML string stored in a variable to a BLOB and to a file (the file is used for HTML Convert to PDF). If the user has selected HTML for output, we send the blob via HTTP SEND BLOB otherwise I call HTML Convert to PDF.

In the case that fails, the customer is trying to create a PDF for 3500 multiple page invoices. If we break that up into smaller chunks everything works fine.

[cid:9a6747.png@209c09c1.42a57c56] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63270375.

miyako commented 10 years ago

running a long process with LAUNCH EXTERNAL PROCESS is not a good idea, perhaps you should consider running a non blocking process, maybe with cmd.exe, and let the browser (through 4D) check if the process has been completed periodically.

I don't think you will get anywhere by opening an issue with the author, if you are not using the software appropriately.

miyako

Well, I overstated the size based on what a colleague told me was going on. It’s only 250 pages. The HTML opens fine. If I manually run the command line version of wkhtmltopdf it will actually convert the document. If I run it from 4D Server via LAUNCH EXTERNAL PROCESS wkhtmltopdf never completes. It just goes to 0% CPU.

[cid:965743.png@cc3ea265.40afbc1d] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

mhpetersaz commented 10 years ago

Thanks for all of your input. I really appreciate it. I’m not sure how to check to see if the LAUNCH EXTERNAL PROCESS has completed via the browser. Could you elaborate on that?

Thanks,

Mike

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Tuesday, November 18, 2014 4:46 AM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

running a long process with LAUNCH EXTERNAL PROCESS is not a good idea, perhaps you should consider running a non blocking process, maybe with cmd.exe, and let the browser (through 4D) check if the process has been completed periodically.

I don't think you will get anywhere by opening an issue with the author, if you are not using the software appropriately.

miyako

Well, I overstated the size based on what a colleague told me was going on. It’s only 250 pages. The HTML opens fine. If I manually run the command line version of wkhtmltopdf it will actually convert the document. If I run it from 4D Server via LAUNCH EXTERNAL PROCESS wkhtmltopdf never completes. It just goes to 0% CPU.

[cid:965743.png@cc3ea265.40afbc1d] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63458403.

mhpetersaz commented 10 years ago

I’m now running an asynchronous process (via LAUNCH EXTERNAL PROCESS) and checking to see when the PDF file get’s closed. What’s happening now is that wkhtmltopdf is becoming “defunct” and never closes the file. Running it from the command line works fine. Any other ideas?

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Tuesday, November 18, 2014 4:46 AM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

running a long process with LAUNCH EXTERNAL PROCESS is not a good idea, perhaps you should consider running a non blocking process, maybe with cmd.exe, and let the browser (through 4D) check if the process has been completed periodically.

I don't think you will get anywhere by opening an issue with the author, if you are not using the software appropriately.

miyako

Well, I overstated the size based on what a colleague told me was going on. It’s only 250 pages. The HTML opens fine. If I manually run the command line version of wkhtmltopdf it will actually convert the document. If I run it from 4D Server via LAUNCH EXTERNAL PROCESS wkhtmltopdf never completes. It just goes to 0% CPU.

[cid:965743.png@cc3ea265.40afbc1d] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63458403.

mhpetersaz commented 9 years ago

Do you have a 4D version 14 of your plugin build for 4D Server Windows x64?

Thanks,

Mike

From: Keisuke Miyako [mailto:notifications@github.com] Sent: Tuesday, November 18, 2014 4:46 AM To: miyako/4d-plugin-html-converter Cc: mike@petersfamily.com Subject: Re: [4d-plugin-html-converter] Attempting to convert large file to PDF causes 4D Server to hang (#1)

running a long process with LAUNCH EXTERNAL PROCESS is not a good idea, perhaps you should consider running a non blocking process, maybe with cmd.exe, and let the browser (through 4D) check if the process has been completed periodically.

I don't think you will get anywhere by opening an issue with the author, if you are not using the software appropriately.

miyako

Well, I overstated the size based on what a colleague told me was going on. It’s only 250 pages. The HTML opens fine. If I manually run the command line version of wkhtmltopdf it will actually convert the document. If I run it from 4D Server via LAUNCH EXTERNAL PROCESS wkhtmltopdf never completes. It just goes to 0% CPU.

[cid:965743.png@cc3ea265.40afbc1d] 宮古啓介セールス・エンジニア Tel: 03-6427-8441 E-mail: Keisuke.Miyako@4d.com Web: www.4D.com/JP

株式会社フォーディー・ジャパン〒150-0043 東京都渋谷区道玄坂1-10-2 渋谷THビル6F Tel: 03-6427-8441 Fax: 03-6427-8449

— Reply to this email directly or view it on GitHubhttps://github.com/miyako/4d-plugin-html-converter/issues/1#issuecomment-63458403.

miyako / 4d-plugin-html-converter

Attempting to convert large file to PDF causes 4D Server to hang #1