Hanging on the Convert method on a website running locally

[davew9999]

I have a website in IIS on my local machine that I use for development, it's set up as a virtual directory. A few times a day it is hanging on the Convert method, the image below shows all the threads stuck on that method call. . I can't get any concrete replications steps, it just seems to stop working after a while. I've just done a test now and managed to download about 10-15 PDFs before it hung.

We have 2 versions of this site running on a web server, both of which are working fine and we aren't seeing any problems with it hanging.

[tuespetre]

I'm wondering if it's because even though the calls to 'Convert' are queued into the synchronized thread, the converter itself (in unmanaged library-land, not the managed IPechkin) is not destroyed until the IPechkin is disposed. I bet that's it. Wkhtmltopdf wouldn't like trying to convert anew with a converter still in existence.

[davew9999]

That makes sense, but could that be what I'm seeing as this is on my local machine with only me accessing it? So IPechkin was being disposed before I made another request. Also on the deployed version of the site all seems fine. Sounds like one of those "it works for me" issues and perhaps just an issue with my setup. If it's any help, before I used TuesPechkin I had a little wrapper over the wkhtmltopdf executable and used a Mutex to eliminate any problems like this. I know TuesPechkin uses PInvoke so I don't know if the same solution can be applied, I have no experience using PInvoke!

[davew9999]

If you come up with a solution I'd be happy to try it out.

[Zoliqa]

Hi, I have a problem which is exactly of the opposite that you're experiencing. Printing works fine locally in the development server but sometimes it hangs under IIS. I created a stress app that send multiple requests to IIS to print and I noticed that if I have 20 requests for print then the whole website crashes and I need to restart IIS for the website to work again. I've tried also the original Pechkin and PechkinSync libraries with the same result. I'm using IIS 7 and Windows 8. What could I do to solve this issue ?

[davew9999]

I guess in the meantime you can wrap your code in a mutex so only 1 thread can access the code at any one time. @tuespetre is this also the permanent solution? Create a mutex when the IPechkin instance is created and release it when it's disposed?

[tuespetre]

@Zoliqa, I am certain that my previous comment applies to your situation.

@davew9999, that will not be a permanent solution. I am just about to publish the permanent solution.

[davew9999]

Or perhaps just wrapping a mutex around the code that calls the wkhtmltopdf dll to do the conversion?

[davew9999]

Excellent thanks! Will give it a test on Monday when I'm back in work. Out of interest what is the solution?

[tuespetre]

It was just moving the 3-ish lines of code that destroy the converter out of Dispose into Convert right before returning the result. If more than one thread calls Convert they each get pended into the synchronized thread dispatch so they will not overlap.

[davew9999]

Ok cool. Will you be putting out a new release soon?

[tuespetre]

v0.9.3.3 is now published

[Zoliqa]

@tuespetre I've tried this new version but I'm still having the same issue under IIS, if I have 10 parallel requests for printing (it works for 5 parallel requests though :d), the whole site gets blocked and I have no choice but to restart IIS to make it work again. Still, it works fine no matter how many requests are made if I run it from the development server from VS. Any thoughts what could I try ?

[davew9999]

@tuespetre I've also been able to replicate the issue on the new version, although I managed to produce more PDFs than before (around 30 instead of 10-15) but this could purely be coincidental. Unlike @Zoliqa I didn't need to do parallel requests to cause it to happen, I was waiting for the PDF to be generated before generating another one through the website UI and was the only person using the site. It also only blocks IIS temporarily for me, after around 1 minute I can browse the site again although PDF generation will still fail and lock the site up for another minute. Resetting IIS makes the PDF generation work again. Just gave it another test after resetting IIS and managed to produce around 50 PDFs before it broke.

[tuespetre]

I'm going to stop being a wuss and write some more robust tests. I've just uncovered a semi-related bug already. I'll keep you updated.

Preview:

            var tasks = Enumerable.Range(0, 30).Select(i => new Task(() =>
            {
                IPechkin sc = Factory.Create(gc);
                sc.Convert(new ObjectConfig(), htmlText.Text);
                MessageBox.Show(String.Format("Conversion {0} finished", i));
            }));

            Parallel.ForEach(tasks, task => task.Start());
            Task.WaitAll(tasks.ToArray());

[tuespetre]

The issue I uncovered was related to the library initialization when running Parallel.Foreach. I corrected that much, then created a test that fires off 50 conversion in parallel and waits for them to complete:

        [Fact]
        public void HandlesConcurrentThreads()
        {
            string html = GetResourceString("PechkinTests.Resources.page.html");
            int numberOfTasks = 50;
            int completed = 0;

            var tasks = Enumerable.Range(0, numberOfTasks).Select(i => new Task(() =>
            {
                Debug.WriteLine(String.Format("#{0} started", i + 1));
                IPechkin sc = ProduceTestObject(new GlobalConfig());
                sc.Convert(html);
                completed++;
                Debug.WriteLine(String.Format("#{0} completed", i + 1));
            }));

            Parallel.ForEach(tasks, task => task.Start());

            while (completed < numberOfTasks)
            {
                Thread.Sleep(1000);
            }
        }

It is passing every time. I think I am going to need your help in solving this; if you can put together a stripped-down solution and some steps to reproduce I would like to check that out.

It might also help if you could set up a file trace writer. Pechkin writes to a trace source called "pechkin:default". Maybe we could get some insight into where it is hanging at.

[davew9999]

If you've made a change is it worth me giving that version a try to see if it fixes the issue I'm getting?

I've only got access to the source at work and some high priority stuff has just come my way so will probably be next week before I can put together a stripped-down solution and setup the file trace writer.

[davew9999]

Although it sounds like @Zoliqa has a stripped-down test app already they could perhaps send across?

[tuespetre]

The issue I uncovered would only affect initialization when two threads try to initialize the library at the same time; I haven't made a commit of it yet. You should be able to set up the trace in your web config; then just run through til it crashes once, drop everything and post the log.

Thanks for your cooperation and patience.

• Derek Gray

On May 13, 2014, at 7:37 PM, "davew9999" notifications@github.com wrote:

If you've made a change is it worth me giving that version a try to see if it fixes the issue I'm getting?

I've only got access to the source at work and some high priority stuff has just come my way so will probably be next week before I can put together a stripped-down solution and setup the file trace writer.

— Reply to this email directly or view it on GitHub.

[davew9999]

Ok will try get that across to you this week as it sounds like a quick job.

[davew9999]

Just tried setting up a file trace writer but I'm not sure exactly what I need to do, I've never done this before sorry! Here's what I've added to my web.config so far but I'm not getting any log file in the root of my E drive. If you could point me in the right direction that would be great.

<configuration>  
  <system.diagnostics>
    <sources>
      <source name="pechkin:default">
        <listeners>
          <add name="text"/>
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add name="text" type="System.Diagnostics.TextWriterTraceListener" initializeData="E:\trace.txt" />
    </sharedListeners>
  </system.diagnostics>
  <system.web>
    <trace enabled="true" localOnly="false" mostRecent="true" pageOutput="false" />       
  </system.web>  
</configuration>

[tuespetre]

I will set up an example trace on the PechkinTestApp. If I remember right the 'system.web trace enabled' you have in there is for some ASP web page tracing, separate from the system.diagnostics tracing.

[tuespetre]

Try this:

<configuration>  
  <system.diagnostics>
    <sources>
      <source name="pechkin:default">
        <listeners>
          <add name="text"/>
        </listeners>
      </source>
    </sources>
    <sharedListeners>
      <add name="text" type="System.Diagnostics.TextWriterTraceListener" initializeData="E:\trace.txt" />
    </sharedListeners>
    <trace autoflush="true" indentsize="4">
      <listeners>
        <add name="text" />
      </listeners>
    </trace>
  </system.diagnostics>
</configuration>

It was just missing the 'trace' section.

[davew9999]

Not having any luck, I pasted exactly what you've said above but it doesn't seem to have worked. I presume it should output something even if there are no errors? I tried using a different path and also creating the text file manually but to no avail. Any ideas?

[tuespetre]

I'm so sorry. Please try adding this one attribute in the following line like so, I have verified that it works with but not without it:

<source name="pechkin:default" switchValue="All">

[tuespetre]

Alright, here's what I saw the last couple of days while prepping version 1.0.0.

• Exceptions happening in the synchronized thread were being swallowed.
• Exceptions happening in the synchronized thread would cause that thread to lose its actual sync and make everything pile up!

Now, I didn't recreate those problems, I just saw they were there in a long, furrowed-brow examination of what might be happening.

Both of these things are taken care of now, so whatever exception is actually occurring for you will bubble out and show its ugly face. Please try it out.

[davew9999]

Just upgrading now, not going as smoothly as I hoped. Was just getting an empty PDF to begin with, then it hung on the Convert method, now I'm back to an empty PDF. Probably one of those Friday afternoon things where I've missed something obvious. I'll post my code if I get well and truly stuck.

[davew9999]

It's hanging every time I change some code and do a build until I do an IIS reset, not having much luck! I managed to get a PDF to be produced by just passing a string to the Convert method but it's just giving me a blank PDF when a pass in an instance of HtmlToPdfDocument. Just doing some process of elimination to try find out the cause.

[davew9999]

Ahhh figured it out, it's because the unit of measurement has changed for the margins, my margins were bigger than the document so it made it look empty! What were the original units prior to version 1.0.0? It appears it was something smaller than millimetres as I'm having to lower the values of my margins to match what I had before.

[davew9999]

Won't be able to look into this again until next week but here's what I've found so far...

Can't get the trace to work, at one point it did have some text in the file which was "Search: Id = 1 username = admin. Found: Id = 1 username = admin" and the file was locked until I reset IIS. Not convinced this is anything to do with TuesPechkin though as that's the id and username of the user I was logged in as (no idea where that text came from!). I've since deleted the file and I've not had anything since.

In terms of the Convert method hanging, it appears to hang now every time I do a build and try to produce a PDF. Also as before, it seems to hang after a while if I keep producing PDFs.

So far I've not had any error message, whether via an exception or in the trace file. I'll look into this further next week but in the mean time if there's anything you can suggest that would be great.

[tuespetre]

They were hundredths of an inch. Now it is represented by double instead of integer. I will be sure to better document that.

• Derek Gray

On May 30, 2014, at 10:31 AM, "davew9999" notifications@github.com wrote:

Ahhh figured it out, it's because the unit of measurement has changed for the margins, my margins were bigger than the document so it made it look empty! What were the original units prior to version 1.0.0? It appears it was something smaller than millimetres as I'm having to lower the values of my margins to match what I had before.

— Reply to this email directly or view it on GitHub.

[davew9999]

Still struggling to get the trace to work unfortunately; here's what I've got in web.config.

<system.diagnostics>
  <sources>
    <source name="pechkin:default" switchValue="All">
      <listeners>
        <add name="text"/>
      </listeners>
    </source>
  </sources>
  <sharedListeners>
    <add name="text" type="System.Diagnostics.TextWriterTraceListener" initializeData="E:\Logs\trace.txt" />
  </sharedListeners>
  <trace autoflush="true" indentsize="4">
    <listeners>
      <add name="text" />
    </listeners>
  </trace>
</system.diagnostics>

I've given 'Full control' permissions to 'Everyone' on the 'Logs' folder just to rule out permissions and I've also tried creating the 'trace.txt' file manually but nothing ever gets written to the file, both when it's producing PDFs fine and when it's hanging.

I've just tried producing lots of PDFs (over 100) and today it doesn't seem to want to hang, it would usually hang after around 50 PDFs or less. However, trying to produce a PDF after doing a build/rebuild of my code causes it to hang straight away and will no longer produce PDFs until I reset IIS.

When it does hang no exceptions are raised, the Convert method is called but it never comes out of it.

[tuespetre]

So far the only reason I have found for IIS to require a restart (actually, an app pool recycle should fix it) is related to issue #14 -- when Windows authentication is being used and the impersonation exception is thrown in the AppDomain.DomainUnload event handler. If using Forms Authentication, this problem does not occur in my test environment. Are you using Windows authentication?

[tuespetre]

Note: I haven't had luck with tracing in an MVC web app now. I don't think IIS plays nicely with it for some reason.

[davew9999]

Nope I'm using Forms Authentication.

I've just upgraded to version 1.0.3 as I noticed one of your releases notes says "Corrected a problem with everything hanging when the AppDomain unloads" and thought perhaps this could be related. It seems to have fixed it! Would you have thought that would be the case? Does the AppDomain unload when a rebuild occurs?

I've just been producing PDFs for a good 15 mins now and it's worked every time so it seems my problem with it hanging every now and again has been resolved, although I'm not sure what's resolved it.

I'd be happy for this issue to be closed.

[tuespetre]

That problem was a regression introduced in 1.0.0 (0.9.3.3 and earlier did not have it.) I'm willing to bet that your problem was in some way related to the thread principal, as that is the only other cause of hanging after rebuild/deploy that I have come across. In any event, I'm glad that things are working properly for you now!

[davew9999]

Thanks for all your help!