Upgrading 2sxc from 11.02 to 11.07.02 the site runs very slow because of ImageResizer DLL

[jeremy-farrance]

I'm submitting a bug report or not sure.

About Upgrading or other / unknown.

Current behavior I started with a fast/small site running DNN 8.00.04, with 2sxc 11.02 installed and working. The only 2 other modules are DNN Sharp's ActionForm (old version, 3.7.3) and EasyDNN News 7.6.0.

I made a backup, installed 2sxc 11.07.02 (latest) and the install appear successful, however, each page-load took 90 to 120 seconds. After more testing and not finding many errors logged, I restored the site and tried again.

I started again with 2sxc 11.02 and upgraded to just v11.03, same exact results.

Should 2sxc 11.03 and higher still be compatible with DNN 8.00.04?

Expected behavior Upgrade 2sxc to latest and everything work AND run at normal speed.

Minimal reproduction of the problem with instructions Not sure, but if you have a DNN 8.00.04 site with same module, try it? ;)

Please tell us about your environment: Win Server 2016, SQL 2016, etc

2sxc version(s): 11.02 upgrading to 11.03 or 11.07.02 Browser all DNN 8.00.04

I can successfully load each page on the site by waiting 2 minutes and no errors get logged.

Here are the only 2 error messages I can find after the update. I am not even sure it is related?

2020-10-03 11:39:30,045 [H2036v06][Thread:40][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.Exception: File load failed, aborting
2020-10-03 11:50:28,205 [H2036v06][Thread:50][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.IO.IOException: The process cannot access the file '... \web\DesktopModules\ToSIC_SexyContent\Upgrade\Log\2020-10-03 16-50-28-2053899-5524-5.log.resources' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at ToSic.Sxc.Dnn.Install.DnnInstallLogger.OpenLogFiles() in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\DnnInstallLogger.cs:line 43
   at ToSic.Sxc.Dnn.Install.InstallationController.IsUpgradeComplete(String version, String note) in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\InstallationController_IsRunning.cs:line 10
   at ToSic.Sxc.Dnn.Install.InstallationController.UpdateUpgradeCompleteStatus() in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\InstallationController_Statics.cs:line 14
   at ToSic.Sxc.Dnn.Install.InstallationController.UpgradeModule(String version) in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\InstallationController.cs:line 139
   at ToSic.SexyContent.Environment.Dnn7.DnnBusinessController.UpgradeModule(String version) in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Environment\Dnn7\DnnBusinessController.cs:line 89
   at DotNetNuke.Entities.Modules.EventMessageProcessor.UpgradeModule(EventMessage message)

[iJungleboy]

The basics

1. 2sxc still works on DNN 7.4.2, so 8.x shouldn't be an issue
2. The access to that file happens to ensure that booting/upgrading a DNN in a Farm won't run the same code twice

as of now, 2sxc.org is already running on 11.7 LTS, so it's certainly not a general issue. I suggest you debug to see if it's a 2sxc module that's slow, and which one. Best thing to do is go to insights:

Then check things like the module-render time

For example, this module took 5ms to load (which is awesome 🚀)

Based on this you should be able to find out more what's taking time, and continue from there.

[iJungleboy]

any update on this?

[jeremy-farrance]

Sorry, I won't be able to get back to this until the upcoming weekend. I ended up restoring from the backup.

[iJungleboy]

No problem. I'll just close this for now, since I assume it's not a 2sxc bug - once we know more, I may reopen it.

[jeremy-farrance]

Okay, third time, this is another site and on a different server than the one I reported above. DNN 9.03.02 site with 2sxc 9.43.2 installed. Note that 2sxc was unused, it looks like it was installed as a convenience so we had ImageResizer available. So no 2sxc Content-App, no other apps, nothing. Only other modules installed: EasyDNNNews 10.6, ActionForm 5.0.7xx, DNNBackup 8.7.2.

I restarted the app pool and then upgraded 2sxc to 11.7.3 the usual way. Immediately after it completed (with no errors reported) the site got very slow. Page loads generally taking 60 to 120 seconds.

Again, 2sxc has not yet been added to a page, so I have no way to get to Insights (or do I?). So yeah, I figured out that even though 2sxc (neither app nor content) has been added to a page, this URL still gets you there. /api/2sxc/sys/insights/help - the help page worked, I clicked on Modules and all I get is this:

Slowness - even this simple URL which does return 'true' takes 22 seconds /api/2sxc/sys/insights/isalive

Next I started watching the tasks/services/processes details in Task Manager and Process Monitor. I noticed something unusual. For this site's IIS app pool, about once every minute or so, csc.exe (the .NET CSharp Compiler) was starting and running for a while, then vanishing for a few seconds and starting again. When I stopped the site and/or app pool in IIS, this process did not come and go, but returned as soon as I spun the site up again.

So next I checked /Portals/_default/logs/2020.11.10.log.resources, does any of this seem to point to a cause?

2020-11-10 19:26:01,077 [H2034v05][Thread:39][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SqlServer.SqlClrProvider, Version=13.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SqlServer.SqlClrProvider, Version=13.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91'
   at Microsoft.SqlServer.Management.Common.ConnectionManager.CallerHavePermissionToUseSQLCLR()
   at Microsoft.SqlServer.Management.Common.ConnectionManager..ctor(SqlConnection sqlConnectionObject)
   at Microsoft.SqlServer.Management.Common.ServerConnection..ctor(SqlConnection sqlConnection)
   at _$tdKdtk9ZsSXpXVEiTFfmYWqJ9GPZiX_O2mtftJYjt5lHMf42kiWL0QYC23ZfuRuQ.ZGHoI4m7B6x8HTJtQXNsd_QmnmiqIyr$D2AcYjhVGu8g()

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

This error (same as originally reported above) also reappeared...

2020-11-10 19:41:32,823 [H2034v05][Thread:40][ERROR] DotNetNuke.Services.Exceptions.Exceptions - System.IO.IOException: The process cannot access the file 'C:\HostingSpaces\[abbv\domain].com\web\DesktopModules\ToSIC_SexyContent\Upgrade\Log\2020-11-11 01-41-32-8237265-5484-8.log.resources' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at ToSic.Sxc.Dnn.Install.DnnInstallLogger.OpenLogFiles() in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\DnnInstallLogger.cs:line 43
   at ToSic.Sxc.Dnn.Install.InstallationController.UpgradeModule(String version) in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Dnn\Install\InstallationController.cs:line 54
   at ToSic.SexyContent.Environment.Dnn7.DnnBusinessController.UpgradeModule(String version) in C:\Projects\2sxc\2sxc\Src\Dnn\ToSic.Sxc.Dnn\Environment\Dnn7\DnnBusinessController.cs:line 89
   at DotNetNuke.Entities.Modules.EventMessageProcessor.UpgradeModule(EventMessage message)

So it seems like some of the architecture changes have tripped up on .NET? Dependecy injection? Does anything above indicate what the problem might be? It does seem like the C# compiler running over and over again is the problem, but what is causing that?

Not sure its applicable, but I double checked anyway, .NET version is 4.8.03761 (Release 528049)

Only other clue I can find is obviously related to the csc.exe starting/stopping (above). If you wait long enough, the pages load (no errors) and you can even get to the PB/Manage, Event Logs and you can see after about an hour, there are no other errors and just the repeating Web Server Updated, Application Started every 20 or so seconds:

[iJungleboy]

The error you're getting is pointing to a SqlClrProvider.

I've never heard of this file before, but when you google it, it seems to be part of the SQL client DLLs used in .net Framework to access the SQL server. It's certainly not something we include in our distribution, so it must be something that's always in a normal windows server installation.

Without being certain, my guess is that you may have an old .net framework installed or something, and need to update it. But I'm not sure - yet think it's not really related to DNN or 2sxc, but something underneath.

[jeremy-farrance]

1. as I said above, and verified, its .NET 4.8+ and very recently updated.
2. why did installing 2sxc trigger this? the site was running fine before (2 years)?
3. this is the 3rd time this has happened installing 2sxc as an upgrade.
4. that error occurred only once about a minute after install, it never recurred.

[iJungleboy]

Since it feels like a very random issue (I don't know anybody else with this) I would suggest some basic things.

My guess is if you flush global cache of compiled asp.net stuff. I'm almost certain your issue will go away. It's usually in some folder like C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\

[jeremy-farrance]

I tried that, seemed like a very good idea. Stopped IIS, removed all the temp/root/{everything}. Started IIS with "slow" site stopped. Made sure 3+ other sites came back up and returned to full speed. All seemed fine. Started and spun up the "slow" site and it is still slow. Still reporting "Web Server Updated" every 20 to 40 seconds. csc.exe running on/off periodically.

I also determined that the SqlClrProvider error was not related. I managed to fix that by including System.Net.Http in web.config in the . No idea what is really going on there, but somehow having that specified for on the fly compiles made that error go away.

So I went a step further. I restored the site from backup, from before (2sxc still at 9.43.2) to ANOTHER server, this one a mostly still clean/recent install of Win Server 2019 and also .NET 4.8+. Got the site running again just fine. Installed 2sxc 11.7.3 and ... slow. csc.exe runs for a bit every minute or so. Same as the other server.

I understand/accept your point of view on the randomness of it. If I do discover more clues I'll drop them here in case it does possibly happen to someone else. Thank you for reading and discussing it with me, cheers!!

[iJungleboy]

Thanks for the update.

If I could do some more guessing, it would be this: Assuming that it happens because of or within 2sxc, then it could be a 2sxc bug or a bug in the DB or in custom razor template/web-api or a combination.

To further narrow it down, I would like to suggest the following:

1. On the portal you're testing, just rename the portals/x/2sxc folder to anything like 2sxc-temp. This will get you a lot of errors, but it then cannot compile the code files lying in that folder.
2. Then start DNN and verify that you're getting the errors, but that the performance hit is gone - I mean basically nothing can be compiled any more etc.
3. If it's still slow, then the amount of influence of 2sxc is very slim - it could still be there (DLLs load), but basically it doesn't have anything left to render. So if it's still happening, it's probably not 2sxc, and you could further verify that by temporarily removing all the ToSic.* DLLs from the bin.
4. If it's fast, then manually move 1 app at a time "back" to the "2sxc" folder till you hit the problem.

[iJungleboy]

BTW: I just saw there was another issue reporting a similar problem and was able to fix it - maybe it helps: #2218

Since this was reported 2x already, I'll re-open in case others have some insights or the same issue.

[jeremy-farrance]

This may surprise you, but as I said, 2sxc is installed, but not used. No Content or App was ever added to a page. This one is an example of a project where we installed it 2sxc ONLY because we wanted ImageResizer (image.jpg?w=440&scale=both)... meaning there is NO 2sxc folder:

I tried a few other odd things, I noticed a few of the DLLs from the /bin folder in the install.zip didn't actually get copied because

Info    Starting Installation - Assembly
Info    Assembly already registered - bin\ToSic.Razor.dll
Info    Assembly already registered - bin\ToSic.Razor.Dnn.dll
Info    Component installed successfully - Assembly

So I manually copied them to bin over the exist files anyhow. Made no difference.

I deleted all the *.pdb files in the /bin folder. Again, no difference.

From what I am seeing. DNN starts up, something triggers a compilation (csc.exe), this fails, though no error is logged that I can find. After a few seconds, it tries again. This repeats in 15 to 20 second intervals until the page is loaded completely. Then the compile stuff stops until the next page request.

I managed to capture the command line that keeps running, not sure how that might be helpful but here it is, The part highlighted in yellow is random every time, but the rest stay identical.

So then I went in to that folder and started watching. The *.cmdline was usually only there for 3 to 5 seconds, but after a few tries I managed to catch it, open in an editor, here is some of the contents (its all one line, 38k, no CRLFs):

/t:library /utf8output 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\3fc6fbbe\003ba87f_f69bd601\ToSic.SexyContent.Razor.dll" 
/R:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Web.Extensions\v4.0_4.0.0.0__31bf3856ad364e35\System.Web.Extensions.dll" 
/R:"C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Smo\13.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.Smo.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\11eb65a9\0044b560_cef2ce01\Evotiva.DNNBackup.FTP.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\f578deb0\00740ed5_037fd301\Microsoft.Data.Tools.Utilities.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\3514ba7b\00519c69_e6ffd401\Dnn.PersonaBar.Extensions.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\3a94e33d\0047e312_e7ffd401\Dnn.PersonaBar.TaskScheduler.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\7a477c80\00af5081_91ddd101\Microsoft.AspNet.SignalR.Core.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\7891cdae\00abcbfb_017fd301\Microsoft.IdentityModel.Clients.ActiveDirectory.Platform.dll" 
/R:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll" 
/R:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Collections\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Collections.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\ffcf24fd\00aeba57_e6ffd401\Dnn.PersonaBar.Sites.dll" 

,,, a few dozen lines skipped

/R:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Web.ApplicationServices\v4.0_4.0.0.0__31bf3856ad364e35\System.Web.ApplicationServices.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\2d68aeef\0069a3b9_037fd301\Microsoft.SqlServer.TransactSql.ScriptDom.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\d5bdad8c\00a13fd6_037fd301\Microsoft.Data.Tools.Schema.Sql.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\e62867d7\00c238e9_d081d601\Connect.Koi.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\049fab09\000e777e_f69bd601\ToSic.Sxc.Dnn.dll" 
/R:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\assembly\dl3\0312d280\00740ed5_037fd301\Microsoft.SqlServer.DacAzureExtensions.dll" 
/R:"C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Drawing\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll" 
/R:"C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.ConnectionInfo\13.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.ConnectionInfo.dll" 
/R:"C:\Windows\assembly\GAC_MSIL\Microsoft.SqlServer.Types\14.0.0.0__89845dcd8080cc91\Microsoft.SqlServer.Types.dll" 
/out:"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\App_Web_ooibradg.dll" 
/debug- /optimize+ /w:4 /nowarn:1659;1699;1701;612;618 
/warnaserror-  
"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\App_Web_ooibradg.31.cs" 
"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\App_Web_ooibradg.3.cs" 
"C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\958fefbb\72ebb12\App_Web_ooibradg.9.cs" 
... then these repeat to the end, about 40+ more lines

[jeremy-farrance]

Tried another thing. Figured out how to turn on .NET Assembly Binding Logging. I did get the logging working, but it only caught one error from another site on the server, not the slow one. So... not sure it was worth ruling out, but it has nothing to do with binding errors. :) https://stackoverflow.com/questions/17681432/how-can-i-enable-assembly-binding-logging

[iJungleboy]

As was mentioned in another thread, even opening a txt file is slow. But if the web.config is renamed/deactivated, it's all fast again.

In this case (if that's the same everywhere) I strongly urge to re-activate the web.config, comment out everything, and step-by-step reactivate parts of the web.config till things suddenly hit slow again.

[MS-BSI]

Yes, that was my debugging method (+ DLL deletion) since 1 hour (because of the slowness).

I found the culprit line of code : <add name="ImageResizingModule" type="ImageResizer.InterceptModule" />

When commented, all is back to the normal ;)

[jeremy-farrance]

I did not see THAT coming. Testing soon...

[iJungleboy]

@MS-BSI Awesome - thanks for the update!!!

I have a vague idea of how that could be. Could you test it?

In one of the later version of 2sxc we upgraded image resizer and added WebP support

  <resizer>
    <pipeline defaultCommands="autorotate.default=true" />
    <plugins>
      <add name="DiskCache" />
      <add name="WebPEncoder" downloadnativedependencies="true" /> <!-- this line -->
    </plugins>
  </resizer>

The mechanism is a bit strange, but it basically will (try to) add another DLL from google to do WebP conversion, and I could imagine that this somehow fails. Maybe the server can't access the internet or whatever - but I'm just guessing.

Could you try to reactivate the image-resizer-module and deactivate the WebP, and if that works, reactivate webP but disable downloadnative... ?

[jeremy-farrance]

Okay, success!!

Here is what I did. On a Dnn 9.03.02 site with unused 2sxc 9.43.2 installed (and no apps, no content app).

• DNNBackup
• Restart (recycled app pool)
• installed 2sxc 11.7.3
• verified slowness
• web.config; changed downloadnativedependencies to false
• tested, still slow
• web.config; removed the add WebPEncoder line
• tested, FIXED

[iJungleboy]

Thanks for finding this and the solution both to @MS-BSI and @jeremy-farrance .

Could you do one more check? If you added the dll which it tries to add yourself - (attached) - could you then re-enable the webp-coder?

libwebp.zip

[jeremy-farrance]

SUCCESS!! Added to /bin/libwebp.dll (says v0.6.0.0), left download...=false, but added the line back to web.config

  <resizer>
    <pipeline defaultCommands="autorotate.default=true" />
    <plugins>
      <add name="DiskCache" />
      <add name="WebPEncoder" downloadnativedependencies="false" />
    </plugins>
  </resizer>

Site is still fast/normal.

[MS-BSI]

Sorry I'm not at the office, I'll run the tests tomorrow if it's still useful ;)

Yes I had spotted the modifications of 11.03 concerning the support of WebP (among other things) which was clearly visible in my web.config (compared to the previous version 10.27.1), hence my tests at this level . Great if a fix is directly applicable !

[MS-BSI]

Hi,

I confirm that adding the libwebp.dll file to the bin folder fixes the problem.

Besides, downloadnativedependencies can remain on true, that doesn't matter since the file is well detected.

Everything is OK, so the file should be able to be added in the next version of 2sxc 👍

Thank you, Mick

[iJungleboy]

That's a really hard issue. The reason we're not including it and allowing auto-install is because the DLL is blocked once it's used, in a way that other DLLs are not. This means that a first install of 2sxc would work, but any upgrade of 2sxc (once that DLL is included) would fail, because that DLL would fail, and then the entire 2sxc-upgrade would get cancelled.

I don't really have a solution for this...

[iJungleboy]

Just fyi: because this is causing too much trouble, we'll drop WebP support again as it's not used yet, and we need to find a more reliable implementation. See #2266

[jeremy-farrance]

I understand and agree. Will you have or leave some docs somewhere on how to add it manually? I think if someone has the need and understands the caveats, its still worth it.

[iJungleboy]

As of now I'll have to push it back a bit - anybody that wants to do this can just google the ImageResizer and the WebP plugin.

For 2sxc we'll probably need to switch to another image resizer in the near future, so I don't want to encourage people to add this, because it could add complexity when we switch.