Steps to Reproduce

Install Forticlient from http://www.forticlient.com/
Clear out your cache from Appdata/Local/Secondlife
Run an HTTP Pipelining enabled viewer.
Explode textures everywhere, followed by crashing a moment later.

Actual Behavior

Upon logging in, the first thing you'll notice is most of the textures around you will fetch improperly, usually a mess of rainbow colors. Shortly there after, after enough of these corrupted textures load in, Secondlife will crash and usually not give you a dmp file to tear apart.

Expected Behavior

To fetch and cache textures properly and not explode all the sculpties and mesh everywhere. Also, not crashing would be nice.

Other information

My computer is directly connected to my modem via ethernet cable. The modem information is listed at the bottom of the Environment field.

The first thing I did when I installed Forticlient was whitelist every single Secondlife related folder I could, Appdata/Local/Secondlife, Appdata/Roaming/Secondlife, Program Files (x86/SecondlifeViewer, and the equivalent folders for all the other viewers installed. I also completely disabled the network sheild as it blocked things in my browser, and I wasn't happy with that. Upon loading Secondlife for the first time, everything seemed perfectly normal (because my cache folder wasn't cleared at the time.) but I would crash randomly, and the crashing eventually started happening much faster each time until it was happening a couple of minutes after login. I soon found the cause when someone walked in front of me that I hadn't seen before. All of her textures were completely corrupted, and a second later, crash. My first instinct was to clear my cache as the cache was obviously corrupted at that point. Upon doing so and loading the viewer again, I crashed before I could even see the world. Logging in again crashed me within a second of seeing the world. What I saw was nothing but corrupted textures on everything that did manage to load. Long story short, I was told to turn off HTTP Pipelining, clear my cache again, and test again. Upon doing so, everything loaded perfectly fine (Albeit very slowly.) and I didn't crash. This was a step in the right direction. It was shortly after this time that my friend Madmac, (The person that suggested Forticlient to me two days prior) skype called me, and I showed him what was going on. He also uses Forticlient, so I sent him a Firestorm nightly that has HTTP Pipelining. Upon him clearing his cache and loading that nightly, he saw the same corruption I was seeing followed swiftly by a crash. He and I both uninstalled Forticlient as that was the only thing our two systems had in common, and after a cache clear and a system reboot, everything was working properly again with HTTP Pipelining enabled. That was when Whirly installed Forticlient herself to test it, and her results were the same, and explained in the comments below.

Attached is a series of log folders numbered in the order they were run in, as well as a description of what I was doing during testing, but I'll explain them here as well. All 6 sessions were done while Forticlient was installed, all the Secondlife folders whitelisted, and the network sheild disabled.
1. This session started with HTTP Piplining enabled, a completely clean cache, and upon logging in, I didn't even get to see the world, it crashed on the black screen.
2. This session also had Piplining enabled, the dirty cache from the last session. Upon logging in, it made it in world, the textures were badly corrupted, followed by another crash.
3. This session had Piplining disabled, and I left the dirty cache from the previous two logins. Upon logging in, the already corrupted textures loaded in corrupted, but it didn't crash this time, I closed Secondlife manually.
4. This session left Piplining disabled, but ran with an empty cache. Upon logging in, the textures and objects around me loaded properly and didn't crash.
5. This session left piplining disabled, kept the dirty cache from number 4, just to prove that the textures that are already fetched load properly from the cache and there was no crash.
6. This session enabled pipelining, kept the non-corrupted cache fetched in the last two sessions. This time, the already properly fetched textures loaded properly, however, any new textures that were not cached loaded in corrupted. There wasn't a crash however.

Attachments

Fig 1 - exploding mesh.png 2015-02-27 00:42:32
Fig 2 - corrupted textures.png 2015-02-27 00:42:32
Fig 3.png 2015-02-27 01:21:48
Greatfox-BUG-8631-logs.rar 2015-02-28 07:51:14
Whirly_Fullstack&_loaded_modules.txt 2015-02-27 02:12:31
Whirly_Log2.log 2015-02-27 01:20:20
Whirly_Logs1.zip 2015-02-27 00:42:32
Whirly_QuickHeal.log 2016-02-09 06:09:48

Links

Original Jira Fields

| Field | Value | | ------------- | ------------- | | Issue | BUG-8631 | | Summary | Complete texture corruption followed by viewer crash, only when HttpPipelining is enabled | | Type | Bug | | Priority | Unset | | Status | Accepted | | Resolution | Accepted | | Reporter | Greatfox Snowpaw (greatfox.snowpaw) | | Created at | 2015-02-26T22:23:15Z | | Updated at | 2016-11-19T21:38:51Z | ``` { 'Business Unit': ['Platform'], 'Date of First Response': '2015-02-26T22:45:13.460-0600', "Is there anything you'd like to add?": 'WIP', 'Severity': 'Unset', 'System': 'SL Viewer', 'Target Viewer Version': 'viewer-development', 'What just happened?': 'Work in progress', 'What were you doing when it happened?': 'WIP', 'What were you expecting to happen instead?': 'WIP', } ```

Greatfox Snowpaw commented at 2015-02-27T04:01:59Z

Will properly file this tomorrow. Please don't delete me.

Monty Linden commented at 2015-02-27T04:45:13Z

Please include information on networking equipment: routers, wifi, tethered cell phone, mifi, etc.

Whirly Fizzle commented at 2015-02-27T05:41:01Z, updated at 2015-02-27T06:42:31Z

Reproduced on Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release) & on a pipelining enabled Firestorm.

Please note that I have never had this problem before using the same computer and same network. The only change I made to my setup was to uninstall AVG antivirus and then install Forticlient antivirus.
I installed Forticlient antivirus from http://www.forticlient.com/
With HttpPipelining enabled on Second Life 3.7.25 (299021), I logged into a region I already had fully cached from before I installed Forticlient.
Everything appeared normal - textures and meshes loaded correctly.
I logged out and deleted the cache folder and then logged back into the same region, still with HTTPPipelining enabled.
All textures & mesh started to load in corrupted - rainbow textures and mesh was just exploding into huge triangles.
The viewer crashed to desktop about 30 seconds after login.
I found that lowering draw distance down to 64m allowed me to stay logged in longer before crashing. I managed a session of a few minutes before crashing and attached the logfile from this session - See Whirly_logs_1.zip
See attached images Fig 1 and Fig 2 showing exploding mesh and corrupted textures.

Since installing Forticlient, when Pipelining is enabled I always have the following warnings in my logs, plus lots of KDU barf:


2015-02-27T05:21:57Z WARNING: LLTextureFetchWorker::callbackHttpGet: data_size = 937 > requested: 600
2015-02-27T05:21:57Z WARNING: LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture d986ff05-1a6f-4b3d-5f46-fe90ac456a14.  Aborting load.
2015-02-27T05:21:57Z WARNING: LLViewerFetchedTexture::updateFetch: d986ff05-1a6f-4b3d-5f46-fe90ac456a14 Fetch failure, setting as missing, decode_priority 2304181.000000 mRawDiscardLevel 32767 current_discard -1 stats 00ce0000
2015-02-27T05:21:57Z WARNING: LLViewerFetchedTexture::setIsMissingAsset: d986ff05-1a6f-4b3d-5f46-fe90ac456a14: Marking image as missing


2015-02-27T05:22:00Z WARNING: LLMeshHandlerBase::onCompleted: Mesh response (bytes [9784..10393]) didn't overlap with request's origin (bytes [0..4095]).
2015-02-27T05:22:00Z WARNING: LLMeshHeaderHandler::processFailure: Error during mesh header handling.  ID:  cc17b450-cf6b-e0fb-df35-1b5517e9e0dd, Reason:  Invalid Content-Range header encountered (Core_4).  Not retrying.
2015-02-27T05:22:00Z WARNING: LLMeshHandlerBase::onCompleted: Mesh response (bytes [9788..10397]) didn't overlap with request's origin (bytes [0..4095]).
2015-02-27T05:22:00Z WARNING: LLMeshHeaderHandler::processFailure: Error during mesh header handling.  ID:  938dc082-128d-649e-94e2-762321040d3e, Reason:  Invalid Content-Range header encountered (Core_4).  Not retrying.

When the viewer crashes, it often does not produce a dmp file - I just get "Secndlife has stopped working"... and the log file just cuts dead. The attached logs do however have a dmp so hopefully a crash report was sent in.
I then relogged with a dirty cache on Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release) with HttpPipelining disabled.
The viewer does not crash when HttpPipelining is disabled.
Any texture or mesh that I had previously seen corrupted when Pipelining was enabled the previous session remains corrupted during this HttpPipelining disabled session but new textures fetched load up correctly.
My logfile from this session didnt have any of the "WARNING: LLTextureFetchWorker::callbackHttpGet: data_size = 937 > requested: 600" or "WARNING: LLMeshHandlerBase::onCompleted: Mesh response (bytes [9784..10393]) didn't overlap with request's origin (bytes [0..4095])." warnings, but it did have lots of KDU blowups.
For example see Fig 3 which shows a region I did not have cached. The only textures that display corrupted are the ones on my avatar that were corrupted the previous session with Pipelining enabled.
I then logged out and deleted my cache folder and logged in with HttpPipelining disabled.
I teleported around the same regions as in the previous session and attached my logfile from this session - see Whirly_Log2
As far as I can tell, the logfile looks healthy - neither of those warnings appeared and no KDU blowups.
The viewer did not crash and all textures and mesh loaded correctly and quickly.

Network Information

Router make: TP-Link
Router Model: TD-W8961ND
Software version: Firmware Version: 3.0.0 Build 120808 Rel.28888, ADSL Firmware Version: FwVer:3.20.17.0_TC3087 HwVer:T14.F7_11.2
IP Address: 88.104.225.131 (not static)
ISP: TalkTalk

Hardwired to router, broadband.


C:\Users\Whirly>tracert login.agni.lindenlab.com

Tracing route to login.agni.lindenlab.com [216.82.54.158]
over a maximum of 30 hops:

1    <1 ms    <1 ms    <1 ms  192.168.1.1
2    19 ms    19 ms    19 ms  88-104-224-1.dynamic.dsl.as9105.com [88.104.224.1]
3    24 ms    24 ms    24 ms  85-210-252-0.dynamic.dsl.as9105.com [85.210.252.0]
4    25 ms    24 ms    25 ms  85-210-252-1.dynamic.dsl.as9105.com [85.210.252.1]
5    30 ms    30 ms    30 ms  host-78-144-9-21.as13285.net [78.144.9.21]
6    31 ms    31 ms    31 ms  host-78-144-13-38.as13285.net [78.144.13.38]
7  1190 ms  1007 ms  1000 ms  unknown.Level3.net [212.187.192.113]
8   166 ms     *      166 ms  ae-1-8.bar1.Phoenix1.Level3.net [4.69.133.29]
9   167 ms     *        *     ae-1-8.bar1.Phoenix1.Level3.net [4.69.133.29]
10   168 ms   167 ms   168 ms  LINDEN-RESE.bar1.Phoenix1.Level3.net [4.53.104.14]
11     *        *        *     Request timed out.
12     *        *        *     Request timed out.
13   167 ms   167 ms   167 ms  login-phx12.agni.lindenlab.com [216.82.54.158]

Trace complete.

System Information


Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release)
Release Notes

CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (3491.97 MHz)
Memory: 16268 MB
OS Version: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: GeForce GTX 750/PCIe/SSE2

Windows Graphics Driver Version: 9.18.0013.4752
OpenGL Version: 4.5.0 NVIDIA 347.52

libcurl Version: libcurl/7.38.0 OpenSSL/1.0.1h zlib/1.2.8
J2C Decoder Version: KDU v7.0
Audio Driver Version: FMOD Ex 4.44.31
Qt Webkit Version: 4.7.1 (version number hard-coded)
Voice Server Version: Not Connected
Built with MSVC version 1600

Whirly Fizzle commented at 2015-02-27T07:06:12Z

Callstack from crash on Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release) with HttpPipelining enabled

Operating system: Windows NT
                  6.1.7601 Service Pack 1
CPU: x86
     GenuineIntel family 6 model 60 stepping 3
     8 CPUs

Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
Crash address: 0x47ebd18f
Assertion: Unknown assertion type 0x00000000

Thread 0 (crashed)
 0  ntdll.dll + 0x2e3be
    eip = 0x77a6e3be   esp = 0x00c0f270   ebp = 0x00c0f2a4   ebx = 0x23109290
    esi = 0x47ebd18b   edi = 0x23109288   eax = 0x00000000   ecx = 0x04b50000
    edx = 0x23109290   efl = 0x00210206
    Found by: given as instruction pointer in context
 1  ntdll.dll + 0x2e022
    eip = 0x77a6e023   esp = 0x00c0f2ac   ebp = 0x00c0f2bc
    Found by: previous frame's frame pointer
 2  kernel32.dll + 0x114ac
    eip = 0x76f014ad   esp = 0x00c0f2c4   ebp = 0x00c0f2d0
    Found by: previous frame's frame pointer
 3  msvcr100.dll!zzz_AsmCodeRange_End + 0x6c
    eip = 0x6cbc016a   esp = 0x00c0f2d8   ebp = 0x00c0f2e4
    Found by: previous frame's frame pointer
 4  msvcr100.dll!_aligned_free [align.c : 472 + 0x7]
    eip = 0x6cc367cc   esp = 0x00c0f2ec   ebp = 0x00c0f2f0
    Found by: call frame info
 5  SecondLifeViewer.exe!LLVBOPool::release(unsigned int,unsigned char volatile *,unsigned int) [llvertexbuffer.cpp : 247 + 0x9]
    eip = 0x01adef3b   esp = 0x00c0f2f8   ebp = 0x00c0f300
    Found by: call frame info
 6  SecondLifeViewer.exe!LLVertexBuffer::destroyGLBuffer() [llvertexbuffer.cpp : 1239 + 0x20]
    eip = 0x01adf064   esp = 0x00c0f308   ebp = 0x00c0f344
    Found by: call frame info
 7  SecondLifeViewer.exe!LLVertexBuffer::~LLVertexBuffer() [llvertexbuffer.cpp : 1044 + 0x4]
    eip = 0x01ae0456   esp = 0x00c0f320   ebp = 0x00c0f344
    Found by: call frame info
 8  SecondLifeViewer.exe!LLVertexBuffer::`vector deleting destructor'(unsigned int) + 0xa
    eip = 0x017bc4eb   esp = 0x00c0f34c   ebp = 0x00c0f350
    Found by: call frame info
 9  SecondLifeViewer.exe!LLPointer<LLVertexBuffer>::unref() [llpointer.h : 156 + 0x15]
    eip = 0x00fcd0a7   esp = 0x00c0f358   ebp = 0x00c0f378
    Found by: call frame info
10  SecondLifeViewer.exe!std::_Tree<std::_Tmap_traits<LLFace *,std::vector<LLPointer<LLVertexBuffer>,std::allocator<LLPointer<LLVertexBuffer> > >,std::less<LLFace *>,std::allocator<std::pair<LLFace * const,std::vector<LLPointer<LLVertexBuffer>,std::allocator<LLPointer<LLVertexBuffer> > > > >,0> >::_Erase(std::_Tree_nod<std::_Tmap_traits<LLFace *,std::vector<LLPointer<LLVertexBuffer>,std::allocator<LLPointer<LLVertexBuffer> > >,std::less<LLFace *>,std::allocator<std::pair<LLFace * const,std::vector<LLPointer<LLVertexBuffer>,std::allocator<LLPointer<LLVertexBuffer> > > > >,0> >::_Node *) [xtree : 1617 + 0x27]
    eip = 0x01129c77   esp = 0x00c0f380   ebp = 0x00c0f394
    Found by: call frame info
11  SecondLifeViewer.exe!LLVolumeGeometryManager::genDrawInfo(LLSpatialGroup *,unsigned int,LLFace * *,unsigned int,int,int,int) [llvovolume.cpp : 5812 + 0x21]
    eip = 0x017d2f17   esp = 0x00c0f39c   ebp = 0x00c0f528
    Found by: call frame info
12  SecondLifeViewer.exe!LLVolumeGeometryManager::rebuildGeom(LLSpatialGroup *) [llvovolume.cpp : 5017 + 0x23]
    eip = 0x017d42af   esp = 0x00c0f530   ebp = 0x00c0f614
    Found by: call frame info
13  SecondLifeViewer.exe!LLSpatialGroup::rebuildGeom() [llspatialpartition.cpp : 396 + 0x11]
    eip = 0x015465ef   esp = 0x00c0f61c   ebp = 0x00c0f66c
    Found by: call frame info
14  SecondLifeViewer.exe!LLPipeline::rebuildGroups() [pipeline.cpp : 3000 + 0x6]
    eip = 0x01828907   esp = 0x00c0f628   ebp = 0x00c0f66c
    Found by: call frame info
15  SecondLifeViewer.exe!display(int,float,int,int) [llviewerdisplay.cpp : 1039 + 0x9]
    eip = 0x016228fb   esp = 0x00c0f674   ebp = 0x00c0f8fc
    Found by: call frame info
16  SecondLifeViewer.exe!LLAppViewer::mainLoop() [llappviewer.cpp : 1454 + 0x20]
    eip = 0x00efde3c   esp = 0x00c0f904   ebp = 0x00c0fad8
    Found by: call frame info
17  SecondLifeViewer.exe!WinMain [llappviewerwin32.cpp : 322 + 0x8]
    eip = 0x01843032   esp = 0x00c0fae0   ebp = 0x00c0fb14
    Found by: call frame info
18  SecondLifeViewer.exe!__tmainCRTStartup [crtexe.c : 547 + 0x1b]
    eip = 0x021dc307   esp = 0x00c0fb1c   ebp = 0x00c0fba8
    Found by: call frame info
19  kernel32.dll + 0x13389
    eip = 0x76f0338a   esp = 0x00c0fbb0   ebp = 0x00c0fbb4
    Found by: call frame info
20  ntdll.dll + 0x39f71
    eip = 0x77a79f72   esp = 0x00c0fbbc   ebp = 0x00c0fbf4
    Found by: previous frame's frame pointer
21  ntdll.dll + 0x39f44
    eip = 0x77a79f45   esp = 0x00c0fbfc   ebp = 0x00c0fc0c
    Found by: previous frame's frame pointer

Callstack from Firestorm 4.7.0 (45028) Feb 25 2015 14:24:48 (Firestorm-Fizzlefire) with OpenSimulator support with HttpPipelining enabled

    ntdll.dll!RtlFreeHeap()  + 0xd0 bytes   
    kernel32.dll!HeapFree()  + 0xa bytes    
>   msvcr100.dll!free(void * pBlock)  Line 51   C
    msvcr100.dll!_aligned_free(void * memblock)  Line 473   C
    FirestormOS-Fizzlefire.exe!LLPrivateMemoryPoolManager::freeMem(LLPrivateMemoryPool * poolp, void * addr)  Line 1919 + 0x9 bytes C++
    FirestormOS-Fizzlefire.exe!LLImageFormatted::~LLImageFormatted()  Line 1334 + 0x19 bytes    C++
    FirestormOS-Fizzlefire.exe!LLImageJ2C::`scalar deleting destructor'()  + 0x14 bytes C++
    FirestormOS-Fizzlefire.exe!LLPointer<LLImageFormatted>::unref()  Line 157   C++
    FirestormOS-Fizzlefire.exe!LLTextureFetchWorker::endWork(int param, bool aborted)  Line 2102 + 0x1f bytes   C++
    FirestormOS-Fizzlefire.exe!LLWorkerClass::checkWork(bool aborting)  Line 370    C++
    FirestormOS-Fizzlefire.exe!LLTextureFetch::getRequestFinished(const LLUUID & id, int & discard_level, LLPointer<LLImageRaw> & raw, LLPointer<LLImageRaw> & aux, LLCore::HttpStatus & last_http_get_status)  Line 2936   C++
    FirestormOS-Fizzlefire.exe!LLViewerFetchedTexture::updateFetch()  Line 1955 C++
    FirestormOS-Fizzlefire.exe!LLViewerTextureList::updateImagesFetchTextures(float max_time)  Line 1090    C++
    FirestormOS-Fizzlefire.exe!LLViewerTextureList::updateImages(float max_time)  Line 725 + 0xb bytes  C++
    FirestormOS-Fizzlefire.exe!display(int rebuild, float zoom_factor, int subfield, int for_snapshot)  Line 890 + 0xc bytes    C++
    FirestormOS-Fizzlefire.exe!LLAppViewer::mainLoop()  Line 1704   C++
    FirestormOS-Fizzlefire.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow)  Line 329  C++
    FirestormOS-Fizzlefire.exe!__tmainCRTStartup()  Line 547 + 0x25 bytes   C
    kernel32.dll!BaseThreadInitThunk()  + 0xd bytes 
    ntdll.dll!RtlUserThreadStart()  + 0x21 bytes

Whirly Fizzle commented at 2015-02-27T07:14:24Z

@Monty - let me know if you want Wiresharks or any other tests done. I want to get this Forticlient crap off my computer :D

MadMac Galaxy commented at 2015-02-27T20:47:52Z

@Whirly Fizzle, never doubt the Forticlient! XD

Whirly Fizzle commented at 2015-02-28T03:19:00Z

After uninstalling Forticlient & a cache folder enema - behaviour is back to normal when running Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release) with HttpPipelining enabled :)

Greatfox Snowpaw commented at 2015-02-28T12:54:06Z

Information and logs provided from my testing. I've also now uninstalled Forticlient and cleared my cache folders and everything is running fine with HttpPipelining enabled.

Whirly Fizzle commented at 2015-02-28T22:30:26Z

I had to reinstall Forticlient to get Wiresharks for Nicky on Firestorm, so I did a capture on the LL viewer too - Second Life 3.7.25 (299021) Feb 13 2015 15:21:02 (Second Life Release) Session is only ~ 1 min long and then the viewer crashed. Plenty of those warnings in the logs though.

Wireshark & log: https://drive.google.com/a/phoenixviewer.com/file/d/0B37KRBZuDSmIbWJrcXB5NWdUV1k/view?usp=sharing QAModeHttpTrace set to 1 Clean cache.

MadMac Galaxy commented at 2015-03-03T13:41:50Z

Solution to the Forticlient bug. Disabling proxy in the advanced settings under file -> settings fixes it. Waiting on Futashy to post confirmation. Special thanks to kolawale_FTNT on the Fortinet forum, for presenting the solution.

Whirly Fizzle commented at 2015-03-04T00:23:11Z

Confirmed that disabling FortiProxy under File -> Settings fixes the problem after a cache folder enema. FortiProxy is enabled by default.

Greatfox (issue filer) also confirmed this fixed the problem on the Firestorm JIRA:

Futashy added a comment - Yesterday

MadMac's fix in the above comment is confirmed to fix the issue entirely. No corrupted textures and no crash after disabling the proxy.

Whirly Fizzle commented at 2015-03-08T00:31:33Z, updated at 2015-03-08T00:31:52Z

Norman Antivirus Security Suite also causes this same crash when HttpPipelining is enabled, though it takes longer to crash (about half an hour as opposed to a few minutes) and there is very little visible texture or mesh corruption - see BUG-8711

Whirly Fizzle commented at 2015-08-19T16:01:37Z

This problem also affects SOPHOS UTM 9 stateful firewall.

Copying over information from a Firestorm ticket:

Sione Lomu added a comment - 9 hours ago

Seeing the exact same thing on 4.7.3.47323 x64. Had it on the beta and rolled back hoping it wouldn't be in release.

No Forticlient or Norton's here. Affecting all PC's behind SOPHOS UTM 9 stateful firewall with all outgoing protocols allowed.

Disabled Intrusion Prevention and Advanced Threat Detection as I thought the packet inspection might be causing it. Still happened.

This is a showstopper.

newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 1106 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture d7b2c342-6736-3270-1f12-313561b679ff.  Aborting load.
newview/llviewertexture.cpp(2061) : 2015-08-19T06:06:19Z WARNING: LLViewerFetchedTexture::updateFetch: d7b2c342-6736-3270-1f12-313561b679ff Fetch failure, setting as missing, decode_priority 7900023.500000 mRawDiscardLevel 32767 current_discard -1 stats 00ce0000
newview/llviewertexture.cpp(2274) : 2015-08-19T06:06:19Z WARNING: LLViewerFetchedTexture::setIsMissingAsset: d7b2c342-6736-3270-1f12-313561b679ff: Marking image as missing
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 4734 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture faa82d88-0e7f-6976-0b0c-fd635d285618.  Aborting load.
llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:19Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE: KDU Error: Kakadu Core Error:

llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:19Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE: KDU Error: Illegal inclusion tag tree encountered while decoding a packet header.  This problem can arise if empty packets are used (i.e., packets whose first header bit is 0) and the value coded by the inclusion tag tree in a subsequent packet is not exactly equal to the index of the quality layer in which each code-block makes its first contribution.  Such an error may arise from a mis-interpretation of the standard.  The problem may also occur as a result of a corrupted code-stream.  Try re-opening the image with the resilient mode enabled.
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: c866dd2b-896b-64ae-2f17-3459321f9abe Discard: 0
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 00f2e9f5-9404-824a-8b57-88b06a89d775 Discard: 0
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::doWork: c866dd2b-896b-64ae-2f17-3459321f9abe: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::doWork: 00f2e9f5-9404-824a-8b57-88b06a89d775: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 19530e28-aa16-17f6-811b-e308897bc2c0 Discard: 2
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:19Z WARNING:#Texture LLTextureFetchWorker::doWork: 19530e28-aa16-17f6-811b-e308897bc2c0: Decode of cached file failed (removed), retrying
newview/llviewertexture.cpp(2061) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::updateFetch: faa82d88-0e7f-6976-0b0c-fd635d285618 Fetch failure, setting as missing, decode_priority 4380056.500000 mRawDiscardLevel 32767 current_discard -1 stats 00ce0000
newview/llviewertexture.cpp(2274) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::setIsMissingAsset: faa82d88-0e7f-6976-0b0c-fd635d285618: Marking image as missing
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 3316 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 0c760aa1-1514-4c73-cf1c-6f432e7e0cbf.  Aborting load.
newview/llviewertexture.cpp(2061) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::updateFetch: 0c760aa1-1514-4c73-cf1c-6f432e7e0cbf Fetch failure, setting as missing, decode_priority 2656181.000000 mRawDiscardLevel 32767 current_discard -1 stats 00ce0000
newview/llviewertexture.cpp(2274) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::setIsMissingAsset: 0c760aa1-1514-4c73-cf1c-6f432e7e0cbf: Marking image as missing
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 6758 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 4b390c6f-6145-20e3-15af-9a753c9db35b.  Aborting load.
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 40d58ca4-c106-697e-54ed-1bac5bb752d5 Discard: 0
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 40d58ca4-c106-697e-54ed-1bac5bb752d5: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 3386 > requested: 600
llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:20Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE: KDU Error: Missing or out-of-sequence tile-parts for tile number 
llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:20Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE: KDU Error: 0
llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:20Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE: KDU Error:  in code-stream!
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 57c6793b-d3de-be32-e023-783ef572a737 Discard: 0
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 4ff7a913-6237-23eb-bbeb-1e4b16fc70be.  Aborting load.
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 57c6793b-d3de-be32-e023-783ef572a737: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 4b76d29d-c535-cc11-a27b-86ff06aeecaf Discard: 0
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 4b76d29d-c535-cc11-a27b-86ff06aeecaf: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 3661 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 6e80ab7a-5ea0-f091-44f8-c9d6b909719f.  Aborting load.
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 6b9c2ce4-7d8b-6f5a-f503-0cadc1d98a0f Discard: 0
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 6b9c2ce4-7d8b-6f5a-f503-0cadc1d98a0f: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 5535 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 72f168c6-adb4-cfcd-e8d1-856cd51df67d.  Aborting load.
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 875 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 78c83043-d379-0538-ab0f-4d32c8522435.  Aborting load.
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 920 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 4bae0ac1-1bba-134b-9588-96a0c2514c0b.  Aborting load.
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 6657 > requested: 4734
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 7c97f066-a654-09bd-5012-377b15296243.  Aborting load.
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 65b7fca0-f655-eac6-fb04-74e12190403b Discard: 0
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 7258 > requested: 937
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 65b7fca0-f655-eac6-fb04-74e12190403b: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 81ad09bd-fb79-7684-93e1-4d09af679fe3.  Aborting load.
newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 67ae8abc-3b14-1226-225d-9291955efe9f Discard: 0
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 67ae8abc-3b14-1226-225d-9291955efe9f: Decode of cached file failed (removed), retrying
llkdu/llimagej2ckdu.cpp(180) : 2015-08-19T06:06:20Z INFO:#LLImageJ2CKDU LLKDUMessageError::put_text: ONCE (10th time seen): KDU Error: Kakadu Core Error:

newview/lltexturefetch.cpp(2444) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: 840ea7eb-ee35-902f-88e3-c5603422f3a9 Discard: 3
newview/lltexturefetch.cpp(1888) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: 840ea7eb-ee35-902f-88e3-c5603422f3a9: Decode of cached file failed (removed), retrying
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 194645 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 150b7da8-64fb-b444-004e-a50fa52e7784.  Aborting load.
newview/llviewertexture.cpp(2061) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::updateFetch: 150b7da8-64fb-b444-004e-a50fa52e7784 Fetch failure, setting as missing, decode_priority 4380488.000000 mRawDiscardLevel 32767 current_discard -1 stats 00ce0000
newview/llviewertexture.cpp(2274) : 2015-08-19T06:06:20Z WARNING: LLViewerFetchedTexture::setIsMissingAsset: 150b7da8-64fb-b444-004e-a50fa52e7784: Marking image as missing
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 2331 > requested: 600
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 85d09bda-700a-55a7-3d21-412405f57235.  Aborting load.
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 2971 > requested: 937
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 7481d0e4-65b4-bdcf-14b4-cc6f85e84132.  Aborting load.
newview/lltexturefetch.cpp(2347) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 3497 > requested: 937
newview/lltexturefetch.cpp(1742) : 2015-08-19T06:06:20Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture 8c6c5528-6d81-6a03-0ae7-118f172c188a.  Aborting load.

Sione Lomu added a comment - 7 hours ago

After some more investigation I found out it was the web filtering system in Sophos UTM that was causing the issue. Disabling it fixed the problem. The Web Filter is a transparent proxy and content filter. Disabling it permanently is not an option as it protects the whole network.

Basically, what I can see is that HTTP pipeline texture data is getting messed up when passing through proxy servers/content filters and KDU is flipping out and crashing the viewer. Not sure what the solution is here, I can probably add exceptions to my UTM but that doesn't fix the problem. More people are going to encounter this problem and not know why it is happening.

Sione Lomu added a comment - 5 hours ago

Ok I see textures are now coming from local Akamai caches. Can't really bypass their ip range for security reasons. This is now a major issue for me.

Ref: Comments on FIRE-15625 starting HERE

Panda antivirus is also causing this problem for some users (not everyone & we can't reproduce it internally with Panda). Panda users are fixed by disabling Pipelining. The problem on Forticlient & Norman AV security suite can be reproduced internally by multiple people.

We are getting a new spate of this issue in the Firestorm 4.7.3 release because many are updating straight from the old 4.6.9 release which didn't have Pipelining.

Is this problem something that the various antivirus/proxy software devs need to fix or is it a bug in the viewer?

Whirly Fizzle commented at 2015-08-19T16:09:34Z, updated at 2015-08-19T16:10:04Z

@Monty Do you want affected users (like Sione above) to file a new BUG report with LL viewer logs if they are using antivirus/Firewall software other than Forticlient (BUG-8631) & Norman Antivirus suite (BUG-8711)?

Monty Linden commented at 2015-08-19T16:53:19Z

The anti-virus companies are going to have to address this for a stable, future-proof experience. A Jira-per-company might help hold their attention with some pointers in this one and other places (fs jira, slu, etc.). And of course, hit the companies' support systems as well and point them in here. They should know better at this point but...

Once they look at traffic to the CDN, it should be obvious what's gone wrong. But we'll have to get their attention first. Maybe time for a Wiki/KB article.

Whirly Fizzle commented at 2015-08-21T06:10:13Z

Ok cheers Monty, I'll pass that on.

Further FS user comments:

Rich added a comment - 2 days ago - edited

I can confirm Sione's findings.

Corporate network sits behind a Sophos/Astaro UTM 9 device. Web Filtering is enabled and operating in transparent mode. Beginning with 4.7.1, textures and mesh objects (rigged and non-rigged) appeared corrupt. Rolling back to 4.6.9 resolved the issue. Installed 4.7.3 this morning and the same issue returned.

I can also confirm that disabling Web Filtering in the UTM device while leaving HTTP Pipeling enabled will resolve the problem, but that's not an option for us either. Leaving Web Filtering enabled and disabling HTTP Pipelining as described by Whirly resolves the problem (after a cache clear).

Edited to add: HTTP Pipelining seems to have been broken with the 9.3 release of UTM. https://ubb.sophos.com/gateway-products/web-protection-web-filtering-application-visibility-control/55213-http-pipelining-broken-after-upgrade-utm-9-3-a.html

Rich's link won't load for me,but this one does: https://www.astaro.org/gateway-products/web-protection-web-filtering-application-visibility-control/55213-http-pipelining-broken-after-upgrade-utm-9-3-a.html

Sione Lomu added a comment - 2 days ago

Thanks Whirly, that (disabling HttpPipelining) fixes it. Fingers crossed that the ability to disable it doesn't become deprecated or it's going to be game over for quite a few people.

I'll just disable it for now and hope one day Sophos will sort it out and I can turn it back on. I would have made bypass exceptions for it if the textures weren't coming from my ISP's Akamai caches, but letting everything in from them is a huge security hole.

From what Rich linked... It appears Sophos should actually support it and it's a bug. Those other local machine proxies may never support it. So this could be an ongoing issue for quite some time for some people.

Whirly Fizzle commented at 2015-10-01T16:08:21Z

Another antivirus found to cause problems when Pipelining is enabled is IObit Malware Fighter - http://www.iobit.com/en/malware-fighter.php

Full details on http://jira.phoenixviewer.com/browse/FIRE-17040 This user was experiencing frequent system crashes on both the LL viewer and Firestorm viewer. She could run a pre-Pipelining version of Firestorm without any problems. Disabling HttpPipelining stopped the system crashes. Uninstalling IObit Malware Fighter allowed Pipelining to be left enabled in the viewers without triggering a system crash & everything appears to be fine now.

Here is the bug check for a system crash while running the LL viewer with IObit Malware Fighter installed and HttpPipelining enabled. Note that the faulting process, IMF.exe, is part of IObit Malware Fighter.

Loading Dump File [C:\Users\Whirly\Desktop\MEMORY.DMP]
Kernel Bitmap Dump File: Only kernel address space is available

************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       SRV*f:\localsymbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*f:\localsymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 8 Kernel Version 9600 MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Built by: 9600.17936.amd64fre.winblue_ltsb.150715-0840
Machine Name:
Kernel base = 0xfffff803`61a8e000 PsLoadedModuleList = 0xfffff803`61d637b0
Debug session time: Tue Sep 22 06:12:32.049 2015 (UTC + 1:00)
System Uptime: 1 days 1:31:14.743
Loading Kernel Symbols
...............................................................
................................................................
..................................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`7ffdf018).  Type ".hh dbgerr001" for details
Loading unloaded module list
................................
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck D1, {8, 2, 0, fffff800fe612ed9}

*** ERROR: Module load completed but symbols could not be loaded for UrlFilter.sys
Probably caused by : NETIO.SYS ( NETIO!StreamInvokeCalloutAndNormalizeAction+59 )

Followup: MachineOwner
---------

6: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000008, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff800fe612ed9, address which referenced memory

Debugging Details:
------------------

READ_ADDRESS: unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSizeOfNonPagedPoolInBytes
 0000000000000008 

CURRENT_IRQL:  2

FAULTING_IP: 
NETIO!StreamInvokeCalloutAndNormalizeAction+59
fffff800`fe612ed9 488b7008        mov     rsi,qword ptr [rax+8]

DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT

BUGCHECK_STR:  AV

PROCESS_NAME:  IMF.exe

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

TRAP_FRAME:  ffffd000211eb810 -- (.trap 0xffffd000211eb810)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=ffffd000211eba40
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff800fe612ed9 rsp=ffffd000211eb9a0 rbp=00000000000000e9
 r8=0000000000000000  r9=0000000000000000 r10=fffff800fe666b80
r11=ffffe00073f889b0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
NETIO!StreamInvokeCalloutAndNormalizeAction+0x59:
fffff800`fe612ed9 488b7008        mov     rsi,qword ptr [rax+8] ds:00000000`00000008=????????????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff80361be94e9 to fffff80361bdd9a0

STACK_TEXT:  
ffffd000`211eb6c8 fffff803`61be94e9 : 00000000`0000000a 00000000`00000008 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
ffffd000`211eb6d0 fffff803`61be7d3a : 00000000`00000000 ffffd000`211ebb28 00000000`00000000 00000000`00000003 : nt!KiBugCheckDispatch+0x69
ffffd000`211eb810 fffff800`fe612ed9 : ffffe000`73f88a80 ffffe000`73f889b0 ffffd000`211ebaf0 ffffe000`73f88a80 : nt!KiPageFault+0x23a
ffffd000`211eb9a0 fffff800`fe61298f : 00000000`00000000 ffffe000`73f889b0 ffffd000`211ebb61 00000000`00000000 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x59
ffffd000`211eba70 fffff800`fe61d0c2 : ffffe000`6b400014 fffff801`0229d1a0 ffffd000`00000001 ffffe000`73c8aaa0 : NETIO!StreamProcessCallout+0x76f
ffffd000`211ebbb0 fffff800`fe603569 : ffffe000`70e20014 ffffe000`73c8aaa0 ffffe000`70e25350 ffffd000`211ec420 : NETIO!ProcessCallout+0x972
ffffd000`211ebd20 fffff800`fe602270 : 00000000`00000000 ffffd000`211ec020 ffffd000`211ec478 00000001`ffffffff : NETIO!ArbitrateAndEnforce+0x2c9
ffffd000`211ebf20 fffff800`fe64ff00 : ffffe000`6b0247d0 ffffd000`211ec4b1 ffffe000`6c9ec3c0 fffff800`fe626179 : NETIO!KfdClassify+0x831
ffffd000`211ec3e0 fffff800`fe64f935 : 00000000`00000000 ffffe000`70e8a902 ffffd000`211ec588 fffff800`ff0a8350 : NETIO!StreamInternalClassify+0x110
ffffd000`211ec500 fffff800`fe64ca48 : 00000000`00000014 ffffe000`757a17b0 00000000`00000000 00000000`00000144 : NETIO!StreamInject+0x1fd
ffffd000`211ec5d0 fffff800`ff570ace : ffffe000`757a17b0 00000000`00000144 00000000`00000000 ffffd000`211ec600 : NETIO!FwppStreamInject+0x138
ffffd000`211ec660 fffff801`02df8c44 : ffffe000`7a41e690 00000000`00000000 00000000`00000000 fffff803`61b22b9a : fwpkclnt!FwpsStreamInjectAsync0+0xfa
ffffd000`211ec6c0 fffff801`02df8d74 : 00000000`00000002 00000000`000098c1 ffffe000`7545a010 ffffd000`00000000 : UrlFilter+0x1c44
ffffd000`211ec760 fffff803`61e47140 : 00000000`00000002 00000000`000036dc ffffd000`211ec841 ffffe000`6ad903c0 : UrlFilter+0x1d74
ffffd000`211ec7c0 fffff803`61f3221d : 00000000`00000005 ffffd000`211ecb80 00000000`00000499 00000000`00000000 : nt!IopSynchronousServiceTail+0x160
ffffd000`211ec890 fffff803`61efd54a : e000719f`7ed0effd 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xd2d
ffffd000`211eca20 fffff803`61be91b3 : 00000000`00000102 ffffd000`211ecad8 00000000`00000000 ffffc001`00000001 : nt!NtDeviceIoControlFile+0x56
ffffd000`211eca90 00000000`770c2352 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0c13f128 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x770c2352

STACK_COMMAND:  kb

FOLLOWUP_IP: 
NETIO!StreamInvokeCalloutAndNormalizeAction+59
fffff800`fe612ed9 488b7008        mov     rsi,qword ptr [rax+8]

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  NETIO!StreamInvokeCalloutAndNormalizeAction+59

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: NETIO

IMAGE_NAME:  NETIO.SYS

DEBUG_FLR_IMAGE_TIMESTAMP:  546029c5

BUCKET_ID_FUNC_OFFSET:  59

FAILURE_BUCKET_ID:  AV_NETIO!StreamInvokeCalloutAndNormalizeAction

BUCKET_ID:  AV_NETIO!StreamInvokeCalloutAndNormalizeAction

ANALYSIS_SOURCE:  KM

FAILURE_ID_HASH_STRING:  km:av_netio!streaminvokecalloutandnormalizeaction

FAILURE_ID_HASH:  {c2ca2d1f-cfdc-88d5-c7bc-7693b8f0de04}

Followup: MachineOwner
---------

6: kd> lmvm NETIO
start             end                 module name
fffff800`fe600000 fffff800`fe678000   NETIO      (pdb symbols)          C:\ProgramData\dbg\sym\netio.pdb\873BD5E25CFD43A2A47494662C917A872\netio.pdb
    Loaded symbol image file: NETIO.SYS
    Image path: \SystemRoot\system32\drivers\NETIO.SYS
    Image name: NETIO.SYS
    Timestamp:        Mon Nov 10 02:58:13 2014 (546029C5)
    CheckSum:         00076D9B
    ImageSize:        00078000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Monty Linden commented at 2015-10-01T22:56:47Z

Wow, it bugchecks windows. Impressive. On the plus side, that is guaranteed to grab someone's attention if a bug report is filed with IObit. Including black hats....

Whirly Fizzle commented at 2016-01-12T15:15:02Z

Updated Sophos bug report link https://community.sophos.com/products/unified-threat-management/f/55/t/46641

Whirly Fizzle commented at 2016-01-25T00:26:01Z

Add Webroot Antivirus to the list. This problem started with Webroot ~ 2 weeks ago with a Webroot update. Previously this was not a known problem with Webroot at all. Firestorm support have seen about 15 users affected so far (to my knowledge, there's probably more cases). Users have confirmed the texture/mesh corruption reproduces on any Pipelining enabled viewer and as with Forticlient, the viewer is also very crashy when Pipelining is enabled.

Disabling Webroot or Disabling Pipelining in the viewer both fix the problem.

Ref: http://jira.phoenixviewer.com/browse/FIRE-17661

Whirly Fizzle commented at 2016-01-30T23:28:42Z

@Monty There are viewer logs on BUG-11314 from a user suffering from the new Webroot+Pipelining problem.

Whirly Fizzle commented at 2016-02-09T11:09:26Z, updated at 2016-02-09T12:05:11Z

And we have a new contender!

Quick Heal Total Security 16.00 for 64 bit Windows: http://www.quickheal.com/quick-heal-total-security 30 day free trial available.

This was reported to Firestorm support by Diageron Resident. I can reproduce it.

Behaviour very much like Forticlient. Extreme corrupted mesh & textures followed by a viewer crash in ntdll.dll!RtlFreeHeap as soon as you enter an area with a lot of uncached textures.

Quick Heal Total Security running on out of the box settings.
Reproduced on LL viewer Second Life 4.0.1 (310054) Jan 14 2016 18:17:17 (Second Life Release)
Corrupted textures & my cats head exploded: http://prntscr.com/a0xti7
Whirly_QuickHeal.log attached from a session on LL 4.0.1 (310054) with clean cache.

Usual barf in logs.

Textures:


2016-02-09T10:57:47Z INFO: LLKDUMessageError::put_text: KDU Error: Kakadu Core Error:

2016-02-09T10:57:47Z INFO: LLKDUMessageError::put_text: KDU Error: Illegal inclusion tag tree encountered while decoding a packet header.  This problem can arise if empty packets are used (i.e., packets whose first header bit is 0) and the value coded by the inclusion tag tree in a subsequent packet is not exactly equal to the index of the quality layer in which each code-block makes its first contribution.  Such an error may arise from a mis-interpretation of the standard.  The problem may also occur as a result of a corrupted code-stream.  Try re-opening the image with the resilient mode enabled.
2016-02-09T10:57:47Z WARNING:#Texture LLTextureFetchWorker::callbackDecoded: DECODE FAILED: f1a831c7-986a-1d6d-8488-a1c03a1d6674 Discard: 0
2016-02-09T10:57:47Z WARNING:#Texture LLTextureFetchWorker::doWork: f1a831c7-986a-1d6d-8488-a1c03a1d6674: Decode of cached file failed (removed), retrying
2016-02-09T10:57:47Z WARNING:#Texture LLTextureFetchWorker::callbackHttpGet: data_size = 2305 > requested: 1449
2016-02-09T10:57:47Z WARNING:#Texture LLTextureFetchWorker::doWork: Partial HTTP response produces break in image data for texture f6834e50-7f42-6842-d3a4-f37e3a765a4b.  Aborting load.

Mesh:


2016-02-09T10:57:44Z WARNING:#Mesh LLMeshHeaderHandler::processFailure: Error during mesh header handling.  ID:  04ab8b4f-67ef-ded5-a287-43bdb1c19794, Reason:  Invalid Content-Range header encountered (Core_4).  Not retrying.
2016-02-09T10:57:44Z WARNING:#Mesh LLMeshLODHandler::processData: Error during mesh LOD processing.  ID:  ac1bdc70-849c-8115-7daa-640ea5d572fb, Unknown reason.  Not retrying.

Viewer crashed as soon as I TP'd into a texture rich area. I cant find the symbol files for that LL viewer build so here's the callstack from the Crash on Firestorm:


ntdll.dll!RtlFreeHeap() Unknown
kernel32.dll!HeapFree() Unknown
msvcr120.dll!free(void * pBlock) Line 51 C
msvcr120.dll!_aligned_free(void * memblock) Line 473 C
> FirestormOS-Fizzlefire.exe!LLPrivateMemoryPoolManager::freeMem(LLPrivateMemoryPool  poolp, void  addr) Line 1916 C++
FirestormOS-Fizzlefire.exe!LLImageFormatted::~LLImageFormatted() Line 1881 C++
FirestormOS-Fizzlefire.exe!LLImageJ2C::`scalar deleting destructor'(unsigned int) C++
FirestormOS-Fizzlefire.exe!LLPointer<LLImageFormatted>::unref() Line 157 C++
FirestormOS-Fizzlefire.exe!LLPointer<LLImageFormatted>::operator=(LLImageFormatted * ptr) Line 101 C++
FirestormOS-Fizzlefire.exe!LLTextureFetchWorker::doWork(int param) Line 1891 C++
FirestormOS-Fizzlefire.exe!LLWorkerThread::WorkRequest::processRequest() Line 186 C++
FirestormOS-Fizzlefire.exe!LLQueuedThread::processNextRequest() Line 450 C++
FirestormOS-Fizzlefire.exe!LLQueuedThread::run() Line 516 C++
FirestormOS-Fizzlefire.exe!LLThread::staticRun(apr_thread_t  apr_threadp, void  datap) Line 139 C++
msvcr120.dll!_callthreadstartex() Line 376 C
msvcr120.dll!_threadstartex(void * ptd) Line 354 C
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown

Edit to add, found symbols, heres the callstack from LL viewer.


> ntdll.dll!@RtlpCreateSplitBlock@28()  + 0x1df bytes 
ntdll.dll!@RtlpAllocateHeap@24()  + 0x34d bytes 
ntdll.dll!_RtlAllocateHeap@12()  + 0x4fad bytes 
msvcr120.dll!malloc(unsigned int size)  Line 92 + 0x3b bytes    C
msvcr120.dll!operator new(unsigned int size)  Line 59 + 0x8 bytes   C++
SecondLifeViewer.exe!std::vector<char,std::allocator<char> >::_Buy(unsigned int) [vector : 1557 + 0x6]
      SecondLifeViewer.exe!std::vector<unsigned char,std::allocator<unsigned char> >::vector<unsigned char,std::allocator<unsigned char> >(std::vector<unsigned char,std::allocator<unsigned char> > const &) [vector : 722 + 0xb]
      SecondLifeViewer.exe!LLSD::operator std::vector<unsigned char,std::allocator<unsigned char> >() [llsd.h : 264 + 0x19]
      SecondLifeViewer.exe!LLVolume::unpackVolumeFaces(std::basic_istream<char,std::char_traits<char> > &,int) [llvolume.cpp : 2407 + 0x2a]
      ntdll.dll + 0x73145
      SecondLifeViewer.exe!LLMeshRepoThread::lodReceived(LLVolumeParams const &,int,unsigned char *,int) [llmeshrepository.cpp : 1754 + 0x19]
      SecondLifeViewer.exe!LLMeshRepoThread::fetchMeshLOD(LLVolumeParams const &,int) [llmeshrepository.cpp : 1630 + 0x15]
      SecondLifeViewer.exe!LLMeshRepoThread::run() [llmeshrepository.cpp : 896 + 0x11]
      SecondLifeViewer.exe!LLThread::staticRun(apr_thread_t *,void *) [llthread.cpp : 132 + 0x7]
      SecondLifeViewer.exe!apr_thread_yield + 0x2f
msvcr120.dll!malloc(unsigned int size)  Line 92 + 0x3b bytes

Disabling Pipelining or disabling Quick Heal Total Security + cache clear to get rid of corrupted cached textures fixes the problem.


Second Life 4.0.1 (310054) Jan 14 2016 18:17:17 (Second Life Release)
Release Notes

CPU: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (3491.96 MHz)
Memory: 16268 MB
OS Version: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601)
Graphics Card Vendor: NVIDIA Corporation
Graphics Card: GeForce GTX 750/PCIe/SSE2

Windows Graphics Driver Version: 10.18.0013.6175
OpenGL Version: 4.5.0 NVIDIA 361.75

libcurl Version: libcurl/7.38.0 OpenSSL/1.0.1h zlib/1.2.8
J2C Decoder Version: KDU v7.2
Audio Driver Version: FMOD Ex 4.44.31
LLCEFLib/CEF Version: 1.5.1-(CEF-WIN-3.2526.1347-32)
Voice Server Version: Not Connected
Built with MSVC version 1800

Whirly Fizzle commented at 2016-02-10T01:28:28Z

Firestorm JIRA issue for Quick Heal: http://jira.phoenixviewer.com/browse/FIRE-17792

Whirly Fizzle commented at 2016-02-13T03:27:01Z, updated at 2016-02-13T03:50:57Z

Since Quick Heal has been installed on my system I've also been seeing textures messed up on various websites when running Firefox web browser with network.http.pipelining.aggressive set to TRUE. I haven't seen any rainbow/corrupted textures in Firefox but several times now I've seen textures getting swapped around & replaced with different textures that appear elsewhere on the page. Example: Observed: http://prnt.sc/a2lf4n Expected: http://prnt.sc/a2lfbs

So far this hasn't reproduced when using Chrome web browser.

I have been unable to reproduce problems with map tiles at http://maps.secondlife.com/ when using Firefox with Pipelining. It's pretty hard to spot if map tiles are getting swapped around though so this isn't a good test.

I have been able to reproduce swapped around map tiles on Open Street Maps with Pipelining enabled on Firefox - using the same repro as given in the comments on issue BUG-11314, however it's too intermittent to be a good test for Pipelining problems outside of the SL viewer. Example of messed up map tiles in Firefox on open street maps with network.http.pipelining.aggressive set to TRUE: http://prnt.sc/a2l7b0

Whirly Fizzle commented at 2016-02-14T16:14:33Z

I didn't test this one yet but users reported that another security software that causes this problem is "G-Buster Browser Defense" which is some Brazillian anti-phising thing that most Brazillian banks require you to install to use their online banking.

Whirly Fizzle commented at 2016-03-13T01:53:18Z

Add Iobit Advanced system care to the list. http://www.iobit.com/en/advancedsystemcarefree.php

Usual symptoms, corrupted textures & mesh followed swiftly by a viewer crash in "ntdll!RtlFreeHeap". Same log barf as seen with all the other problem software. Had 2 cases since Friday come into FS support. Disabling Pipelining "fixes" them.

Whirly Fizzle commented at 2016-11-19T21:38:52Z

Sophos have finally fixed this problem.

Chotaire Seelowe added a comment - Yesterday

Finally, in Sophos XG SFOS 16.* this issue is fixed. There was also another issue with meshes to suddenly stop loading and requiring a relog, this is also finally eradicated. If anyone is running Sophos XG, upgrade (SFOS 15 cannot be updated to 16, you need to reinstall).

secondlife / jira-archive

[BUG-8631] Complete texture corruption followed by viewer crash, only when HttpPipelining is enabled #16223

Steps to Reproduce

Actual Behavior

Expected Behavior

Other information

Attachments

Links

Related