Startup is very slow on HDD

[zdland]

This is related to #358 . I'm reopening this since I've narrowed down the cause of it

Filemon.zip

The first startup of Supermium after each system reboot is very slow (Filemon1.log, you can see that it takes about a minute). Subsequent Supermuim startups takes only a few seconds (Filemon2.log).

This is not related to number of cores. My physical machine has 4 cores and I tried running in physical machine, running in VM with all cores or restricted to one core, and the results are all the same.

It seems the majority of time is spent reading chrome.dll. In Filemon1.log, note that from 22:40:14 to 22:40:17, it spent at least 3 seconds just reading chrome.dll! This definitely looks wrong to me. Also notice that the chrome.dll reads are in very small thunks and in random order. Why is that? In this manner the system is likely unable to do effective caching.

In contrast on subsequent startups (see Filemon2.log) the chrome.dll reads are sequential and in bigger thunks. I believe this difference is the cause to difference in startup times, but I don't know why the pattern in Filemon1.log (maybe parallel loading in multiple threads?)

I don't know if the behaviour is the same on Chrome 122 since I can't install it on my OS.

I'm using 122.0.6261.85 official build (x86) with progwrp 1.1.0.5002 Windows Server 2003 SP2 (x86)

[IDA-RE-things]

I tnink (and this is as OS works) : this is bacause OS reads DLL file from HDD to map it in memory. OS loads 4-kb DLL pages if code executed/read from its. Else it not load pages. Therefore we have small non-sequental parts. Next starts will be more quick, because the file pages already cached in in-memory cache. Other internal OS optimizations also speedups next start.

Also a bunch of files in user profile., which is cached on next hot start.

Also, is the Chrome.dll file defragmented on HDD disk , and disk is optimized by Prefetcher ?

Can you compare your results with start from SSD ?

[zdland]

I have checked that chrome.dll is already fully commited in the virtual memory of chrome.exe immediately after startup (even on first startup), so that I don't think this is the case as you describe. In other words it's still reading from disk after the DLL is already loaded!

I don't have an SSD. But I think this will not be a problem for SSD since it doesn't have seeking overhead, so that whether the DLL is read in order or not it would be equally fast.

[zdland]

It does have something to do with caching though, since if I delete and re-create chrome.dll (this purges any cache) then the startup time goes back to around 1 minute.

[IDA-RE-things]

Then try to create RAM-disk in memory, if you have sufficient memory. And do browser start from it.

I have checked that chrome.dll is already fully commited in the virtual memory of chrome.exe just seconds after startup (even on first startup),

under "Commited" you mean memory allocated for it (virtual address space), or DLL fully loaded from the disk ? I think OS loads it on start as memory-mapped file and new pages loaded on request, producing "Working Set".

[zdland]

Yes the content of the DLL is in memory. I checked 12000000h, 13000000h, ..., 1d000000h (for every 01000000h) and a few random addresses in between and they are all loaded with the DLL's content, immediately after I start Supermium.

I could be misled though; it may be the case that these pages are actually loaded by the debugger (ie. they are loaded whenever I want to dump them in the debugger, but otherwise not loaded in Supermium). I don't know much how the memory manager works in the OS.

So you may be right as the cause of the disk reads! But can you confirm it is the case?

[IDA-RE-things]

I could be misled though; it may be the case that these pages are actually loaded by the debugger (ie. they are loaded whenever I want to dump them in the debugger, but otherwise not loaded in Supermium). I don't know much how the memory manager works in the OS.

:) I think it is this case. Any time any program accessed to page, ==> its contents loads to memory.

So you may be right as the cause of the disk reads! But can you confirm it is the case?

you should try to use XP/2003 Prefetcher (google about it). And defragment disk using it.

Also try to use RAM disk as I mentioned above.

[zdland]

I think you are probably right. Immediately after I started Supermium, I suspended it and did a search over the whole memory range of chrome.dll, and there are quite a bit disk activity. This suggests the DLL is not really mapped into memory. (but appears to be, because each page is loaded on demand whenever I try to read it).

Can you possibly add a hack in Supermium that explicitly loads the DLL in full before using it to check if this is the problem? And leave the hack there in future releases if this is indeed the cause? This is really important for systems running on HDD. It saves nearly a minute on startup!

[zdland]

Do you mean the EnablePrefetcher registry setting? I already have it enabled

[IDA-RE-things]

I remember, that somehow the system disk defragmenter can be forced to run, using prefetching info, and optimize files on disk. I doing this in the past. Its why I have recommended to google it for details. Have deleted old C:\WINDOWS\Prefetch\Layout.ini. (if it has obsolete info) Then re-run several programs to be optimized by Prefetcher. Then run Defrag. The prefetcher should be tuned to "Applaunch and Boot" in the registry.

[IDA-RE-things]

You can do brute speed test of dll reading/loading just by copying it into NUL, with favorite File Manager. (i'm using FAR). If the problem only this DLL.

[zdland]

copy chrome.dll nul finishes instantly. there is probably special optimization for it. copying that file elsewhere takes only about a second, which makes sense for a 200MB file. based on the Filemon logs the majority of startup time is spent reading the chrome.dll file so I believe (at least most of) the problem is this dll. It's not a bug though, it's more like some optimization needed.

[IDA-RE-things]

copying that file elsewhere takes only about a second, which makes sense for a 200MB file.

So this time 1s you have after system restarting, and before the file was read and cached ? Which result about startup time you will have, it after start you copy the file this way (so it will be in cache), then run the browser. ?

Also how many time takes another browser -- Thorium for XP to start ?

[zdland]

copying that file elsewhere takes only about a second, which makes sense for a 200MB file.

So this time 1s you have after system restarting, and before the file was read and cached ?

Yes. Also, copying other files of similar size also takes about 1s.

Which result about startup time you will have, it after start you copy the file this way (so it will be in cache), then run the browser. ?

It is about the same (1 minute). It seems the cache is only valid after I run Supermium. I tried other ways such as reading the file in an editor, scrolling until no disk activity happens (so the file is fully in cache). It doesn't improve Supermium's startup time at all

Also how many time takes another browser -- Thorium for XP to start ?

Thorium is about the same (1 minute).

[IDA-RE-things]

Now I have confirmed this. I took slow laptop with Celeron 1.6 Gz , 2Gb DDR2 ram, and slow HDD., OS: WinXp With no tabs opened. And using --single process. And after system restart, browser starts very slowly. The "Process Working Set" size enlarges very slowly. While second run is some seconds.

So I do more investigation on this.

[IDA-RE-things]

I have created workaround and it works on my laptop, And it will be inside my https://github.com/IDA-RE-things/Chrome-xp-api-adapter-dll (progwrp alternative) I'll inform when it's available., because some additional issue while.

[IDA-RE-things]

Yesterday and today I researched, developed and tested it, and now have time to relax.

Now all interested people can try it with Supermium/Thorium. https://github.com/IDA-RE-things/Chrome-xp-api-adapter/releases

This works with my slow notebook (although not so perfect speedup, as its theoretically possible), And I interested if it will be visible cold start speedup on your machines without SSD. Not remember to rename to progwrp.

[zdland]

Thank you for creating this patch! Is it supposed to replace progwrp.dll? It's not working for me, it failed to locate kernel32.SetProcessDEPPolicy on startup (Win2003 SP2 - x86).

[zdland]

And I checked on MSDN that this API is not supported on Windows 2003

[SidCos]

IDA-RE-things Well, you've done a great job! Thank you!

WinXPSP3 / HDD / Core2Quad / 4Gb / Supermium 122.0.6261.152 (R4)

Your file: chrome-xp-api-adapter.dll (renamed to progwrp.dll) Cold startup time: 3-4 seconds (!) o_O This is not a joke! Restart time: 1,5-2 seconds (!) - always

Original: progwrp.dll (version 1.1.0.5010) by win32ss Cold startup time: 29-31 seconds (!) Restart time: 2-8 seconds + sometimes even longer delays are possible

One more remark. It is very good that with your file, the Windows "Event Log" does not show messages about starting the [WMI Performance Adapter] service. Although the service itself is initially started in processes, but then unloaded. When using the original progwrp.dll (by win32ss), each time the browser is launched, 3 copies of this service are launched and the "Event Log" constantly shows system messages about the launch of this service. This is very tiresome.

Once again, you have done a great job! I hope you will not stop here and continue to please us, ordinary users!

[IDA-RE-things]

SidCos thanks for reply and testing.

zdland Ok, I will check with that function. Because I have tested it only on my XP SP3 machines. Fix for Win2003 SP2 will be available soon.

[IDA-RE-things]

zdland, fix for your OS now available.

[zdland]

@IDA-RE-things , it's still not working for me. It crashed, debug.log: [0506/032746.718:ERROR:registration_protocol_win.cc(136)] TransactNamedPipe: 管道已结束。 (0x6D) [0506/032746.718:ERROR:crashpad_client_win.cc(144)] crash server failed to launch, self-terminating and it did not create a crash dump.

[zdland]

(e48.e44): Unknown exception - code c00002b4 (first chance) (e48.e44): Unknown exception - code c00002b4 (!!! second chance !!!) eax=00000001 ebx=010b4000 ecx=0000ffff edx=00000000 esi=ffffffff edi=00000000 eip=00452049 esp=0163fc44 ebp=0163fc7c iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246 WARNING: Unable to verify checksum for chrome.exe ERROR: Symbol file could not be found. Defaulted to export symbols for chrome.exe - chrome+0x52049: 00452049 f30f5c0528ea5800 subss xmm0,dword ptr [chrome!IsSandboxedProcess+0x111d58 (0058ea28)] ds:0023:0058ea28=4f000000 0:000> u chrome+0x52049: 00452049 f30f5c0528ea5800 subss xmm0,dword ptr [chrome!IsSandboxedProcess+0x111d58 (0058ea28)] 00452051 f30f2cc8 cvttss2si ecx,xmm0 00452055 21d1 and ecx,edx 00452057 09c1 or ecx,eax 00452059 8d143f lea edx,[edi+edi] 0045205c 83ff03 cmp edi,3 0045205f 73b4 jae chrome+0x52015 (00452015) 00452061 31c0 xor eax,eax 0:000> k ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong. 0163fc7c 0044176e chrome+0x52049 0163fcd0 0044cafa chrome+0x4176e 0163ffa0 0044c621 chrome+0x4cafa 0163ffbc 7c82484f chrome+0x4c621 0163fff0 7c8125b5 kernel32!BaseThreadStart+0x34 0163fff0 00000000 kernel32!BaseFiberStart+0x17 0:000> ~ . 0 Id: e48.e44 Suspend: 1 Teb: 7ffdf000 Unfrozen 1 Id: e48.e24 Suspend: 1 Teb: 7ffde000 Unfrozen

[IDA-RE-things]

zdland, Ok.

I will try to create virtual machine with your OS. Seems some function, which not called on XP, was called on Win2k3. Or something else. I have replaced not called functions by debugbreak() stubs. From the start of developing it. To catch such situations. Its a pay for minimum size of DLL :)

But this not looks as debugbreak(). The exception code for debugbreak is 0x80000003

c00002b4 (first chance) is STATUS_FLOAT_MULTIPLE_FAULTS . its strange.

The crashdump creation was gone after HDD-optimizations as side effect. Now it not intended for crashes :). May be I fix crashdumps creation later.

BTW it now supports multicore CPU with new release.

Will go to fix that W2k3 problem now. (if it will be reproduced) But its strange that Chrome.exe/dll symbols was not resolved. Are you have downloaded them for (R4) ? Or which version you use.

[IDA-RE-things]

zdland, good news. finally it works on VM with srv2003. The problem was in fiber creation function, intended only for XP before. STATUS_FLOAT_MULTIPLE_FAULTS was because of floating point context was not saved. Now it was reworked and fixed. You can start to test.

[JoachimHenze]

Thank you so much for working on that @IDA-RE-things . I do also have one of those super-slow-starting HDD-laptops as a 2nd machine, running on XPSP3. And I do also have use for the browser on 2k3sp2 within VM.

[JoachimHenze]

I do confirm that your patch also improves the startup performance for me on 2k3sp2 32bit within a VirtualBox VM, 512MB RAM, just a single core is exposed to the VM-guest, and the flag --single-process. Awesome! I would vote for integrating your progwrp-changes into the next official build.

Here is one follow-up-ticket that might be interesting to look into: https://github.com/win32ss/supermium/issues/552

[zdland]

It now works! Cold start time is now only about 3 seconds. This is incredible. Thank you @IDA-RE-things for the great job. Do you mind me asking some questions?

1. What is the relationship between these two dll's? For example is one based on the other, do they share a common codebase or are they developed independently?
2. Can this optimiazation be merged into the next official build?
3. It seems the two dll's have same functionality, how come one is 133KB and the other is only 21.5KB?
4. How did you solve this problem? (eg. by pre-loading chrome.dll or not?)
5. Is source code available?
6. Now Supermium always creates debug.log whereas with the old progwrp.dll it creates this file only when a failure occurs. Can this debug.log be suppressed in normal operation?

[IDA-RE-things]

1). This have no common codebase. I had no sources of original at all. And its because I started my own implementation, based on reverse-engineering work, knowledge and existing experience. (see readme in my project) This is not a patch of his dll. This is my rework of my code. Also it has own limitations (see readme).

5). Both DLLs are closed source now (or while). (it was discussed early)

2). Its can't be included with official builds of Browser, because its different project. Download and use it separately. I have no any relation with browser owner. He dont want to share his know-how, and to see co-authors around. I also in this case :)

6). The issue with debug.log will be fixed later. it was side-effect of optimization. :)

[zdland]

OK I've read the readme Did you create this just for Supermium? Or is it also used elsewhere (other than Thorium etc)? Is it going to be kept up to date so that it's always possible to replace it for progwrp in future Supermium releases?

[IDA-RE-things]

zdland, Yes now just for Supermium/Thorium. Because for now this technique is not used anywhere else :) To use this, other browsers should integrate other XP-specific Chromium changes from Supermium codebase, wich Thoruim has done. Else it will not work. And even in case they integrate its, The other known issues, specific for XP will still here, before it will be fixed. I mean GDI font rendering problems (still not fixed), and lack of hardware acceleration.

Будет ли он обновляться, чтобы всегда можно было заменить его на progwrp в будущих выпусках Supermium?

Yes, I constantly experiment with it, and update in releases in case of valuable changes. I like to use it on my machines instead of original dll .

The problem with crashpad fixed, by the way.

[NS-Clone]

extremely offtop

is there any way to run filemon on W7 x64?