Skip zero-filling on Full Allocation (on NTFS-Windows)

[GoogleCodeExporter]

The main problem when using any torrent client in windows based on libtorrent 
is the allocation.

There's two mods of allocations in libtorrent, one is spares which cause way 
too much of fragmentation, the other is full allocation which prevents 
fragmentation completely. 
Then, when you try to download a 40 GB file with full allocation the client 
will first zero-fill the hdd with 40GB of zeros which will take long time.
On the other hand on ext3/4 the write of the data starts right away unlike on 
NTFS.

Thus uTorrent managed to overcome this issue by making the option: 
"diskio.no_zero - Enabling this option causes µTorrent to skip the 
zero-filling process for file allocation"
http://www.netcheif.com/Articles/uTorrent/html/AppendixA_02_12.html#diskio.no_ze
ro

In the end, on windows with NTFS file system when downloading a big torrent 
with uTorrent you're able to download it right away and you dont get any 
fragmentation what so ever, while using a client based on libtorrent you'll 
have to wait very long time before you're able to start writing all the data on 
the drive which makes it very annoying, though i wanna use libtorrent based 
clients.

I wonder if it's possible to make the same option on libtorrent that when you 
use full allocation to be able to skip the zero filling stage just like in 
uTorrent?

Original issue reported on code.google.com by Class...@gmail.com on 4 Feb 2014 at 7:03

[GoogleCodeExporter]

is is possible. however, I would not imagine the nozero feature to come into 
effect very often. You can only do that when running with administrator 
privileges. So, whenever you're running on vista+ (presumably you aren't 
running as admin anymore on those systems), you still will have to suffer the 
pre-allocation.

are you running uTorrent as administrator?

Original comment by arvid.no...@gmail.com on 4 Feb 2014 at 10:33

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I'm using Deluge and qBittorrent.
I'm using Windows 7.
All of my friends and I included running on windows 7/8 and we're using the 
administrator user.

I'm helping many users online on few forums and many times it gets to remote 
control, and from all the 30+ users I've remote controlled, all of them were 
using the Administrator user.

So I know that such feature would come into effect so many times, no doubt 
about it.

so to your question, yes im using uTorrent as well as Administrator.
once you implemented it in libtorrent it would be very easy to change in deluge 
source code.

I love libtorrent but thats the only downside in my mind, and I know few 
individuals that because of that issue still using uTorrent although clients 
based libtorrent are faster and much better in many areas.

Original comment by Class...@gmail.com on 4 Feb 2014 at 10:44

[GoogleCodeExporter]

could you (or someone) please test this patch on windows? It's against RC_0_16 
(should apply cleanly against 0.16.14).

http://dpaste.com/hold/1586689/

Original comment by arvid.no...@gmail.com on 5 Feb 2014 at 10:46

[GoogleCodeExporter]

I guess it shouldnt be too hard, though i dont know where can i get 0.16.14 
since newest .msi is 0.16.10. I guess i could build the 0.16.13 version but the 
one you're referring to I dont know where to get it.

Original comment by Class...@gmail.com on 5 Feb 2014 at 11:40

[GoogleCodeExporter]

the .msi is a prebuilt package anyway. you need the source to apply this patch 
and rebuild. Here's the download page:

http://sourceforge.net/projects/libtorrent/files/libtorrent/

Original comment by arvid.no...@gmail.com on 5 Feb 2014 at 5:09

[GoogleCodeExporter]

btw. you may be able to ask the deluge developers for help making a test build.

Original comment by arvid.no...@gmail.com on 5 Feb 2014 at 5:10

[GoogleCodeExporter]

Yea, I will,talking to Cas :) thank you!
I'll update you when I got news.

Original comment by Class...@gmail.com on 5 Feb 2014 at 5:12

[GoogleCodeExporter]

You can use this temp build of qbt: 
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha20140205patched_libtorrent_set
up.exe

It's based on RC_0_16+this path, boost 1.55 and qbt git master build using 
mingw (gcc 4.8.0)

Original comment by hammered...@gmail.com on 5 Feb 2014 at 9:36

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I've just tried this build.
I've put Pre-Allocate ON, when it should start doing the zero filling (stage) 
the client crash.
I've tried few different file sizes, 40GB and 1GB as well, same results.
After it start downloading the first piece and it has to write the whole zeros 
it just crash.
Though, Sparse option still works as it was.

I'm not aware of debug mode on qBittorrent, is there one ? so i could give you 
more info.

http://dpaste.com/1588009/
here's the info when it crash.
It keeps crashing at the same stage.

Original comment by Class...@gmail.com on 6 Feb 2014 at 4:37

[GoogleCodeExporter]

If there's a way for me to give you more info, any debugging option, I'd love 
to, just tell me how.

Original comment by Class...@gmail.com on 6 Feb 2014 at 4:38

[GoogleCodeExporter]

make sure the session_settings::disk_io_write_mode and 
session_settings::disk_io_read_mode are both set to enable_os_cache. It's 
possible that if those settings are set to something else, that could cause the 
stall when allocating space.

I'll have to look closer into the crash, it sounds like it's preventing any 
interesting test for now. I'll try to reproduce it.

Original comment by arvid.no...@gmail.com on 6 Feb 2014 at 6:02

[GoogleCodeExporter]

This crash may be directly linked to something else. I was about to make a new 
issue(arvid tell me if I need to open a new one).
I released qbt v3.1.8 with 0.16.10+svn9664 (vanilla). Many users reported 
crashes and have tracked it down. I reverted r9627 and one user so far has 
reported that the crashes don't happen. So probably that is the culprit:

relevant qbt bug: https://github.com/qbittorrent/qBittorrent/issues/1386

Original comment by hammered...@gmail.com on 6 Feb 2014 at 7:51

[GoogleCodeExporter]

EDIT: Many users reported crashes and have tracked it down to the preallocation 
option.

Original comment by hammered...@gmail.com on 6 Feb 2014 at 7:54

[GoogleCodeExporter]

that's without this patch also then?

Original comment by arvid.no...@gmail.com on 6 Feb 2014 at 8:19

[GoogleCodeExporter]

I tried qbittorrent 3.1.8 that was on the website yesterday 13:34 GMT+2 and I 
didnt get any crash while trying to allocate.
So I assume I got the reverted one?

Original comment by Class...@gmail.com on 6 Feb 2014 at 8:34

[GoogleCodeExporter]

The test build in the qbt issue was vanilla RC_0_16 with r9627 reverted.

Original comment by hammered...@gmail.com on 6 Feb 2014 at 8:35

[GoogleCodeExporter]

@comment 17 
No the official release from the site(linking to sf.net) contains 
0.16.10+svn9664 (vanilla)

Only the one linked in the qbt issue has the reverted commit.

Original comment by hammered...@gmail.com on 6 Feb 2014 at 8:36

[GoogleCodeExporter]

Can you make the r9627 one with his patch and I'll test it?

Original comment by Class...@gmail.com on 6 Feb 2014 at 8:38

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Indeed 9664 crash for me, even the one from the website!
the same crash that i've seen with the one you made for me to test.

Is it possible that you'd make with libtorrent patch but with SVN9627?

Original comment by Class...@gmail.com on 6 Feb 2014 at 9:21

[GoogleCodeExporter]

hammered999: please create another ticket for the crash related to patch: 9627.
please include clarification on: was this the only commit you reverted, or did 
you revert _to_ that revision? do you have any more information about the 
crash? like a stack trace?

Original comment by arvid.no...@gmail.com on 6 Feb 2014 at 9:47

[GoogleCodeExporter]

@22 here is a build with 9627 reverted and the patch applied: 
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha20140207_libtorrentpatchedandr
everted_setup.exe

Original comment by hammered...@gmail.com on 6 Feb 2014 at 10:20

[GoogleCodeExporter]

@24 I get no crashes nor errors, but it still does the Zero Filling, the whole 
size of the torrent.

I used HdTune to measure how much of data it actually wrote, and right when the 
torrent started after he got about half of the first piece it started the 
allocation and wrote zeros.

uTorrent on the other hand I could see on HdTune which said only the amount I 
downloaded and not the whole torrent size.

Original comment by Class...@gmail.com on 6 Feb 2014 at 10:55

[GoogleCodeExporter]

Thanks for all the testing. I have a potential explanation for why that patch 
failed. If qBT was built with windows 2000 SDK (i.e. _WIN32_WINNT set to win2k 
support). The function call would never happen in that case. I've fixed this in 
the new patch:

   http://dpaste.com/1590359/

Original comment by arvid.no...@gmail.com on 7 Feb 2014 at 9:01

[GoogleCodeExporter]

also, btw, the crash has been fixed and checked in to RC_0_16 and trunk. In 
case it's easier, this patch most likely also applies cleanly to 
libtorrent-0.16.13

Original comment by arvid.no...@gmail.com on 7 Feb 2014 at 9:02

[GoogleCodeExporter]

another interesting experiment would be to disable the diskio.no_zero in 
utorrent, to see if that makes allocation take a long time, and actually writes 
zeroes. One possibility is that since files aren't opened in unbuffered mode 
anymore, the kernel may zero out the blocks in the background anyway.

Original comment by arvid.no...@gmail.com on 7 Feb 2014 at 10:11

[GoogleCodeExporter]

I did test it my self about uTorrent, I did disable diskio.no_zero few days 
earlier before I posted here.
When you disable it, it will work just like deluge or qbittorrent, meaning it 
will start writing all the zeros and write the whole torrent just as you start 
downloading it.

Hammer, if you could please compile with the new patch?

Original comment by Class...@gmail.com on 7 Feb 2014 at 5:38

[GoogleCodeExporter]

Here's a build with the new patch: 
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha20140208_33b0a2b55f_libtorrent
patched_setup.exe

@arvid, Just so you know libtorrent itself is built without any custom defines. 
I use mingw gcc 4.8.0. Maybe it is predefined by it. (_WIN32_WINNT)

Original comment by hammered...@gmail.com on 7 Feb 2014 at 11:36

[GoogleCodeExporter]

-- IMPORTANT UPDATE--
Cheers!! it seems to be working as I had it in my mind.

I did try downloading various file types, sizes, torrent piece sizes, single 
file and many files as well, all seems to be working flawlessly.

There's no fragmentation whatsoever and the file size is at max size while 
skipping the zero writing!!!

SUPERB!!

Are there any more tests I should do in your mind?

Original comment by Class...@gmail.com on 8 Feb 2014 at 12:17

[GoogleCodeExporter]

anything you can think of to make you confident all the edge cases work. It 
sounds like the patch is good to go though. I will probably make a 0.16.15 
release this weekend, to replace the buggy 0.16.14.

Original comment by arvid.no...@gmail.com on 8 Feb 2014 at 1:58

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Yea, I did want to make few more tests and so I did, and I found a problem.

If you close qbittorrent and open it again and you add a new torrent, the FIRST 
torrent will not skip the zero filling, but the second and higher torrent you 
add will skip the zero filling.

I tried as well to delete completely qbittorrent and install it from scratch, 
and I got the same thing, the first torrent does zero filling and the ones 
after will skip as it should.

So as long as you keep qbittorrent open and you passed the first torrent 
adding, then the rest will skip the zero filling.

Original comment by Class...@gmail.com on 8 Feb 2014 at 3:16

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

The FIRST torrent will not skip the zero filling, but the second and *next* 
torrents (was too tired).

When I mean the first torrent I mean you can start download a torrent, no 
matter its size you can just wait for it to finish zero filling and then just 
add another torrent and this next one will skip the zero filling. 

Another test I did is:
Add 10GB torrent file, pause it at 32KB (amount downloaded) and it wrote only 
1GB of zeros, then adding a new torrent, that new torrent WILL skip the zero 
filling, BUT if you then resume the first torrent we added (and paused at 32KB 
amount downloaded) it will continue to write zeros again after a minute (the 
rest of the 9GB left).

So it seems that the first torrent no matter what it will always going to do 
zero filling, no matter if you pause it or not.

Original comment by Class...@gmail.com on 8 Feb 2014 at 2:27

[GoogleCodeExporter]

Hi mate.
I wonder, are there any updates? and is there a way for me to help you?

Original comment by Class...@gmail.com on 13 Feb 2014 at 2:51

[GoogleCodeExporter]

the fact that it fails on the first torrent is a bit mysterious. Would you be 
able to reproduce it under process monitor? (you can download it from 
microsoft, it's like strace for windows).

Please keep the test as simple as possible. just one single-file torrent first 
(to fail) and then one single file torrent that succeeds for instance. I'm 
particularly interested in the file operations (you may need to make sure they 
are included in the filter before you start).

Also, when I'm running this on win7, I don't have the privilege to disable the 
zero filling. Do I need to do something special (e.g. start the debugger with 
"Run as administrator")?

Original comment by arvid.no...@gmail.com on 14 Feb 2014 at 7:35

[GoogleCodeExporter]

About the process monitor I'll check it out and give you update on it.

On windows im logged on as administrator account.
On task manager qbittorrent says my username, and not system or something like 
that.
In the control panel > User Accounts > Change User Account Control Settings, is 
set up to Never Notify. 

Original comment by Class...@gmail.com on 16 Feb 2014 at 4:58

[GoogleCodeExporter]

I was able to filter all the other things I had running on my PC (you can see 
my filter list) http://imageshack.com/a/img545/8858/y0yo.png

I did 2 tests: after opening the program and adding the torrent - getting the 
"where to save" window I cleaned the output of ProcessMonitor, so when I 
pressed "OK" to start downloading will be your first output.
1 test was with zero filling (the first torrent), the 2nd test was without zero 
filling (skipped).

I did save the output in every output I can: PML, CSV and XML, which I zipped 
them together for each torrent.

zero-filling.zip contains the zero filling (all 3 files) which is the first try 
after opening the program.
skip-zero-filling.zip contains the skip zero filling (all 3 files) which is the 
2nd try.

Original comment by Class...@gmail.com on 17 Feb 2014 at 1:52

[GoogleCodeExporter]

here are the links:
http://www4.zippyshare.com/v/4518886/file.html
http://www4.zippyshare.com/v/25867766/file.html

Original comment by Class...@gmail.com on 17 Feb 2014 at 1:52

[GoogleCodeExporter]

Thanks for the logs!

This is the case where it succeeds:

"3:32:07.7549310 
PM","qbittorrent.exe","4464","CreateFile","D:\Downloads\ubuntu-12.04.4-dvd-amd64
.iso","SUCCESS","Desired Access: Generic Read/Write, Disposition: OpenIf, 
Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, 
ShareMode: Read, AllocationSize: 0, OpenResult: Created"
"3:32:07.7550693 
PM","qbittorrent.exe","4464","QueryStandardInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 0, EndOfFile: 0, 
NumberOfLinks: 1, DeletePending: False, Directory: False"
"3:32:07.7550857 
PM","qbittorrent.exe","4464","SetEndOfFileInformationFile","D:\Downloads\ubuntu-
12.04.4-dvd-amd64.iso","SUCCESS","EndOfFile: 1,758,167,040"
"3:32:07.7552039 
PM","qbittorrent.exe","4464","SetAllocationInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 1,758,167,040"
"3:32:07.7552320 
PM","qbittorrent.exe","4464","SetValidDataLengthInformationFile","D:\Downloads\u
buntu-12.04.4-dvd-amd64.iso","SUCCESS","ValidDataLength: 1,758,167,040"

This is the case where it fails:

"3:30:14.3472893 
PM","qbittorrent.exe","4464","CreateFile","D:\Downloads\ubuntu-12.04.4-dvd-amd64
.iso","SUCCESS","Desired Access: Generic Read/Write, Disposition: OpenIf, 
Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, 
ShareMode: Read, AllocationSize: 0, OpenResult: Created"
"3:30:14.3473946 
PM","qbittorrent.exe","4464","QueryStandardInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 0, EndOfFile: 0, 
NumberOfLinks: 1, DeletePending: False, Directory: False"
"3:30:14.3474048 
PM","qbittorrent.exe","4464","SetEndOfFileInformationFile","D:\Downloads\ubuntu-
12.04.4-dvd-amd64.iso","SUCCESS","EndOfFile: 1,758,167,040"
"3:30:14.3474850 
PM","qbittorrent.exe","4464","SetAllocationInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 1,758,167,040"
"3:30:14.3477158 
PM","qbittorrent.exe","4464","SetValidDataLengthInformationFile","D:\Downloads\u
buntu-12.04.4-dvd-amd64.iso","PRIVILEGE NOT HELD","ValidDataLength: 
1,758,167,040"

i.e. for some reason it fails with "PRIVILEGE NOT HELD" the first time. I 
wonder why. I'll take a look at the order of my logic.

Original comment by arvid.no...@gmail.com on 20 Feb 2014 at 2:38

[GoogleCodeExporter]

Where do you think its coming from?
qbittorrent? libtorrent? the filesystem?

Original comment by Class...@gmail.com on 20 Feb 2014 at 2:55

[GoogleCodeExporter]

I can't find any calls that resemble attempting to add the privilege token 
though. do you think it got edited out of the logs? maybe those aren't system 
calls that would be captured by process monitor.

Original comment by arvid.no...@gmail.com on 22 Feb 2014 at 8:13

[GoogleCodeExporter]

If you want me to take a filter out, you can just take a look at the filters I 
made and tell me which one you want me to take out.

If you could compile it for windows (libtorrent with the patch), I would be 
able to test it with Deluge if you can do that.

Original comment by Class...@gmail.com on 22 Feb 2014 at 8:17

[GoogleCodeExporter]

I may have figured it out. I put in an attempted fix in RC_0_16 and trunk. I 
believe I needed to get the permissions before opening the file.

Original comment by arvid.no...@gmail.com on 23 Feb 2014 at 7:11

[GoogleCodeExporter]

I'll upload a new build in a few hours.

Original comment by hammered...@gmail.com on 23 Feb 2014 at 7:14

[GoogleCodeExporter]

Here is a new build with r9731: 
http://builds.shiki.hu/temp/qbittorrent_3.2.0alpha23022014_0303d2bf5566_libtorre
ntr9731_setup.exe

Original comment by hammered...@gmail.com on 23 Feb 2014 at 8:26

[GoogleCodeExporter]

It still acts the same.

"11:26:47.4413572 
PM","qbittorrent.exe","1152","CreateFile","D:\Downloads\ubuntu-12.04.4-dvd-amd64
.iso","SUCCESS","Desired Access: Generic Read/Write, Disposition: OpenIf, 
Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, 
ShareMode: Read, AllocationSize: 0, OpenResult: Created"
"11:26:47.4414751 
PM","qbittorrent.exe","1152","CreateFileMapping","D:\Downloads\ubuntu-12.04.4-dv
d-amd64.iso","FILE LOCKED WITH WRITERS","SyncType: SyncTypeCreateSection, 
PageProtection: "
"11:26:47.4414818 
PM","qbittorrent.exe","1152","QueryStandardInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 0, EndOfFile: 0, 
NumberOfLinks: 1, DeletePending: False, Directory: False"
"11:26:47.4415181 
PM","qbittorrent.exe","1152","QueryStandardInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 0, EndOfFile: 0, 
NumberOfLinks: 1, DeletePending: False, Directory: False"
"11:26:47.4415198 PM","qbittorrent.exe","1152","TCP Receive","Admin-PC:51646 -> 
94.41.215.204.dynamic.ufanet.ru:17580","SUCCESS","Length: 3472, seqnum: 0, 
connid: 0"
"11:26:47.4415280 
PM","qbittorrent.exe","1152","SetEndOfFileInformationFile","D:\Downloads\ubuntu-
12.04.4-dvd-amd64.iso","SUCCESS","EndOfFile: 1,758,167,040"
"11:26:47.4415295 PM","qbittorrent.exe","1152","TCP Receive","Admin-PC:51646 -> 
94.41.215.204.dynamic.ufanet.ru:17580","SUCCESS","Length: 256, seqnum: 0, 
connid: 0"
"11:26:47.4416196 
PM","qbittorrent.exe","1152","SetAllocationInformationFile","D:\Downloads\ubuntu
-12.04.4-dvd-amd64.iso","SUCCESS","AllocationSize: 1,758,167,040"
"11:26:47.4417393 PM","qbittorrent.exe","1152","TCP Receive","Admin-PC:51646 -> 
94.41.215.204.dynamic.ufanet.ru:17580","SUCCESS","Length: 1696, seqnum: 0, 
connid: 0"
"11:26:47.4418648 
PM","qbittorrent.exe","1152","SetValidDataLengthInformationFile","D:\Downloads\u
buntu-12.04.4-dvd-amd64.iso","PRIVILEGE NOT HELD","ValidDataLength: 
1,758,167,040"

Original comment by Class...@gmail.com on 23 Feb 2014 at 9:39