[request]Allow multiple torrent hash checks when data is save in different disks

[GoogleCodeExporter]

Current behavior: All torrent hash checks are queued and only one torrent at a 
time does a hash check.

Desired behavior: Queue all torrent hash checks, but torrents saved to 
different hard disks should hash-check simultaneously. Meaning queue per disk. 
This will improve things speed wise, if you want to check multiple torrents.

Original issue reported on code.google.com by hammered...@gmail.com on 7 May 2013 at 5:14

[GoogleCodeExporter]

is there a reliable way of knowing which device a path belongs to? Keep in mind 
edge cases where raid is used, or files are moved away so a single torrent is 
spread out on multiple disks.

Original comment by arvid.no...@gmail.com on 8 May 2013 at 3:53

[GoogleCodeExporter]

(I haven't used any of these APIs)

On Windows there is GetVolumeInformationByHandleW(): 
http://msdn.microsoft.com/en-us/library/windows/desktop/aa964920%28v=vs.85%29.as
px

On Linux I can't find a suitable system/kernel API but glib/gio manages to pull 
it somehow:
1. g_file_find_enclosing_mount() 
->https://developer.gnome.org/gio/stable/GFile.html#g-file-find-enclosing-mount
2. g_mount_get_drive() -> 
https://developer.gnome.org/gio/stable/GMount.html#g-mount-get-drive

On MacOS I have no idea. 

Unforutnately boost.filesystem doesn't seem to have a suitable API either.

>Keep in mind edge cases where raid is used
Maybe the above APIs return special values for RAID disks so you can identify 
them

>or files are moved away so a single torrent is spread out on multiple disks.
How is that possible by the current libtorrent API?

Original comment by hammered...@gmail.com on 8 May 2013 at 11:09

[GoogleCodeExporter]

> > or files are moved away so a single torrent is spread out on multiple disks.
> 
> How is that possible by the current libtorrent API?

You can rename_file() with a new path that starts with a few ../ to move out of 
the download directory and potentially the volume being downloaded to

Original comment by arvid.no...@gmail.com on 9 May 2013 at 1:05

[GoogleCodeExporter]

I would expect, the drive that torrent_handle::save_path is on to be used then.

Or again control the recheck behavior by flag that will be exposed to the user 
ie serialize_all, parallelize_per_disk. (maybe not a good idea after all).

Original comment by hammered...@gmail.com on 9 May 2013 at 9:01

[GoogleCodeExporter]

> > > or files are moved away so a single torrent is spread out on multiple 
disks.
> > 
> > How is that possible by the current libtorrent API?
> 
> You can rename_file() with a new path that starts with a few ../ to move out 
of the > download directory and potentially the volume being downloaded to

The ideal solution that comes to mind for me is for disk i/o in general to be 
queued in such a way that i/o for one device will never block i/o for another 
device.  Don't look at it as the torrent as whole, but as the specific piece 
being read from the disc.  If you have multiple read and/or write requests, 
parallelize them according to physical device (or for RAID -- the virtual 
multi-physical device -- unless you took the trouble to determine the exact 
raid disk topology, and further optimized things by ordered your reads and 
writes such that you were certain to maximize parallelization on the RAID, but 
I suspect the gains of specialized RAID treatment wouldn't be all the great).  
Also careful re-ordering of disk queue operations could be made to try to 
serialize them per device in blocks (without ever deferring any i/o for long to 
do so), in order to maximize disk and cache efficiency.

Original comment by a...@lovetour.info on 11 May 2013 at 10:54

[GoogleCodeExporter]

I'll add to my prior idea (which broadens this bug request to a more general 
optimization of disk i/o)
- if this were done, hash checks would not need queuing in and of themselves 
(further note a bit below)
- but i/o operations would need prioritization of some kind, beyond just the 
order they are received, and all these optimizations might take some time to 
evolve (uTorrent has had all kinds of problems with their v3 work in this 
field, 'though I haven't followed their developments in a few months)
- examples of prioritization/reordering that might be desired -- and perhaps 
could be handled similarly to how processor sharing is done by the kernel -- by 
weights rather than "no priority 2 can be done if we have any priority 1"
  *  Give a higher weight/priority to older i/o requests (first-est in)
  *  Accept a weight factor from the calling function with an i/o operation
- In addition to priority weights, have priority categories - that should be 
used sparingly.  These are either strictly ordered or so nearly so that they 
practically are ["practically" would only be to help mitigate possible 
problems/bugs due to i/o never gotten to on overloaded hardware].  Weights 
would be considered within each category, though.  This will allow for:
  *  critical i/o operations (possibly libtorrent data file updates) to always be handled regardless of what else is in queue;
  *  very important i/o operations to always be handled next, and before important operations (possibly disk writes for downloads, or disk reads for seeding -- depending on a user configurable option)
  *  important i/o operations next (e.g. the converse of my example for very important)
  *  low-priority operations next (e.g. hash checking)
  *  idle-only operations (stay at end of queue if anything else is before them -- the converse of critical priority category; not sure what would go here, but seems like it should be an option and may be useful; some may want hash checking done here, and there may be other things that really can wait indefinitely... maybe extensions involving searching, indexing, and hashing files, like the BEP.38 proposal might involve).
  **** these could be handled with a long int or floating point value where the max value is critical, the minimum value is idle, and values in-between are all available, allowing for sophisticated implementations)
  **** one could consider ways to share i/o resources fairly with other processes, especially non-critical i/o.

The reason to rely on weights more than strict ordering (besides a weight based 
on FIFO) is because disks may be overloaded, and most disk operations have SOME 
importance no matter what.

All of the above are done on a per-disk basis -- each disk essentially has a 
separate queue or is prioritized as if it did.  It might be going overboard to 
try to get this 100% right in every environment and every edge case.  Optimize 
I/o requests for all raid combinations?  If you've got the time and 
expertise...  It is worth bearing in mind that there is a spaghetti-like 
relationship between file systems, disks, disk partitions, virtual volumes, 
etc..  This can be helped by querying for data about where a file is located.  
But, sectors of a file may not be written in sequential order; if the volume is 
part of a striped disk array (RAID or software equivalent) or the volume spans 
physical partitions and disks to create a volume larger than any partition or 
disk, then one or more part of a file could be on eone physical disk and other 
parts on another.

Ideally, i/o parallelization is done on the lowest possible.
For Windows, if you want to take the trouble to handle the odd file that 
overlaps the extents of two physical disks properly, the following might be of 
help.  And possibly if the code is useful enough require little effort to bring 
a lower level of parallelization into things all around: 
http://blogs.msdn.com/b/dsadsi/archive/2010/03/19/mapping-volumeid-to-disk-parti
tion-using-the-deviceiocontrol-api.aspx 

I tried scouring the various Windows APIs and WMI references myself without 
seeing how to accomplish the feat, before stumbling on the above blog post.  
Some time must learn coding besides VB6 so I can contribute in bigger ways to 
good projects.  Thanks for your work on this one Arvid.

Original comment by a...@lovetour.info on 12 Jun 2013 at 8:44

[GoogleCodeExporter]

One very important aspect is caching and sequential reads. reading data 
sequentially is at least twice as fast (assuming spinning disks) than random 
access. The risk of throwing all torrents hash jobs into a pool and prioritize 
them is that we get more random access and less sequential access.

Regarding raids, the most efficient setup would be to not stripe disks, or 
rather, stripe disks with very large stripes, like 8 MB per disk. This would 
ensure that any one request would only lock up a single disk, and other 
requests could go unaffected to other disks. Achieving the most parallelism.

Original comment by arvid.no...@gmail.com on 12 Jun 2013 at 10:09

[GoogleCodeExporter]

Yes, I know what you mean regarding sequential access.  Good point on RAIDs, 
and it's probably best to not worry to much about them and assume they were 
configured efficiently.

Sequential access should ideally be the same as the sequence of the blocks of 
the file; but that is up to the O/S.  That would be a lower priority to 
determine actual sector-to-block mapping and order by sector, since usually the 
order will be the same if the O/S and file system are doing their job right.  
And if this were implemented, it would probably not be efficient to query this 
data constantly.  And larger reads and write operations (even multi-block) can 
be much more efficiently handled by the OS, but shouldn't be so large in an 
application of this type as to block a disk for long.

In regards to sequential ordering and in general --- care should be taken in 
how jobs are prioritized, but given equal priorities [my ideas might need some 
thinning down to achieve more equal priorities :)], the queue for each disk can 
be handled in sequential order; as the queue grows in size every request should 
be inserted in a position that balances disk efficiency and application 
priority; and then submitted to the OS in chunks as large as possible up to a 
maximum size to avoid holding up one disk's queue for too long in case 
something more important comes up.

Some elements of my ideas might need dropping or to be handled differently, and 
I may have suggested more complexity than required.  Any reduction in one disk 
blocking other disks would improve performance to the degree that multiple 
disks are involved.  Unless the pieces (of more complex code) just fall into 
place, or otherwise end up requiring implementation for everything to work, 
treating RAIDs, mirrored disks, and striped arrays of any sort like any other 
disk seems very reasonable, although I do like the idea of noticing separate 
disks when it comes to spanned-disk volumes, which I've used in the past.

In fact, I like spanned volumes so much that if I weren't on a laptop and 
needing to use various types of interfaces for my disks (2 sata internals, 
usb2, usb3, and I have esata and firewire interfaces not in use), but a 
desktop, I would probably configure my system with only two volumes, no matter 
how many disks: system/software volume and data volume.  I have just enough RAM 
that I'm presently considering how best to incorporate a RAM disk (8 GB)...

Another factor to consider might be how to interact with OS caching.  I know 
(can't think what off the top of my head) that some applications specifically 
bypass operating system disk caches.  If you have a more specialized caching 
system than the OS, it might make sense to bypass the OS cache if the OS might 
otherwise mess with your strategy.  Without exclusive access to the disk, it 
would still mean if the OS has other I/O, it will do some i/o pipeline sharing 
with you, but the ordering and amount of caching of your operations shouldn't 
be affected.  I don't know whether those applications are designed smartly in 
this regard or not.  It might reduce CPU load and memory copying, but on the 
other hand, it might either have a positive or a negative effect to 
disallow/allow the OS to also consider it's other pending operations along with 
yours.

As a simplest strategy, submitting (sequentially) reads or writes to what 
appear to be independent disks in parallel using even a rudimentary 
identification of disks, will be much better than no parallel strategy.

My setup consists at present of 4 physical disks on 3 controllers (one disk 
particularly slow -- USB 2.0 connection) and have torrents at present on 3 of 
those disks (one partition in use per disk--nothing fancy--except perhaps that 
I use junctions in Windows (mount points)) and would benefit from such a 
strategy if I need to hash check many torrents (e.g. importing torrents, fast 
connection, strange bugs causing torrents to be rechecked too often) and/or am 
performing other disk intensive tasks on the system at the same time.

Original comment by a...@lovetour.info on 13 Jun 2013 at 10:40

[GoogleCodeExporter]

p.s. I did have in mind that each entire file, in cases where you will be 
reading (or possibly writing) entire files should generally be queued 
themselves; and then parallelized per disk; and also allow other higher 
priority operations to be inserted into the datastream while that is happening 
(e.g. if libtorrent's only tasks are to hash 6 files: 2 on disk A, 2 on disk B, 
and 2 on disk C, then hash the files on disk A in the order those 2 files were 
requested to be hashed while also hashing disk B one file and then the next, 
while also hashing disk C one file and then the next).  But personally I would 
never want hashing to block seeding or downloading, so time sharing between 
different types of tasks should occur.

Original comment by a...@lovetour.info on 13 Jun 2013 at 10:59

[GoogleCodeExporter]

Though I'm no HDD expert, I will offer my thoughts. The problem of maximal HDD 
utilization is not limited to torrent clients in any way. For example, audio 
conversion software converts 2 files on different HDDs in background. So we 
need to insert this HDD/RAID checking code into every such program. IMHO we 
would be better moving this function to a HDD driver. And this does not require 
to change OS I/O interface.

My proposal. LibTorrent just sends a I/O request for every piece of every 
torrent; it does not wait when any particular request is completed. Though this 
may do harm. If the HDD driver performs requests in the order in which they 
have been received, the HDD head will move after almost every request. This 
will diminish efficiency of reading. To optimize, the HDD driver should pick a 
request from the queue which is *the closest on the HDD surface* to the request 
just completed.

In this scheme, LibTorrent receives no data about HDD/RAID installed in the 
computer. Additionally, LibTorrent should offer a choice between sequential 
hashing and parallel hashing in case of a HDD driver not optimized in the vein 
I described above.

Original comment by rabes...@gmail.com on 8 Feb 2015 at 1:13

[GoogleCodeExporter]

The problem with that proposal is that in order to issue a read request for 
every piece, libtorrent would have to allocate as much memory as the sizes of 
all the files it wants to check up-front, which is not very efficient.

Original comment by arvid.no...@gmail.com on 8 Feb 2015 at 2:03

[GoogleCodeExporter]

>This will diminish efficiency of reading. To optimize, the HDD driver should 
pick a request from the queue which is *the closest on the HDD surface* to the 
request just completed.

I think you just described NCQ: 
http://en.wikipedia.org/wiki/Native_Command_Queuing

Original comment by hammered...@gmail.com on 8 Feb 2015 at 2:05

[GoogleCodeExporter]

> libtorrent would have to allocate as much memory as the sizes of all the 
files it wants to check up-front, which is not very efficient.
I have not thought about that. :-) Also we may overflow the queue with requests.

First, I propose to limit the number of simultaneous I/O requests. For example, 
data are divided into big chunks, 1 request at a time for every chunk. (Or 
simply consider torrents as chunks.) Second, I propose to use "mmap". AFAIK, it 
allocates memory only when an HDD transmits data. There would be a OS thread 
for every chunk because a thread may be blocked when it reads mapped memory.

Original comment by rabes...@gmail.com on 8 Feb 2015 at 4:05

[GoogleCodeExporter]

Another idea. Request 1 byte from a chunk. If a request for a chunk is 
answered, read that chunk further.

Original comment by rabes...@gmail.com on 8 Feb 2015 at 5:24

[GoogleCodeExporter]

1 byte per request would be too inefficient IMO.

Original comment by hammered...@gmail.com on 8 Feb 2015 at 5:53

[GoogleCodeExporter]

First you request 1 byte. Then you request normal amount of data.

Original comment by rabes...@gmail.com on 8 Feb 2015 at 6:36

[GoogleCodeExporter]

Poll and Epoll would be suited for this task. A buffer is required only when 
some data are available for a file descriptor. But they don't work for regular 
files. Epoll fails with a permission error. Poll always reports that data are 
available for every file descriptor. Obviously, this is wrong.

Original comment by rabes...@gmail.com on 9 Feb 2015 at 12:17