Open GoogleCodeExporter opened 8 years ago
is there a reliable way of knowing which device a path belongs to? Keep in mind
edge cases where raid is used, or files are moved away so a single torrent is
spread out on multiple disks.
Original comment by arvid.no...@gmail.com
on 8 May 2013 at 3:53
(I haven't used any of these APIs)
On Windows there is GetVolumeInformationByHandleW():
http://msdn.microsoft.com/en-us/library/windows/desktop/aa964920%28v=vs.85%29.as
px
On Linux I can't find a suitable system/kernel API but glib/gio manages to pull
it somehow:
1. g_file_find_enclosing_mount()
->https://developer.gnome.org/gio/stable/GFile.html#g-file-find-enclosing-mount
2. g_mount_get_drive() ->
https://developer.gnome.org/gio/stable/GMount.html#g-mount-get-drive
On MacOS I have no idea.
Unforutnately boost.filesystem doesn't seem to have a suitable API either.
>Keep in mind edge cases where raid is used
Maybe the above APIs return special values for RAID disks so you can identify
them
>or files are moved away so a single torrent is spread out on multiple disks.
How is that possible by the current libtorrent API?
Original comment by hammered...@gmail.com
on 8 May 2013 at 11:09
> > or files are moved away so a single torrent is spread out on multiple disks.
>
> How is that possible by the current libtorrent API?
You can rename_file() with a new path that starts with a few ../ to move out of
the download directory and potentially the volume being downloaded to
Original comment by arvid.no...@gmail.com
on 9 May 2013 at 1:05
I would expect, the drive that torrent_handle::save_path is on to be used then.
Or again control the recheck behavior by flag that will be exposed to the user
ie serialize_all, parallelize_per_disk. (maybe not a good idea after all).
Original comment by hammered...@gmail.com
on 9 May 2013 at 9:01
> > > or files are moved away so a single torrent is spread out on multiple
disks.
> >
> > How is that possible by the current libtorrent API?
>
> You can rename_file() with a new path that starts with a few ../ to move out
of the > download directory and potentially the volume being downloaded to
The ideal solution that comes to mind for me is for disk i/o in general to be
queued in such a way that i/o for one device will never block i/o for another
device. Don't look at it as the torrent as whole, but as the specific piece
being read from the disc. If you have multiple read and/or write requests,
parallelize them according to physical device (or for RAID -- the virtual
multi-physical device -- unless you took the trouble to determine the exact
raid disk topology, and further optimized things by ordered your reads and
writes such that you were certain to maximize parallelization on the RAID, but
I suspect the gains of specialized RAID treatment wouldn't be all the great).
Also careful re-ordering of disk queue operations could be made to try to
serialize them per device in blocks (without ever deferring any i/o for long to
do so), in order to maximize disk and cache efficiency.
Original comment by a...@lovetour.info
on 11 May 2013 at 10:54
I'll add to my prior idea (which broadens this bug request to a more general
optimization of disk i/o)
- if this were done, hash checks would not need queuing in and of themselves
(further note a bit below)
- but i/o operations would need prioritization of some kind, beyond just the
order they are received, and all these optimizations might take some time to
evolve (uTorrent has had all kinds of problems with their v3 work in this
field, 'though I haven't followed their developments in a few months)
- examples of prioritization/reordering that might be desired -- and perhaps
could be handled similarly to how processor sharing is done by the kernel -- by
weights rather than "no priority 2 can be done if we have any priority 1"
* Give a higher weight/priority to older i/o requests (first-est in)
* Accept a weight factor from the calling function with an i/o operation
- In addition to priority weights, have priority categories - that should be
used sparingly. These are either strictly ordered or so nearly so that they
practically are ["practically" would only be to help mitigate possible
problems/bugs due to i/o never gotten to on overloaded hardware]. Weights
would be considered within each category, though. This will allow for:
* critical i/o operations (possibly libtorrent data file updates) to always be handled regardless of what else is in queue;
* very important i/o operations to always be handled next, and before important operations (possibly disk writes for downloads, or disk reads for seeding -- depending on a user configurable option)
* important i/o operations next (e.g. the converse of my example for very important)
* low-priority operations next (e.g. hash checking)
* idle-only operations (stay at end of queue if anything else is before them -- the converse of critical priority category; not sure what would go here, but seems like it should be an option and may be useful; some may want hash checking done here, and there may be other things that really can wait indefinitely... maybe extensions involving searching, indexing, and hashing files, like the BEP.38 proposal might involve).
**** these could be handled with a long int or floating point value where the max value is critical, the minimum value is idle, and values in-between are all available, allowing for sophisticated implementations)
**** one could consider ways to share i/o resources fairly with other processes, especially non-critical i/o.
The reason to rely on weights more than strict ordering (besides a weight based
on FIFO) is because disks may be overloaded, and most disk operations have SOME
importance no matter what.
All of the above are done on a per-disk basis -- each disk essentially has a
separate queue or is prioritized as if it did. It might be going overboard to
try to get this 100% right in every environment and every edge case. Optimize
I/o requests for all raid combinations? If you've got the time and
expertise... It is worth bearing in mind that there is a spaghetti-like
relationship between file systems, disks, disk partitions, virtual volumes,
etc.. This can be helped by querying for data about where a file is located.
But, sectors of a file may not be written in sequential order; if the volume is
part of a striped disk array (RAID or software equivalent) or the volume spans
physical partitions and disks to create a volume larger than any partition or
disk, then one or more part of a file could be on eone physical disk and other
parts on another.
Ideally, i/o parallelization is done on the lowest possible.
For Windows, if you want to take the trouble to handle the odd file that
overlaps the extents of two physical disks properly, the following might be of
help. And possibly if the code is useful enough require little effort to bring
a lower level of parallelization into things all around:
http://blogs.msdn.com/b/dsadsi/archive/2010/03/19/mapping-volumeid-to-disk-parti
tion-using-the-deviceiocontrol-api.aspx
I tried scouring the various Windows APIs and WMI references myself without
seeing how to accomplish the feat, before stumbling on the above blog post.
Some time must learn coding besides VB6 so I can contribute in bigger ways to
good projects. Thanks for your work on this one Arvid.
Original comment by a...@lovetour.info
on 12 Jun 2013 at 8:44
One very important aspect is caching and sequential reads. reading data
sequentially is at least twice as fast (assuming spinning disks) than random
access. The risk of throwing all torrents hash jobs into a pool and prioritize
them is that we get more random access and less sequential access.
Regarding raids, the most efficient setup would be to not stripe disks, or
rather, stripe disks with very large stripes, like 8 MB per disk. This would
ensure that any one request would only lock up a single disk, and other
requests could go unaffected to other disks. Achieving the most parallelism.
Original comment by arvid.no...@gmail.com
on 12 Jun 2013 at 10:09
Yes, I know what you mean regarding sequential access. Good point on RAIDs,
and it's probably best to not worry to much about them and assume they were
configured efficiently.
Sequential access should ideally be the same as the sequence of the blocks of
the file; but that is up to the O/S. That would be a lower priority to
determine actual sector-to-block mapping and order by sector, since usually the
order will be the same if the O/S and file system are doing their job right.
And if this were implemented, it would probably not be efficient to query this
data constantly. And larger reads and write operations (even multi-block) can
be much more efficiently handled by the OS, but shouldn't be so large in an
application of this type as to block a disk for long.
In regards to sequential ordering and in general --- care should be taken in
how jobs are prioritized, but given equal priorities [my ideas might need some
thinning down to achieve more equal priorities :)], the queue for each disk can
be handled in sequential order; as the queue grows in size every request should
be inserted in a position that balances disk efficiency and application
priority; and then submitted to the OS in chunks as large as possible up to a
maximum size to avoid holding up one disk's queue for too long in case
something more important comes up.
Some elements of my ideas might need dropping or to be handled differently, and
I may have suggested more complexity than required. Any reduction in one disk
blocking other disks would improve performance to the degree that multiple
disks are involved. Unless the pieces (of more complex code) just fall into
place, or otherwise end up requiring implementation for everything to work,
treating RAIDs, mirrored disks, and striped arrays of any sort like any other
disk seems very reasonable, although I do like the idea of noticing separate
disks when it comes to spanned-disk volumes, which I've used in the past.
In fact, I like spanned volumes so much that if I weren't on a laptop and
needing to use various types of interfaces for my disks (2 sata internals,
usb2, usb3, and I have esata and firewire interfaces not in use), but a
desktop, I would probably configure my system with only two volumes, no matter
how many disks: system/software volume and data volume. I have just enough RAM
that I'm presently considering how best to incorporate a RAM disk (8 GB)...
Another factor to consider might be how to interact with OS caching. I know
(can't think what off the top of my head) that some applications specifically
bypass operating system disk caches. If you have a more specialized caching
system than the OS, it might make sense to bypass the OS cache if the OS might
otherwise mess with your strategy. Without exclusive access to the disk, it
would still mean if the OS has other I/O, it will do some i/o pipeline sharing
with you, but the ordering and amount of caching of your operations shouldn't
be affected. I don't know whether those applications are designed smartly in
this regard or not. It might reduce CPU load and memory copying, but on the
other hand, it might either have a positive or a negative effect to
disallow/allow the OS to also consider it's other pending operations along with
yours.
As a simplest strategy, submitting (sequentially) reads or writes to what
appear to be independent disks in parallel using even a rudimentary
identification of disks, will be much better than no parallel strategy.
My setup consists at present of 4 physical disks on 3 controllers (one disk
particularly slow -- USB 2.0 connection) and have torrents at present on 3 of
those disks (one partition in use per disk--nothing fancy--except perhaps that
I use junctions in Windows (mount points)) and would benefit from such a
strategy if I need to hash check many torrents (e.g. importing torrents, fast
connection, strange bugs causing torrents to be rechecked too often) and/or am
performing other disk intensive tasks on the system at the same time.
Original comment by a...@lovetour.info
on 13 Jun 2013 at 10:40
p.s. I did have in mind that each entire file, in cases where you will be
reading (or possibly writing) entire files should generally be queued
themselves; and then parallelized per disk; and also allow other higher
priority operations to be inserted into the datastream while that is happening
(e.g. if libtorrent's only tasks are to hash 6 files: 2 on disk A, 2 on disk B,
and 2 on disk C, then hash the files on disk A in the order those 2 files were
requested to be hashed while also hashing disk B one file and then the next,
while also hashing disk C one file and then the next). But personally I would
never want hashing to block seeding or downloading, so time sharing between
different types of tasks should occur.
Original comment by a...@lovetour.info
on 13 Jun 2013 at 10:59
Though I'm no HDD expert, I will offer my thoughts. The problem of maximal HDD
utilization is not limited to torrent clients in any way. For example, audio
conversion software converts 2 files on different HDDs in background. So we
need to insert this HDD/RAID checking code into every such program. IMHO we
would be better moving this function to a HDD driver. And this does not require
to change OS I/O interface.
My proposal. LibTorrent just sends a I/O request for every piece of every
torrent; it does not wait when any particular request is completed. Though this
may do harm. If the HDD driver performs requests in the order in which they
have been received, the HDD head will move after almost every request. This
will diminish efficiency of reading. To optimize, the HDD driver should pick a
request from the queue which is *the closest on the HDD surface* to the request
just completed.
In this scheme, LibTorrent receives no data about HDD/RAID installed in the
computer. Additionally, LibTorrent should offer a choice between sequential
hashing and parallel hashing in case of a HDD driver not optimized in the vein
I described above.
Original comment by rabes...@gmail.com
on 8 Feb 2015 at 1:13
The problem with that proposal is that in order to issue a read request for
every piece, libtorrent would have to allocate as much memory as the sizes of
all the files it wants to check up-front, which is not very efficient.
Original comment by arvid.no...@gmail.com
on 8 Feb 2015 at 2:03
>This will diminish efficiency of reading. To optimize, the HDD driver should
pick a request from the queue which is *the closest on the HDD surface* to the
request just completed.
I think you just described NCQ:
http://en.wikipedia.org/wiki/Native_Command_Queuing
Original comment by hammered...@gmail.com
on 8 Feb 2015 at 2:05
> libtorrent would have to allocate as much memory as the sizes of all the
files it wants to check up-front, which is not very efficient.
I have not thought about that. :-) Also we may overflow the queue with requests.
First, I propose to limit the number of simultaneous I/O requests. For example,
data are divided into big chunks, 1 request at a time for every chunk. (Or
simply consider torrents as chunks.) Second, I propose to use "mmap". AFAIK, it
allocates memory only when an HDD transmits data. There would be a OS thread
for every chunk because a thread may be blocked when it reads mapped memory.
Original comment by rabes...@gmail.com
on 8 Feb 2015 at 4:05
Another idea. Request 1 byte from a chunk. If a request for a chunk is
answered, read that chunk further.
Original comment by rabes...@gmail.com
on 8 Feb 2015 at 5:24
1 byte per request would be too inefficient IMO.
Original comment by hammered...@gmail.com
on 8 Feb 2015 at 5:53
First you request 1 byte. Then you request normal amount of data.
Original comment by rabes...@gmail.com
on 8 Feb 2015 at 6:36
Poll and Epoll would be suited for this task. A buffer is required only when
some data are available for a file descriptor. But they don't work for regular
files. Epoll fails with a permission error. Poll always reports that data are
available for every file descriptor. Obviously, this is wrong.
Original comment by rabes...@gmail.com
on 9 Feb 2015 at 12:17
Original issue reported on code.google.com by
hammered...@gmail.com
on 7 May 2013 at 5:14