asset failed with large file when saved to multi-files with remap_files

[GoogleCodeExporter]

split large file (>2Gib) into multi-files of 1GiB.  
So far, 2GiB file works fine. But this torrent is larger than 5GiB and always 
failed. It changes to seeding mode when the first 1GiB file finished. And gave 
assert failed sometime. 

policy::iterator policy::find_connect_candidate(int session_time)
{
    INVARIANT_CHECK;

    int candidate = -1;
    int erase_candidate = -1;

=>    TORRENT_ASSERT(m_finished == m_torrent->is_finished());

    int min_reconnect_time = m_torrent->settings().min_reconnect_time;
    address external_ip = m_torrent->session().external_address();

....
}

Original issue reported on code.google.com by ygao....@gmail.com on 19 Feb 2014 at 2:27

[GoogleCodeExporter]

what's the stack trace?

Original comment by arvid.no...@gmail.com on 19 Feb 2014 at 5:56

[GoogleCodeExporter]

#0  0x401dd992 in ?? () from /test/obj/local/armeabi-v7a/libc.so
#1  0x401db074 in abort () from /test/obj/local/armeabi-v7a/libc.so
#2  0x5e327dd4 in assert_fail (expr=0x5e5dd334 "m_finished == 
m_torrent->is_finished()", line=651, file=0x5e5dce28 "policy.cpp", 
function=0x5e879a38 "libtorrent::policy::iterator 
libtorrent::policy::find_connect_candidate(int)", value=0x0) at assert.cpp:243
#3  0x5e47b014 in libtorrent::policy::find_connect_candidate (this=0x5c6bd870, 
session_time=1113) at policy.cpp:651
#4  0x5e47b50e in libtorrent::policy::connect_one_peer (this=0x5c6bd870, 
session_time=1113) at policy.cpp:1484
#5  0x5e354e70 in libtorrent::aux::session_impl::on_tick (this=0x5c59fcb0, 
e=<optimized out>) at session_impl.cpp:3539
#6  0x5e357c82 in operator() (p=<optimized out>, this=0x60507dd4, a1=...) at 
/test/torrent/boost/build/include/boost/bind/mem_fn_template.hpp:165
#7  operator()<boost::_mfi::mf1<void, libtorrent::aux::session_impl, const 
boost::system::error_code&>, boost::_bi::list1<const 
boost::system::error_code&> > (a=<synthetic pointer>, f=..., this=0x60507ddc) 
at /test/torrent/boost/build/include/boost/bind/bind.hpp:313
#8  operator()<boost::system::error_code> (a1=..., this=0x60507dd4) at 
/test/torrent/boost/build/include/boost/bind/bind_template.hpp:47
#9  operator() (this=0x60507dd4) at 
/test/torrent/boost/build/include/boost/asio/detail/bind_handler.hpp:40
#10 asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, 
boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::system::error_code 
const&>, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, 
boost::arg<1> > >, boost::system::error_code> > (function=...) at 
/test/torrent/boost/build/include/boost/asio/handler_invoke_hook.hpp:64
#11 invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, 
boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::system::error_code 
const&>, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, 
boost::arg<1> > >, boost::system::error_code>, boost::_bi::bind_t<void, 
boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::system::error_code 
const&>, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, 
boost::arg<1> > > > (function=..., context=<optimized out>) at 
/test/torrent/boost/build/include/boost/asio/detail/handler_invoke_helpers.hpp:3
9
#12 boost::asio::detail::wait_handler<boost::_bi::bind_t<void, 
boost::_mfi::mf1<void, libtorrent::aux::session_impl, boost::system::error_code 
const&>, boost::_bi::list2<boost::_bi::value<libtorrent::aux::session_impl*>, 
boost::arg<1> > > >::do_complete (owner=<optimized out>, base=0x5cd80c80) at 
/test/torrent/boost/build/include/boost/asio/detail/wait_handler.hpp:64
#13 0x5e3237a2 in complete (owner=..., this=0x5cd80c80) at 
/test/torrent/boost/build/include/boost/asio/detail/task_io_service_operation.hp
p:35
#14 boost::asio::detail::task_io_service::do_one (this=0x5c8e9670, lock=..., 
this_idle_thread=0x60507e7c) at 
/test/torrent/boost/build/include/boost/asio/detail/impl/task_io_service.ipp:278
#15 0x5e324440 in boost::asio::detail::task_io_service::run (this=0x5c8e9670, 
ec=<optimized out>) at 
/test/torrent/boost/build/include/boost/asio/detail/impl/task_io_service.ipp:130
#16 0x5e355e68 in libtorrent::aux::session_impl::main_thread (this=0x5c59fcb0) 
at session_impl.cpp:4756
#17 0x5e357d36 in operator() (p=<optimized out>, this=<optimized out>) at 
/test/torrent/boost/build/include/boost/bind/mem_fn_template.hpp:49
#18 operator()<boost::_mfi::mf0<void, libtorrent::aux::session_impl>, 
boost::_bi::list0> (f=<optimized out>, this=<optimized out>, a=<optimized out>) 
at /test/torrent/boost/build/include/boost/bind/bind.hpp:253
#19 operator() (this=<optimized out>) at 
/test/torrent/boost/build/include/boost/bind/bind_template.hpp:20
#20 boost::asio::detail::posix_thread::func<boost::_bi::bind_t<void, 
boost::_mfi::mf0<void, libtorrent::aux::session_impl>, 
boost::_bi::list1<boost::_bi::value<libtorrent::aux::session_impl*> > > >::run 
(this=<optimized out>) at 
/test/torrent/boost/build/include/boost/asio/detail/posix_thread.hpp:82
#21 0x5e31ee06 in boost::asio::detail::boost_asio_detail_posix_thread_function 
(arg=<optimized out>) at 
/test/torrent/boost/build/include/boost/asio/detail/impl/posix_thread.ipp:64
#22 0x401d13dc in __thread_entry () from /test/obj/local/armeabi-v7a/libc.so
#23 0x401d0ac8 in pthread_create () from /test/obj/local/armeabi-v7a/libc.so
#24 0x00000000 in ?? ()

Original comment by ygao....@gmail.com on 19 Feb 2014 at 2:29

[GoogleCodeExporter]

If split into multi files for storage, it always goes to finished state early 
before download whole. how to trace this issue?

Original comment by ygao....@gmail.com on 20 Feb 2014 at 8:31

[GoogleCodeExporter]

if you would like to help, writing a small example program that reproduces the 
issue would significantly help me to fix it. ideally in the form of a unit test.

if this happens consistently, you could set a breakpoint in 
torrent::finished(), which is called when the download completes. if that's 
being called early, inspecting the frames up the stack would probably reveal 
why it thinks the torrent is completed.

Original comment by arvid.no...@gmail.com on 28 Feb 2014 at 5:18

[GoogleCodeExporter]

I wrote a unit test remap_files and I'm not able to reproduce this. Are there 
any more hints you can provide to help me track this down?

Original comment by arvid.no...@gmail.com on 5 Mar 2014 at 9:38

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

torrent.hpp:    bool is_finished() const{

num_filtered        788
num_have        256
num_pieces      1044

}
why these pieces are filtered out in picker?

Original comment by ygao....@gmail.com on 8 Mar 2014 at 4:38

[GoogleCodeExporter]

it's possible that if writing to the files fails for some reason (other than 
disk full), those files may be deselected to avoid the error again. If this is 
the case, you should have seen error alerts.

Original comment by arvid.no...@gmail.com on 9 Mar 2014 at 12:57

[GoogleCodeExporter]

Before split file, I set a file priority for one file. After split, it becomes 
a multi files. But file priority still keep the same. That is why it become 
seeding after first part done. I believe if you can reproduce this assertion 
fail by this way.

Original comment by ygao....@gmail.com on 9 Mar 2014 at 4:50

[GoogleCodeExporter]

you mean the new files will get priority 0 when you split? and only the first 
file will have the priority you set?

Original comment by arvid.no...@gmail.com on 10 Mar 2014 at 1:32

[GoogleCodeExporter]

Only the file being downloaded will get priority 1 and other files are 0. If 
that file is too large, it will get split into multi parts. So if priority 
setting isn't changed, only the first file part get priority 1. 

Original comment by ygao....@gmail.com on 10 Mar 2014 at 3:02

[GoogleCodeExporter]

I see. That makes a lot of sense. remap_files() does not adjust any file 
priorities. File priorities are assigned by the index of the file, so if you 
insert more files at the start, the priorities you set for the previous file 1 
will apply to the new file you added. Any files that don't have a priority set 
(i.e. the last file that would be pushed out beyond the last priority) would 
default to priority 1.

I realize this should probably be documented in the remap_files() call. The 
reason for libtorrent not performing any shifting of file priorities is because 
remap_files() is very generic. It wouldn't necessarily be intuitive behavior if 
you're mapping from two completely disconnected sets of files (for instance, if 
the file structure is not important and the whole torrent file is simply 
chopped up in equal chunks).

Can you think of anything libtorrent could do to simplify this case?
would it be reasonable to clear all file priorities (back to default of 1)? 
maybe that would be an easier error to catch.

Otherwise, for now it seems like this is primarily a documentation issue. Would 
you agree? or am I missing something?

Original comment by arvid.no...@gmail.com on 10 Mar 2014 at 7:07

[GoogleCodeExporter]

oh, right.. there's the assert too. I'll look into that.

Original comment by arvid.no...@gmail.com on 10 Mar 2014 at 7:10

[GoogleCodeExporter]

I extended the test_remap_files unittest to try to cover this case without any 
luck reproducing it.
Could you describe exactly how you add the torrent, remap the files and set 
file prioritiy?

If you use remap_files(), you do that on the torrent_info object _before_ it's 
added, right? If you then try to set file priorities on the files you'll hit an 
assert if you specify the wrong number of files. As far as I can tell, there is 
no way to remap files _after_ having set priorities.

thanks

Original comment by arvid.no...@gmail.com on 10 Mar 2014 at 9:26

[GoogleCodeExporter]

In client_test:
int add_torrent(libtorrent::session* ses, ...) {

  using namespace libtorrent;
  boost::intrusive_ptr<torrent_info> t;
....
if (file_entry_array != NULL) {
        file_storage st;
        for (int i = 0; i < file_split_num; ++i)
            st.add_file(*file_entry_array[i]);
        t->remap_files(st);        
    }
  add_torrent_params p;
...
}

void setTorrentFilesPriority() {
  if (handle->has_metadata()) {
    libtorrent::torrent_info const& info = handle->get_torrent_info();
    std::vector<int> priorities;

    for (int i = 0; i < info.num_files(); ++i) {
        if (i == file_need)
            priorities.push_back(1);
        else
            priorities.push_back(0);
    }
    handle->prioritize_files(priorities);
  }
}

bool handle_alert() {

  if (add_torrent_alert* p = alert_cast<add_torrent_alert>(a))
  {
....
      handle = new torrent_handle(h);
      setTorrentFilesPriority();
      return;
  }
}

void beforeAddTorrent() {

  file_entry_array = new libtorrent::file_entry* []
...
  file_entry_array[entry_index] = new libtorrent::file_entry();
  file_entry_array[entry_index]->path = file_entry.path;
  file_entry_array[entry_index]->offset = file_entry.offset;
  file_entry_array[entry_index]->size = file_entry.size;
  file_entry_array[entry_index]->file_base = 0;                

...
}

Original comment by ygao....@gmail.com on 11 Mar 2014 at 3:00