[BUG] Android - only maxSimultaneouslyAssignedPieces are downloaded if PieceSelector.getNextPieces supplies subset of all pieces in torrent

[pvishnyakov]

Let's assume we have some torrent containing N pieces and we want to download only first 5 of them using the customized PieceSelector and the following code:

    int[] piecesToDownload = new int[]{0,1,2,3,4};
        final PieceSelector pieceSelector = new PieceSelector() {
        @Override
        public IntStream getNextPieces(BitSet bitSet, PieceStatistics pieceStatistics) {
            return IntStream.of(piecesToDownload);
        }
    };

        Torrent torrent;
        final int selectedFileIndex = 0;
    final FilePrioritySkipSelector filePrioritySkipSelector =
            torrentFile -> {
                if (torrent != null) {
                    return torrent.getFiles().indexOf(torrentFile) == selectedFileIndex
                            ? FilePriority.HIGH_PRIORITY
                            : FilePriority.SKIP;
                } else return FilePriority.SKIP;
            };

    BtRuntime createRuntime() {
        try {
            Config config = new Config() {
                @Override
                public EncryptionPolicy getEncryptionPolicy() {
                    return EncryptionPolicy.PREFER_ENCRYPTED;
                }
                @Override
                public int getNumOfHashingThreads() {
                    return Runtime.getRuntime().availableProcessors();
                }

                @Override
                public int getAcceptorPort() {
                    return 6891;
                }
            };

            Module dhtModule = new DHTModule(new DHTConfig() {
                @Override
                public boolean shouldUseRouterBootstrap() {
                    return true;
                }
            });

            return BtRuntime.builder(config)
                    .autoLoadModules()
                    .module(dhtModule)
                    .module(new UpnpPortMapperModule())
                    .module(new PeerExchangeModule())
                    .module(new HttpTrackerModule())
                    .build();
        } catch (Exception e) {
            System.out.println("Error creating BT runtime "+e);
            return null;
        }
    }

    BtClient createClient(URI uri, BtRuntime runtime, Storage storage,
                                  FilePrioritySkipSelector filePrioritySkipSelector,
                                  PieceSelector pieceSelector) {
        try {
            return Bt.client(runtime)
                    .storage(storage)
                    .torrent(uri.toURL())
                    .afterTorrentFetched(t->{torrent = t;})
                    .fileSelector(filePrioritySkipSelector)
                    .selector(pieceSelector)
                    .stopWhenDownloaded()
                    .initEagerly()
                    .build();
        } catch (Exception e) {
            System.out.println("Error creating BT client "+ e);
            return null;
        }
    }

    Storage storage = new FileSystemStorage(Paths.get(context.getFilesDir().getAbsolutePath()+"/tordl"));

    void startDownload(URI uri) {
        System.out.println("Creating BT runtime...");
        BtRuntime runtime = createRuntime();

        System.out.println("Creating client...");
        BtClient client = createClient(uri, runtime, storage, filePrioritySkipSelector, pieceSelector);

        System.out.println("Starting download...");
        client.startAsync(state -> {
            final int activePeers = state.getConnectedPeers().size();
            final int piecesComplete = state.getPiecesComplete();
            System.out.println("Peers: " + activePeers
                + "; Downloaded: " + piecesComplete + " pieces, current range: "
                + Arrays.toString(piecesToDownload));
        }, 5000);
        System.out.println("Download started");
    }

On Android device (tested Android 9 and 11) we will not get 5 pieces, just 3 pieces (0,1,2) will be downloaded which is equivalent to default Config.maxSimultaneouslyAssignedPieces value. If we increase maxSimultaneouslyAssignedPieces to 5 or more, all 5 pieces will be downloaded but only from single peer. Here comes the next trouble - any IntStream of pieces provided by custom PieceSelector in Android will not be distributed between peers, in the best case (range size<=maxSimultaneouslyAssignedPieces) all pieces will be assigned to one peer and downloaded, in worst case (range size>maxSimultaneouslyAssignedPieces) only maxSimultaneouslyAssignedPieces pieces from desired range will be assigned to one peer and downloaded.

To reproduce try to download using any torrent link (for example http://d.rutor.info/download/934413):

        startDownload(URI.create("http://d.rutor.info/download/934413"));

If we try the same code on desktop (Windows), everything works as expected - any piece range supplied by custom PieceSelector will be distributed between peers and downloaded assigning up to maxSimultaneouslyAssignedPieces pieces to each peer.

If we use another PieceSelector (for example SequentialSelector) - whole torrent will be downloaded and piece distribution between peers seems to work as expected even on Android device.

What may be the problem with Android?

[pvishnyakov]

Found the problem, customized PieceSelector is wrapped into the PrioritizedPieceSelector and then into ValidatingSelector which is making impossible to control the pieces feed. Changed PrioritizedPieceSelector back to PieceSelector in all occurrences from Assignment and up, result is more predictable now, I can feed the pieces range by range maintaining the desired sequence order.

It doesn't explain why it worked in desktop Java, however, it's working now. Please consider update to the Core library.

[pyckle]

Hi @pvishnyakov. I believe what you wrote is not a valid piece selector. The interface specifies that the piece selector returns chunks that are relevant. In your selector, it may return chunks that are already downloaded.

I believe what you're trying to do was already implemented in the class https://github.com/atomashpolskiy/bt/blob/master/bt-core/src/main/java/bt/torrent/selector/SequentialSelector.java

Instead of returning all pieces, it only returns the pieces that haven't been downloaded or already allocated.

[pvishnyakov]

I want the file to be downloaded range by range, for example [0,1,2,3], then [4,5,6,7], [8,9,10,11] and so on. So my getNextPieces is always like return IntStream.of(range) where range is the int[] array of pieces to download.

ValidatingSelector is still there to prevent the downloaded pieces from being re-assigned and it's ok to have ValidatingSelector(MySelector) wrapping. But originally we have ValidatingSelector(PrioritizedPieceSelector(MySelector)) wrapping which is making hard to control the sequence. Without PrioritizedPieceSelector it works as expected now, I don't know why but it does.