The question of increasing flexibility

[master255]

@gubatron I tried to use this library and it's okay. But there are problems. I tried to discuss these problems with the developer: https://github.com/arvidn/libtorrent/issues/4387. But he said he couldn't change the functionality of the library in any way. This library is made up of different components. I figured out how each component works. The most complex logic is the UTP protocol in conjunction with DHT on one UDP port. Everything else, it's very easy to write in any programming language. That's why everyone needs the library of UTP and DHT technologies together, and this library doesn't give necessary flexibility for development. https://github.com/bittorrent/libutp or https://github.com/rtttech/utp https://github.com/bittorrent/libbtdht Can you help with porting these libraries to Java? This will give maximum flexibility in developing torrent applications.

[master255]

Ok. No answer. Goodbye.

[gubatron]

Damn... ok, it takes a while to get to all tickets.

FYI, there are other ongoing efforts to break the DHT portion into its own independent component.

But with that attitude good luck dude. Patches are welcome, but I doubt you have the patience.

[master255]

@gubatron

Damn... ok, it takes a while to get to all tickets.

Maybe there's something I don't understand. There's two issues. What's the problem with keeping track of it? I've been waiting two weeks for an answer. I hope you'll answer more often.

FYI, there are other ongoing efforts to break the DHT portion into its own independent component.

DHT must be with UTP. If there are attempts to make a separate component, only with UTP+DHT, then this is very good. It's just that a separate DHT is not interesting.

At this point I have a separate module (DHT + UTP Server (on one port))+UTP clients. It is written in pure Java for Android. But it has some bugs that are not yet debugged. Yesterday I wrote an algorithm to get clean data from UDP packets that make up UTP packets that make up Torrent messages. This gave the maximum performance when downloading data.

The main problem is that I don't have any information for development. No one is helping.

If they do the work just to separate the DHT module, it is not interesting. If they are doing work just to separate the UTP module, it is not interesting.

Tell them that I will finish my module faster than they can do anything.

[master255]

@gubatron So two ways to help me:

1. It is possible to port the libraries that I have indicated above. And we'll see together how that can help.
2. You can answer my questions about UTP and DHT technologies and help me write my library.

And the third way: do nothing. Admit your weakness.

[gubatron]

1. Yes.
2. You're funny. "Admit your weakness.", how about "busy", "not interested". RTFC.

[gubatron]

DHT must be with UTP. If there are attempts to make a separate component, only with UTP+DHT, then this is very good. It's just that a separate DHT is not interesting.

I agree, separate DHTs are not interesting.

If they do the work just to separate the DHT module, it is not interesting.

I believe this is a branch for such effort, perhaps you can help him (given your ego and claims of being faster than anybody else, show me your code), a lot has to be done before you can extract DHT component and have it fully independent.

[gubatron]

would you be down to implement a stupid simple as f$$k UTP library in plain old C? if it becomes too hard not having/using things like hashmaps we can build simple abstractions inspired by Java's collection , this should be doable if you already have

It is written in pure Java for Android. an implementation, that'd makes me think you're familiar with Map<Map<K,V>>, List<T>.

did you manage to the code congestion control in your Java implementation?

... Well, then imagine you have a rock solid and performant easy to use, event-loop-based, C library.

From there you make a high-level wrapper for C++, or Java, or GoLang, or even for NodeJS.

[master255]

@gubatron

would you be down to implement a stupid simple as f$$k UTP library in plain old C?

For me, the Java version is interesting because of thread control. Frankly, I have no idea how to use the C UTP library in Android. How hard do you think it is? I wasn't looking at the C library.

did you manage to the code congestion control in your Java implementation?

That's a good question. I'm trying to figure out how to do it right now. Here's my code: https://github.com/master255/UtpJava My main problem now is: the download stops at the start for a short time and then stops when the maximum speed is reached. The server starts to send the packets very slowly. There is no way that I can figure out what is going on. There is no packet loss, though. Can you explain how to add congestion protection? TCP doesn't have that problem.

[master255]

@gubatron Yesterday, I tested first what happens when send window size 0. And then I realized how it works. I read RFC 5681 and didn't understand anything. But then I wrote an algorithm that prevents data transfer overload. Now the data transfer is stable. I'm using a slow start. The algorithms still have to be written, but I understood the main principle.

[master255]

Okay. You've been answering too long. Bye.

[gubatron]

hi, sorry for long lag, maybe you've managed to answered some questions. I believe I'll be getting back to this project starting tomorrow and I'll be a lot more responsive, deploying another unrelated project, hadn't checked email in days

[gubatron]

got a 404 on your project https://github.com/master255/UtpJava couldn't find any public repo on your github account that looked anything like it.

[master255]

@gubatron For today, I believe that the UTP protocol works well and stably. I tested it on large files. Now I have a problem with DHT. I'm at a dead end. My DHT module is not working well enough. So I'm trying to study other modules: https://github.com/zpqsunny/dht. https://github.com/the8472/mldht And I can't use those modules. Since they are written in Java patterns, which are only supported from Android 8, I need support from Android 4.4. At the minimum, I need to translate them to normal Android patterns. And I need to transfer their logic to my module, because my module works with UTP on one port and uses a special cache to receive and send packets. You can help with a recommendation on how to change the code, but I think that is a lot of work. Or you can do porting of DHT C++ libraries. This could help, too.

[gubatron]

My advice: Don't reinvent the wheel, use jlibtorrent to connect to the actual DHT using the standard, optimized and battle tested C++ implementation. (What we do at FrostWire)

I'd announce my peers on some pre-ordained hash id:

SessionManager s = new SessionManager();
s.addListener(myAlertLoopListener);
s.start();

// announce all peers on the people square
s.dhtAnnounce(yourNetworkAnnounceSha1AddressHere, port, flags);

// and then ask who's here to connect with
ArrayList<TcpEndpoint> endPoints = dhtGetPeers(yourNetworkAnnounceSha1AddressHere, timeout);

then do as you please with those end points, have them exchange information necessary to set up your uTP connections. Not sure if that makes sense at all for what you're trying to do.

You can announce and find items on the DHT using the jlibtorrent API

Here's a shell program so you can play with it and see the API usage https://github.com/frostwire/frostwire-jlibtorrent/blob/master/src/test/java/com/frostwire/jlibtorrent/demo/DhtShell.java

[master255]

@gubatron I decided not to stop at the DHT and move on (now i use mlDHT temporarily). Thank you for your answer. I'll probably do it in the future, but I need more flexibility. Otherwise I won't be able to do UTP and DHT on the same port. I need IN OUT DatagramPacket. Maybe you don't know what I want to do and for what I'm trying to write my torrent components. My main problem: Lack of sparse mode on disks for my program. My program always has one file for downloading through torrents. I created a bug: https://github.com/arvidn/libtorrent/issues/4616 To solve this problem, I need to download a torrent from a certain byte. The library doesn't have that option. I need not only to set priority to the piece, I need to set it an offset. And download different pieces of the same file into different files. Do you have any idea how to do this with a library?? I know about channel aggregation and hash verification. But we need to do it.

[gubatron]

you talk about byte granularity, can't you try and use the piece message with libtorrent? piece: <len=0009+X><id=7><index><begin><block> (where X is the size of block)

you'd have to do the math to map on which piece your byte offset belongs to and then make the necessary request across pieces if necessary to obtain the data.

you should be able to grab the PIECE_FINISHED alert message and have access to the received bytes, then you should be free to save those bytes into the different files you mention.

basically it sounds like you're doing your own piece picker.

http://libtorrent.org/reference-Alerts.html#read_piece_alert

[master255]

@gubatron

basically it sounds like you're doing your own piece picker.

No. I need to download data sequentially, but starting with a certain byte. When the necessary part of the file is downloaded, I will simply turn off the download.

For drives without sparse mode support, I invented a simple technology. Read this, it works very easily: https://github.com/arvidn/libtorrent/issues/4616#issuecomment-625876774.

you talk about byte granularity, can't you try and use the piece message with libtorrent? piece: (where X is the size of block) you'd have to do the math to map on which piece your byte offset belongs to and then make the necessary request across pieces if necessary to obtain the data.

problems:

1. The main problem is creating a file. The library will try to create an entire file on disk. It takes a very long time from 10 minutes to 3 hours. This is not right. I need to download files from 2GB to 120GB (single file size). Libtorrent hangs completely at this point and cannot download these files without sparse mode. I'm trying to solve this problem.
2. Files must be stored on the disk starting with a certain byte. Otherwise, it is very difficult to navigate from which file should be read. My multimedia player already works with this file storage technology.

[frostwire]

No. I need to download data sequentially, but starting with a certain byte.

Exactly, you need your own piece picker.

Even if you tell libtorrent to download sequentially, the piece picker won't strictly pick pieces contiguosly, just watch a torrent download on sequential mode and you'll see it's not that strict.

If I were you I'd try turning off the piece picker and create my own, that given a byte offset it would do the simple math of determining to which piece it belongs to and then do the request for such piece at some given piece offset height and length, to satisfy your certain byte need.

as of "the library will try to create an entire file on disk", you can also turn this off, we've done this for our Torrent Fetcher, we implemented an "In Memory Storage" interface, I'm sure you can do your own as well, and then you can use whatever disk allocation strategy you need.

[frostwire]

Not exactly what you need, but you can always turn off the storage and store the bytes received yourself. This is how we did it for fetching a magnet, perhaps you can do the same by disabling storage and handling messages when pieces are received.

public byte[] fetchMagnet(String uri, int timeout, final boolean extra, final int maxSize) {
        if (session == null) {
            return null;
        }

        error_code ec = new error_code();
        add_torrent_params p = add_torrent_params.parse_magnet_uri(uri, ec);

        if (ec.value() != 0) {
            throw new IllegalArgumentException(ec.message());
        }

        p.set_disabled_storage(); //👈

        final sha1_hash info_hash = p.getInfo_hash();
        final byte[][] data = {null};
        final CountDownLatch signal = new CountDownLatch(1);

        AlertListener listener = new AlertListener() {
            @Override
            public int[] types() {
                return METADATA_ALERT_TYPES;
            }

            @Override
            public void alert(Alert<?> alert) {
                torrent_handle th = ((TorrentAlert<?>) alert).swig().getHandle();
                if (th == null || !th.is_valid() || th.info_hash().op_ne(info_hash)) {
                    return;
                }

                AlertType type = alert.type();

                if (type.equals(AlertType.METADATA_RECEIVED)) { // 👈
                    MetadataReceivedAlert a = ((MetadataReceivedAlert) alert);
                    int size = a.metadataSize();
                    if (0 < size && size <= maxSize) {
                        data[0] = a.torrentData(extra); //👈
                    }
                }

                signal.countDown();
            }
        };
...

[master255]

@frostwire Thank you. I'll try and write about the results.

[master255]

@frostwire Okay. Now I have: (ReadPieceAlert)Alerts.cast(a)).bufferPtr(). It's a long value. How do I convert it to byte[]? Or is it the wrong message type?