joriscarrier
commented
4 years ago Closed master255 closed 4 years ago
joriscarrier
commented
4 years ago 
master255
commented
4 years ago @joriscarrier No no! DHT is not a complex module, but a complex UTP module. DHT - must be part of UTP, because they are on the same UDP port. It receives and distributes UDP packets to each other. DHT does not interest me as a separate module. It is not too complicated.
master255
commented
4 years ago But I want an answer from @arvidn . @arvidn , you're the author of this https://www.bittorrent.org/beps/bep_0029.html. You're writing an article about a very complex data transfer algorithm. But there is no description of this algorithm anywhere. Just general phrases and nothing concrete. I can't use your libtorrent library because it's conservative. There are no descriptions of the UTP protocol anywhere on the Internet. If you invented the technology, either make it paid, or publish the algorithm in a schematic form, or make the library module in all major programming languages. Otherwise, this is inadequate. None of the programmers do that. Will you decide whether you want others to use your algorithm or not?
arvidn
commented
4 years ago You're writing an article about a very complex data transfer algorithm. But there is no description of this algorithm anywhere. Just general phrases and nothing concrete.
You should look harder. There's source code implementing uTP on github.
There are no descriptions of the UTP protocol anywhere on the Internet.
You should look harder.
If you invented the technology, either make it paid, or publish the algorithm in a schematic form, or make the library module in all major programming languages.
I don't want to take credit for uTP. I would say the bulk of the novel work was done by the author of the IETF RFC. But, again, you should look harder.
I'm positive you can hire an engineer to figure this out for you, if you want it paid.
Otherwise, this is inadequate. None of the programmers do that.
If you want people to help you, I would strongly suggest you just ask nicely. Someone might even post links for you. Being rude and demanding that people work for free for you won't get you anywhere I don't think.
In fact, demonstrating good faith attempts of trying to understand it yourself would be a really good start.
master255
commented
4 years ago @arvidn
You should look harder. There's source code implementing uTP on github.
Really? I'm an Android Java programmer and write the most complicated UTP data transfer algorithm and have to study c++ code to understand how it works? It sounds like fiction. Even in Java it is very difficult for any programmer. In c++, it's fantastically complicated! Nobody does that. There is no second, working UTP library in the whole world! Even more so in a different language! Open your eyes! I know a lot of these projects: https://github.com/iiljkic/utp4j But none of them work and will never earn! That's because you don't publish all the information.
There are no descriptions of the UTP protocol anywhere on the Internet.
You should look harder.
Okay. Where is the description of the order of sending and receiving packages? I got the UDP data packet, what do I do next? What ack should I start with and what should I do when the numbering reaches 65535? Where is the full information?
Usually, normal people, if they try to describe an algorithm, especially a complex algorithm, they do an algorithm diagram with rhombuses and circles - start, finish. You know that. From your article, I do not understand how this algorithm works. I read it 100 times, no exaggeration.
If you want people to help you, I would strongly suggest you just ask nicely.
I have great respect for all programmers in this world. But sometimes emotions take over. I have great respect for your work and achievements. But my emotions are necessary for us to move on. Achievement is good, and progress is even better. And I'm overwhelmed by emotions if someone stops me on the way to progress. It's nothing personal, it's karma.
Someone might even post links for you.
I have all the links I can get on the UTP subject. I'm sure of it. That's why I'm so angry. I checked the entire Google search for 20 pages. I've looked all over Githab. I read and looked for comments. I've read the entire website www.bittorrent.org.
FranciscoPombal
commented
4 years ago @master255 The congestion control algorithm uTP uses is called LEDBAT, and it is documented in an IETF RFC: https://tools.ietf.org/html/rfc6817
Besides libtorrent's implementation of uTP, here is a non-exhaustive list (includes several other languages):
libutp)uTP itself is documented here: https://www.bittorrent.org/beps/bep_0029.html. If you think this spec needs improvements, you are more than welcome to suggest them. Documents like these benefit from multiple people trying to implement the spec in different languages. Their experiences in doing so can provide valuable feedback for improving the documentation of the spec. For example I just came across this (https://github.com/jlouis/erlang-utp/blob/master/TODO.org), which seems to be quite a goldmine of info that could be used to figure out where the spec needs clarifications.
Okay. Where is the description of the order of sending and receiving packages? I got the UDP data packet, what do I do next? What ack should I start with and what should I do when the numbering reaches 65535? Where is the full information?
Usually, normal people, if they try to describe an algorithm, especially a complex algorithm, they do an algorithm diagram with rhombuses and circles - start, finish. You know that. From your article, I do not understand how this algorithm works. I read it 100 times, no exaggeration.
Demanding stuff like this isn't helpful. You should ask nicely, or, better yet, suggest clarifications and/or improvements to the document (constructive criticism). You are not the first to criticize the spec: https://github.com/anacrolix/utp/blob/master/README.md. But I don't see people who've been through implementing uTP suggesting concrete improvements to the document based on their experience...
master255
commented
4 years ago @FranciscoPombal
https://github.com/bittorrent/libutp - This is the only library https://github.com/rtttech/utp C++ language, too. Next version? https://github.com/anacrolix/utp deprecated \ not worked https://github.com/anacrolix/go-libutp wrapper. Not Library! https://github.com/mafintosh/utp - not worked. In development. https://github.com/jlouis/erlang-utp Ha ha))) It's a very old language. Last modification eight years ago. It's not working. https://github.com/meqif/rust-utp Read read.me. Incomplete. Not even version 1. It doesn't work. Can't be used.
KTorrent, Transmission, Vuze/Azureus/BiglyBT
What's that? Give me the links to the libraries and I'll tell you what's wrong with them. Total: 1 work library libutp or rtttech/utp It doesn't matter. Look at how many people were willing to use UTP protocol and how we see no one could! That proves what I said!
The congestion control algorithm uTP uses is called LEDBAT, and it is documented in an IETF RFC: https://tools.ietf.org/html/rfc6817
It's the same as sending me to learn C++ to learn some of the most complex algorithms in the world. Do not teach to live, help materially)))
uTP itself is documented here: https://www.bittorrent.org/beps/bep_0029.html. If you think this spec needs improvements, you are more than welcome to suggest them.
Okay. I suggest we add an algorithm diagram. You can do it in Visio or any other schema editor.
It's a very long text. Very random information. It's like constructing a car by reading mechanics' diaries.
But I don't see people
Of course there's nobody. Because there are no instructions or clear explanations for how the UTP protocol works. There is one library in c++. And that's all.
master255
commented
4 years ago And it would be nice to explain how DHT works. Because DHT is very much connected to UTP. But I think the mission is impossible. Right?
arvidn
commented
4 years ago I don't really know what you expect from an "algorithm diagram". Could you show me an example of how TCP is described with a diagram?
I assume by "algorithm" you mean the congestion controller. That's the closest thing in there. It's explained both in the RFC and in BEP. It's a single formula that's evaluated every time an ACK is received. Just like TCP, uTP is ACK clocked.
Btw, the LEDBAT congestion controller has been implemented and understood by many people. I think it's safe to assume the documentation is complete in this regard
libswift is another example (from delft university)
FranciscoPombal
commented
4 years ago @master255 you keep claiming most libraries don't "work". Don't work for what purpose exactly? Maybe not for yours specifically, but in that case I'm sure their respective authors would love for you to submit improvements to them.
Okay. I suggest we add an algorithm diagram. You can do it in Visio or any other schema editor.
How about you do it, instead of complaining? Get the work started, ask questions as you reach points where you think the documentation is insufficient, and people will collaborate with. Iterate, rinse, and repeat. Barging in and demanding things be done accomplishes nothing.
And it would be nice to explain how DHT works. Because DHT is very much connected to UTP. But I think the mission is impossible. Right?
The concept of the DHT has been well understood and documented for a long time. You are clearly not looking hard enough. There are countless youtube videos, webpages, academic papers and existing implementations that you can study.
To put all this behind us and start being productive, I would recommend you start with this:
I don't really know what you expect from an "algorithm diagram". Could you show me an example of how TCP is described with a diagram?
master255
commented
4 years ago @arvidn
I don't really know what you expect from an "algorithm diagram". Could you show me an example of how TCP is described with a diagram?
I assume by "algorithm" you mean the congestion controller.
Not only that. By the algorithm I understand the whole algorithm of UTP + interaction with DHT. This is a mechanism for distributing incoming data packets. And many other things...
Okay. If you're not good at algorithm scheming, then I have a compromise. We need to build a library that everyone can use. Java is much easier to understand for any programmer than C++. I'm just writing a library like this and publishing it. The library is written in pure Java, for all Android versions. The library is already working, but there are many questions. Let's create a new issue: Questions about the UTP protocol. Ok?
arvidn
commented
4 years ago Very much material: ...
Most of those are graphs explaining the congestion control. Please read all those papers you found, they will help you understand transport protocols in general, which is helpful in understanding uTP as well. If you want graphs for uTP, here are some: https://libtorrent.org/utp.html
You can also generate your own with a python script in the libtorrent repo.
Not only that. By the algorithm I understand the whole algorithm of UTP + interaction with DHT. This is a mechanism for distributing incoming data packets. And many other things...
I see, you're using the word "algorithm" to generally refer to "how something works". Explaining how uTP works can surely be aided by illustrations. But that would just be a pedagogical tool. The target audience of most uTP documentation is someone who already knows how transport protocol works in general. There's plenty of text books and papers you can learn from.
Okay. If you're not good at algorithm scheming,
I'm just not interested, but sure. :)
then I have a compromise. We need to build a library that everyone can use. Java is much easier to understand for any programmer than C++. I'm just writing a library like this and publishing it. The library is written in pure Java, for all Android versions.
Sure, go ahead. I don't have time (or interest really) to be involved though.
@arvidn Hello. I've already realized that your library doesn't have enough flexibility for my development. Well, that's normal. Here's a question: if I can't use the library fully, can I use it by modules? For example, I now need a DHT+UTP module. Can I use the library only to get a list of peers from DHT and send them my commands via the UTP protocol? And get data like with the TCP protocol.
Everything else has a very simple logic. As opposed to UTP and DHT on the same port.