[FILES AND CHANNELS] Time to focus to get a profesional product.

[felisucoibi]

[FILES AND CHANNELS] Time to focus to get a profesional product.

Lot of things are being done in files and channels the last 12 months, i think now is the moment to make a good product, it's time to focus.

There are 4 things very important in retroshare by now that needs lot of work, FILES CHATS CHANNELS nad FORUMS.

I propose now for 0.6.4 focus in FILES and CHANNELS and for 0.6.5 focus in CHATS and FORUMS.

Why first FILES and CHANNELS, because FORUMS and CHATS need more work, and we can finis 0.6.4 fast, and also is what people most uisng right now in RS.

I can write the same text for 0.6.5 where i can talk about what forums need, what chat needs, and the android app needs, for example we can learn a lot from TOX https://tox.chat/faq.html#how-tox-privacy to create our app, and maybe we can include TOX as a replacement of our chat system.

When i say focus, i mean to have a TIER1 product, not just a toy product.

What's the difference between a toy product and a professional product? They look similar. They do the same thing. But the professional one works with heavy work on it, the toy just breaks.

RS is still a toy, a toy that is used everyday by more people, and more will come but is not ready for professional use, and when the heavy work arrives it will break, and people will use another tool.

So i write this just to try to help in the task of make a professional tool with retroshare.

Let's go for the professional missing stuff of RS in Files and Channels.

Please add to this thread anything you thing i miss for a professional tool and I can add them.

CHANNELS:

CHANNEL POSTS:

1. HEAVY LOAD OF FILES AND SPEED IN A CHANNEL POST. Rs works as a toy, you can create a post in channel with 10 files and more or less work, you add 20 and starts to go slow, add 50 and you have hangs, add 100 and you can't browse the channel post. We need for professional use to be able to handle 500 - 1000 files in a channel post without slownesses. PROPOSAL TO FIX IT: Code optimization, open a channel post in a tab, not in the same channel posts list. Maybe this is fixed if NEW LINKS USED instead of attached files.

CHANNEL:

1. HEAVY LOAD OF CHANNEL POSTS IN A CHANNEL. Rs works as a toy, you can create a channel with 10 files and more or less work, you add 20 and starts to go slow, add 50 and you have hangs, add 100 and you can't browse the channel post. We need for professional use to be able to hande 500 - 1000 posts in a channel without slowneses. PROPOSAL TO FIX IT: Code optimization, review the way channel posts are shown, and maybe like in LINKS pages to go NEXT and NEXT but without loosing the search engine to search.

CHANNEL TOP POST: We need to mark some posts as important in channels, so they are on the top of the channel posts lists. This can be usefull to offer special information for the channel, like rules, or special things to be offered, like the latest RS BUILD in a channel with lot of things inside.

FILES:

FILE LISTS: File lists are the heart of FILES, without good file lists we don't have FILES experiences, just kind of not very good search engine....

2. BIG FILE LISTS BROWSING: In rs as a toy a file list works, but if you put 5000 files inside a directory RS hangs, 5000 files is not too much for a directory, is very common in lot of uses to have such quantity, is not professional to hang with 5000 files. Or less, i think it starts to hang wint 500.... PROPOSAL TO FIX: First i proposed a new to show file lists, in a three level way like DC++ does, using: first section with three columns inside for usernames + total shared TB + total files. second section only for directory tree of a user, not files loaded there. third section for filenames and directories. Doing this you can change very easily from a user to another just in first section, you can easyli change between directories in second section, and you only load files in third section, for files, so hopefully load 5000 files is just easy and yo don't loose the control of file list browsing.

3. Handle file downlaods when disk is full, now if disk is full temproal file completed is not moved and is lost forever, rs can check if there is enough space to move a file first, and if is not warn user, and wait to finish download and move to have enough space.

DOWNLOAD QUEUES:

1. BIG DOWNLOAD QUEUES: If there are 5000 files directories, is very easy to add them to download queue, what happens before lot of people knows, RS HANGS again.... Adding lot of files, and loading them in download queue should be fast, instant.

THINGS ALREADY DONE IN OCTOBER NOVEMBER 2017: THINGS ALREADY DONE IN OCTOBER NOVEMBER 2017: THINGS ALREADY DONE IN OCTOBER NOVEMBER 2017:

DONE : CHANNEL POST DOWNLOAD WITH ONE CLICK WITH DIRECTORY STRUCTURE: DONE: We need in a channel post a way with one click in DOWNLOLAD, after that in oru download directory a directory structure with files inside, now you can do it but without directory structure. I proposed a way here, but i'm open to another ways, but must be easy to add content with DRAG AND DROP and easy to download with ONE CLICK, that's why i hate RSCOLLECTIONS. DONE This is my proposal to do it : https://github.com/RetroShare/RetroShare/issues/635 But if you know a better way, now is too complciated because uploader needs to create the rscollection manually, share this file somewhere, you need to download that file, open it, and add to dq the structure.

1. DONE FILE LISTS CREATION symlinks (linux) and direct access (windows): The creation of files lists needs in windows and linux the use of symlinks, in professional environments the amount of data is very high and the normal solution is to use symlinks to order things so not need to have duplicate content. RS in linux follows symlinks but is not working very well and hide lot of content marked as duplicate, should be shown, in windows don't follow by now the direct accesses .lnk files. PROPOSAL TO FIX IT: Redo the indexing algorithm in windows and linux, instead of algorithms trying to discover loops or duplicate content use a technique of DEEP LEVELS, how many DEEP LEVELS of directories we want in RS, for a normal user 5 levels are enogh to order data, for kind of very estrange users 10 deep levels is enough, a loop with symlinks uses infinite levels, so is very easy to not let RS index loop, just not indexing more than for example 10 levels deep. Levels deep means directory inside directory inside directory, all these are one deep level each. And also in windows follow .lnk files, optional like symlinks. To not duplicate hash there are functions to detect if a file inside a symlinked directory is a duplicate one, if it's duplicate, dont hash again, just get the has from the hash table and designate to that, file.

2. DONE SLOW RESTARTING OF DOWNLOAD QUEUE: It takes like 1 or 2 secs per file to be shown after restart in RS, you if you have 10K files in the queue you will need 20K secs to see the whole download queue. Don't makes sense. PROPOSAL TO FIX: Show directly at restart all the files, and after that check file by file if it's available, now the check is made while adding files to download queue i guess, and that's why takes so long.

DONE : HASING: The hashing process slowdown the whole computer and should be easy to stop it, and have stats about the process. RS works well hasing 50 files, but if you add 90.000 files and 20TB, rs will use your cpu for a long time, and needs to be stopped, some work has been done, so now you can pause hasing, but by now there is no a place to see how hashing is going on, is not possible to pause in the middle of a hash, you can stop the hasing but you have to wait to finish hashing, if you need to pause hasing for a reason, you can't wait this 20GB file to finish hashing, you need it now. PROPOSAL TO FIX IT: DONE In Files, a tab called HASHING, where you can see which file is being hashed, how many Mb/s your cpu is hashing files, files to hash, and be able to pause it instantly. Now all this information is more or less in the status bar, and is also good there but this tab is needed with more info.

There are lot of things already done, like lot of bugs fixed for download queues in direct and anonimous ways. Pause hasing. Better upload integration. Direct connections forced. Friends upload limits. Channel edit post. etc.

[csoler]

at least I can explain why there's 1 file added every second in DL queue: it's a timer that handles the queue, only one item every time it's called. It's pretty easy to change this so as to add more files at once. The code of adding files is super small.

[felisucoibi]

what i'm saying not necessary to see them while adding them also to search for sources. Load all of them and in backgruond search for results that matches.

[csoler]

for the big lists, the main problem is that Qt is not good as displaying lists with many elements as it happens with file lists; I believe that most of the time, the sorting kills the performance. For channels, the problem is 2-fold: there's a cost in accessing the database, that we cannot really compress. There's also a cost with handling of long Qt widget lists. For this, we can implement a Qt container that only retrieves and displays the 4-5 widgets that can be made simultaneously visible.

[felisucoibi]

We already talk in forums but i leave it here to... "the main problem is that Qt is not good as displaying lists with many elements as it happens with file lists" Just remember esikaltdc++ is in qt and can handle 20.000k / directory filelists without problems, maybe the problem is we use trees instead of table with line elemets, that can be fixed with my proposal of filelsits (the third column is a list not a tree), let's see the develoeprs working on optimization if they find something.

[GiordanoArman]

Another needed feature would be proper support for the mobile devices. None of my friends would leave Whatsapp without a working mobile version of RS. Maybe exploiting the web interface you could have a cross platform mobile application without going through the fuss of having different UIs for every platform.

I think I can setup a Kickstarter campaign. What would be the budget needed to have a mobile application? Is there anyone I can write to and eventually talk to? I would first need to collect addresses and contacts to advertise the campaign.

[felisucoibi]

I update the done things.

@GiordanoArman : That is also important and is on the way i hope.

[GiordanoArman]

@felisucoibi I have studied a little more the other tools and found Tox. The official repo has not seen a push in a while, but there is a fork of it that is developed by a group called The TokTok Project. Tox has already some functionalities working out of the box for mobile. We really need to think how to do this, since developing two apps that do the same thing is a waste of energies that could be used to have a single, very good solution.

[felisucoibi]

@GiordanoArman i proposed to put tox inside rs for chats a few months ago, desktop sharing, chatrooms and all the things tox already does better than rs, all this for desktop, and in mobile just use tox.... but this is dream and maybe devs hates the idea.

[GiordanoArman]

@felisucoibi The main problem of these projects is that they serve a purpose that the general public does not understand. There is no demand, therefore the adequate supply isn't there. This needs to be fixed if we want better tools, motivate developers and involve even more developers. We could start using social media to explain people why the privacy issue is important. After this there is the issue regarding the amount of tools doing the same thing but not quite reaching their goals. A way to solve this could be joining the efforts and create a flexible solution, whether a library or an abstract protocol that then everybody can implement their own way. Tox does this, but I've read that some are concerned about the implementation of the security model and this is not ok, it needs to be bullet proof and proven to be bullet proof. I am thinking that asking universities to be involved in this would be a good idea, they could validate the security model implementation, for example.

PS: I am not saying that the developers should be blamed for anything.

[felisucoibi]

go for it, i'm trying to help in similar way with retroshare, i helped as much as i can in simplify, make normal user proof etc... still lot of work, and devs poor all the time suffering me, but step by step, this is the only way....