Add publish file method

[khubo]

planning to implement @nikolaiwarner PR for sending files :dancer:

[khubo]

solves #25

[cblgh]

ah nice, i remember @noffle and i had some discussions about the format to send. i forget what we came up with though, i'll look through my logs :~

[cblgh]

@khubo @noffle here are the notes: https://gist.github.com/cblgh/787e78a3479a4b0801bee2768933ffda

it was apparently karissa and noffle brainstorming! and here's the hackmd the discussion was based on https://hackmd.io/BeZ49Q2ARZavH1v2JbbJKw

[cblgh]

btw @khubo the api.md file is autogenerated using the comments from the source code! if you look at the code description of publishMessage you can see a good example :3

take the the stuff you've manually written in api.md for publishFile and rewrite it inside cabal-details for the publishFile function definition and then run npm run doc to generate api.md :)

(otherwise your descriptions in api.md will be replaced next time we generate the docs!)

[khubo]

@cblgh haha.. oops. I will fix that. I have also read the chat log and docs. I think, I partially understood, and its similar to what @nikolaiwarner tried to implement, ig. I wil try to get it working first, and then maybe we iterate on it?

[cblgh]

yeah +1 on gettin it workin & iterating on it :3 i can help you test throughout the week ^_^

[cblgh]

fwiw it might be worth treating images separately, or including a metadata field as part of the chat/files message schema. reason being that we can then always display file messages containing an image as a blurhash (npm) even if we haven't downloaded it (it's especially useful for public cabals to not show any images as a default)

[khubo]

@cblgh yes, thats how its currently implemented. A seprate dat-noe for each user, and he adds his files to it. other people fetching it will also join the swarm.

I am making it sparse, but not sure if it might be the best case scenario. Again, this same dat maybe can be used for adding a dp :D

[hackergrrl]

This seems like a pretty significant feature: it could be useful to have e.g. a short design doc on how you plan to implement this, so we could discuss & decide on how we'd like to see this feature show up in cabal.

[khubo]

@noffle @cblgh @nikolaiwarner Each user will have their own dat-node per cabal for sharing their files. There can be a directory at each cabal level to dump all the dat-nodes. When a user sends(uploads?) a file in cabal, he will import the file into his dat-node, and adds a feed to his chat log - something similar to

{
    type: 'chat/file',
    contents: {
        key: '' 
        filename: ''
    }
}

maybe instead of key and filename, we should make it something more generic to support resources from other origins? like web? or even torrents?

When someone receives a message of the above mentioned type, the client should try to clone the dat-node (sparse?), - in the above mentioned directory for dat-nodes- if not already cloned, and read the file from archive. At the client side, the component can fallback to a default loading content while the image is being fetched or in case its unavailable.

Everyone who cloned a dat-node will also be swarming it. I was thinking sparse replication because many of the users would be downloading contents from channels that they are not a part of, but the problem with this approach would be that most people are rarely online, and hence unavailable to seed the files.

[hackergrrl]

The question of "who seeds shared files" is tough.

If everybody auto-seeds all files forever, that means:

• Each cabal node needs to participate in N+1 swarms: the main cabal swarm, plus one for each shared file. If a cabal has seen a hundred files shared, 101 swarms. Being part of a p2p swarm is expensive and network intensive. I'm worried this won't scale well.
• All nodes will automatically download and share content blindly, including things the cabal participants might not consent to, because of either size (sharing a 2GB file) or content (pornographic media, alt-right trash, etc).

I like your idea @khubo of generalizing the file share so that it could be a nonspecific "p2p link" (webtorrent, dat, ssb, etc).

---

Another idea might be to store files as additional feeds part of the cabal multifeed. This would reuse the existing cabal swarm for file sharing. Instead of auto-downloading, cabal-client could simply surface the files, and clients could opt-in to downloading using various criteria. I could see clients having different preferences based on the content type:

• maybe max-4kb avatars are auto-downloaded
• maybe once @cblgh's trust system is implemented, you only autodownload from users you trust
• maybe once @substack's moderation frontend is implemented, you never download from blocked users

Overall, a flexible client policy mechanism feels like a huge piece of the consent part of this feature.

---

All this said, I think if this was implemented as-is but was opt-in for downloads, and had controls for maybe how long to reseed content, it seems like it would be a useful addition.

[khubo]

@noffle I was thinking more in lines of 1 dat-feed / user / cabal. So if there are n user present in a cabal, there will be n+1 swarms. But yes, it will also have scaling problems if a user is involved in too many cabal or cabals with high number of users. Maybe as you pointed out its better if they are parts of feeds that are in the same swarm as the chat feeds. Once we have sparse and moderation this will be the better way, ig :smile_cat: