search

dat-ecosystem / dat

:floppy_disk: peer-to-peer sharing & live syncronization of files via command line
https://dat.foundation
BSD 3-Clause "New" or "Revised" License
8.24k stars 449 forks source link

Integrate pinning / seeding daemon with CLI #1081

Open RangerMauve opened 5 years ago

RangerMauve commented 5 years ago

I'm using this to track progress on the work I'll be doing integrating a pinning / seeding service with the Dat CLI.

joehand commented 5 years ago

If you want to start with any of these as an extension we can add them, they just need to be published as dat-something and then whitelisted. Then can be run as top level commands dat something.

RangerMauve commented 5 years ago

I was thinking I'd need to modify the existing registry code so that dat publish would automatically target a daemon. Though launching the daemon could easily be a separate module.

RangerMauve commented 5 years ago

Here's the progress on the extension: dat-pin.

It's "working" locally, but I'll need to add unit tests and stuff next week before I release it.

RangerMauve commented 5 years ago

dat-pin is tested and released. Working on adding it to the CLI

RangerMauve commented 5 years ago

Bikeshedding discussion on IRC:

``` 10:11 <@pfrazee> Paul haha 10:11 going to invite a couple other people. Bikesheds are brutal as open discussions 10:12 RangerMauve I'll have some tissues on hand in case I break out into tears 10:12 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion 10:12 <@pfrazee> Paul ha 10:15 rangermauve: good work on the cli btw 10:15 RangerMauve Thank you! It was handy that you already made the pinning client and garbados made dat-librarian. I just needed to add a bit of glue and persistence 10:22 → jhand joined (sid114474@gateway/web/irccloud.com/x-yujpcnodgzqjzuuj) 10:22 <@pfrazee> Paul https://gist.github.com/pfrazee/199bb8fe92f98ade6c259aa965232cdb closepfrazee — 2 Apr 2019 10:23 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion. Current list: https://gist.github.com/pfrazee/199bb8fe92f98ade6c259aa965232cdb 10:23 <@pfrazee> Paul so my current ideas are "mirroring service" and "persistence service" 10:24 RangerMauve I still really like "publishing to a resharing service" TBH 10:24 Persistence and mirroring are pretty good. They sound pretty techy though 10:24 What's your main concern with pinning? 10:25 <@pfrazee> Paul I don't think it's very descriptive 10:25 RangerMauve I agree. It doesn't relate to what it's actually doing and why it's needed 10:26 <@pfrazee> Paul I'd like to be able to say to somebody "You need a X" and they be like "oh yeah makes sense" 10:26 You need a Permahost 10:26 RangerMauve What kind of somebody are you targeting? I think that unless they already understand P2P, it'll be hard to find a word for it. 10:27 <@pfrazee> Paul Technical to semi-technical, with the potential for it to become vernacular 10:27 people understand what having a "cloud" means now 10:28 RangerMauve And at the same time, if they already know about P2P, then they might already be comfortable with words from other projects like IPFS or BitTorrent 10:28 <@pfrazee> Paul like, if you said "oh no somebody hacked my cloud" people would understand that you had some service that hosted your data and it got broken into 10:28 rangermauve: right, that's why I was comfortable with pinning before now 10:28 and it may be our best option still 10:29 RangerMauve Seeding is also pretty popular due to BT 10:29 <@pfrazee> Paul "Seeding service" is pretty darn clear if you know BT 10:29 "You need a seed" 10:29 RangerMauve From the poll I ran on twitter it seems those two were pretty close. https://twitter.com/RangerMauve/status/1110604165233954816?s=20 10:29 <@pfrazee> Paul oh nice thanks 10:30 RangerMauve I don't think that either of them are intuitive unless you already know what those words mean in the context of P2P. :P 10:30 <@pfrazee> Paul Emilia's point about the connotations of BitTorrent is pretty accurate 10:30 rangermauve: agree 10:31 "Durability" is an interesting theme to play with 10:33 RangerMauve Durability seems good, but I think it would also be aimed more towards sysadmins and distributed systems people. I think that outside those contexts it implies that it keeps your Dat from snapping in half somehow 10:33 <@pfrazee> Paul good point... 10:33 okay maybe there's an "Uptime" or "Availability" theme? 10:33 RangerMauve Same with mirroring, since that's more for people used to file sharing websites that have that 10:33 I think those have the same audience. :P 10:34 Which is OK, IMO 10:34 <@pfrazee> Paul well with uptime you can get an intuitive sensibility for some people, like, it's your remote presence 10:34 the positive about "mirror" is that it suggests it's a dumb thing that's just being a stand-in for you 10:35 RangerMauve Yeah, I think mirror is also a pretty common word used for something like this. 10:35 <@pfrazee> Paul it's in the "not bad" category for me 10:35 RangerMauve People thinking of Dat in the context of file sharing would already know what it means to some extent. 10:36 <@pfrazee> Paul it's also got history with websites 10:36 RangerMauve Yeah! 10:36 Brb, feeding cats 10:36 <@pfrazee> Paul kk 10:37 another idea: "Keeper" 10:39 another idea: "Steward" 10:39 <@pfrazee> Paul another idea: "Base" 10:45 → mafintosh joined (sid28509@gateway/web/irccloud.com/x-dupfcvhgfkzoouca) 10:49 RangerMauve I just have this nightmare where Amazon gets into P2P hosting and starts extending things with closed source functionality 10:49 Or some other big company 10:49 <@pfrazee> Paul well IMO, if dat becomes a thing, you should count on Amazon and Google and Microsoft doing that 10:50 pinning services are a hosting resource and they have hosting resources 10:51 RangerMauve I guess they'll _have_ to add proprietary extensions if they want to differentiate themselves 10:51 <@pfrazee> Paul perhaps. I get that anxiety. It's literally what google is doing right now with AMP on the Web and Email 10:51 Mathias Buus Hello 10:51 <@pfrazee> Paul mafintosh: hey 10:52 RangerMauve 👋 Thinking of words that are more intuitive than pinning to people that don't already know what pinning is 10:52 I'm a fan of mirroring and seeding, but those still require you to have a certain background 10:53 <@pfrazee> Paul rangermauve: if we were to focus on there being lots of pinning (or X) services, what I'd feel the need to solve within beaker is a toolset for people to choose providers 10:54 so what I'd want to do is setup arrangements with providers to maintain an SLA that promises they'll stick around in exchange for being recommended by beaker 10:54 RangerMauve Yeah. Part of the reason I really wanted to get this pinning stuff in the CLI (and the discovery-swarm-web stuff before) was to make the standards easy to use. There's a small bunch of companies working with Dat content, and I think they're all inventing their own pinning services and gateways 10:55 If we had a way to tell them "Here's what a pinning service should look like" then we'd have less fragmentation and something like that would be more feasible 10:55 <@pfrazee> Paul rangermauve: agree, and Im really glad you're doing that. Also, to be clear, Beaker will always make custom providers easy to setup too. 10:56 yes 11:01 <@pfrazee> Paul so, we can circle back to the "many providers vs a few" topic in the future, but from my perspective with beaker, what I need is reliable partners 11:02 RangerMauve Agreed 11:02 <@pfrazee> Paul I'm going to have beaker include one-click default providers to signup for. I'd be 1000% happy if beaker didnt run them, and if there was more than one 11:04 but pinning providers should *not* have a way to differentiate from each other, and theyre incredibly low margin 11:04 so- let's setup another conversation about that and get darius involved 11:04 RangerMauve Sounds good. :P 11:05 So all that to say, I think mirroring is a good term because there's usually lots of mirrors for a given file and it's a known term for people that do file sharing but maybe not p2p 11:06 <@pfrazee> Paul yeah, mirroring isnt bad. Let's try a few more terms 11:06 mafintosh: just to check, do you have any bikeshed suggestions on that? 11:07 Mathias Buus are there any other things other than ipfs that use the word pinning today? 11:07 <@pfrazee> Paul no 11:07 RangerMauve Should we use an existing term or invent a new one? 11:08 <@pfrazee> Paul I think both should be on the table 11:08 Mathias Buus mirror is good 11:08 <@pfrazee> Paul (I'm just going to throw out random ideas that come into my head as we talk) 11:08 idea: Courier/carrier 11:09 RangerMauve I'm personally more into using existing terms since it'll result in fewer people to educate. I'm usually pretty skeptical of P2P/Blockchain projects that invent a bunch of new terms for existing concepts 11:09 <@pfrazee> Paul rangermauve: oh yeah I hear that. 11:10 RangerMauve There was one I read recently, and it had like 20 terms and most of them were just existing concepts that got turned into marketing material 11:10 Mathias Buus pfrazee: courier is no good for non native speakers 11:10 carrier also, they'd think aircraft carriers 11:10 <@pfrazee> Paul mafintosh: yeah. (No need to respond to any of my ideas if they dont land, Im just throwing em out) 11:11 Mathias Buus :) 11:11 uptime service 11:11 <@pfrazee> Paul I was toying with "uptime" earlier 11:14 RangerMauve Yeah, so it'd be cool to ~steal~, I mean _borrow_ existing terms 11:14 (I mean, that's why I think it would be cool) 11:15 <@pfrazee> Paul yeah 11:16 Mathias Buus what are some existing terms? other than mirror 11:16 and hosting 11:16 <@pfrazee> Paul server 11:16 cloud 11:16 RangerMauve seed, and pin? :P 11:16 Mathias Buus replica 11:17 <@pfrazee> Paul replica... 11:18 RangerMauve I think replica is a bit too techy. I don't think everyone knows about databases 11:18 <@pfrazee> Paul node 11:18 Mathias Buus copy 11:18 RangerMauve file share? 11:18 <@pfrazee> Paul store 11:18 Mathias Buus i like store 11:19 RangerMauve repository (that's what's in the CLI now) 11:19 Mathias Buus 'can you store this dat for me' 11:19 <@pfrazee> Paul 'store service' 11:19 RangerMauve storage service? 11:19 Mathias Buus store sounds friendly to me 11:20 it's the kinda thing you'd say to your neighbor 11:20 <@pfrazee> Paul `dat store add` 11:20 RangerMauve `dat store set http://hashbase.io` `dat store dat://foobar` 11:20 <@pfrazee> Paul that might have legs 11:20 RangerMauve I like it :D 11:21 <@pfrazee> Paul okay let's sit on that for a bit 11:21 rangermauve: if you can just hold off merging your pinning tools into the CLI till we're sure 11:21 RangerMauve Would you say "Store your dat in a store" or "Store your dat in a storage service" or "storage provider"? 11:21 Yeah, I've still got some work to do around configuration 11:22 Which is what I'm doing now 11:22 <@pfrazee> Paul I think store and storage can be somewhat interchangeable 11:23 but I'd call the service a "dat store" maybe? 11:23 I think this is pretty good 11:23 RangerMauve What about when there's a marketplace of Dat apps? :P 11:23 I guess it'd be external to the protocol itself 11:24 <@pfrazee> Paul rangermauve: my thinking with beaker is that services and apps will be two different marketplaces with their own sets of sub-types 11:24 "dat stores" being one kind of service 11:24 and beaker will compose them for users 11:24 RangerMauve Sounds good to me 11:29 RangerMauve I'll get started on renaming the commands and stuff 11:29 Joe Hand cloud friend 11:31 Joe Hand can we make the command blue though 11:31 RangerMauve :P 11:31 ?? 11:31 <@pfrazee> Paul jhand: I prefer chartreuse 11:31 rangermauve: bikeshedding jokes 11:32 RangerMauve Ha ha, that totally went over my head. 😂 11:32 <@pfrazee> Paul okay thanks yall for indulging me on this 11:32 Joe Hand I like store! 11:32 <@pfrazee> Paul yeah me too :D 11:32 RangerMauve Yeah, it's pretty nice 11:32 <@pfrazee> Paul I'll keep everyone updated on the meeting with darius, jhand I'll invite you as well 11:32 Joe Hand though that gives us less flexibility around "registry" type pinning services ```

Mildly redacted to stay on topic

Conclusion:

We'll rename dat-pin to dat-store and use dat store dat://somekey along with saying "Put your read key in your dat store".

@pfrazee Maybe you could update the wording on hashbase to align with this term?

pfrazee commented 5 years ago

@RangerMauve Yeah I'll have to get that on my todos